Receiving CRC errors on every switch connected to a Canopy

I am having an issue with CRC errors received on Cisco switches from Canopy SM’s. I have over 100 SM’s on site all of which are connected to Cisco switches. The switchport connecting the canopy SM on every switch is receiving input error of which every error is a CRC error. The Cisco switches are models WS-C2950T-24, IOS version 12.1(22)EA12, WS-C2960-8TC-L, IOS version 12.2(46)SE. The canopy is a 6 AP cluster using 5251AP’s, Software Version 7.3.6. The SM’S are a mix of Platform 8 and Platform 9 Model 5201SM, Software Version 7.3.6. All Canopies are configured using Sofware scheduling. I have tried changing the negotiate settings on both the switch and the sm’s to no avail. The SM’s seem to recognize the Cisco switch but the cisco switch does not seem to notice any changes to the SM’s, instead they only see the other Cisco switches on the other side. If I change the speed or duplex settings on the Cisco switch the SM will change its speed or duplex settings but the Cisco switch will not notice that change, the Cisco switch on the other side of the link will begin to log duplex or speed mismatches to the Cisco switch that the change was made on. As far as the Cisco switches are concerned the Canopy SM’s are not even there. Changing the Trunk port to speed 100 Full Duplex and setting the SM to only “100 Base T Full Duplex” will cause the SM and Switch to no longer communicate. No matter what combination os settings, the CRC errors are still reported on the Switch interface. The SM will show no such issues on the Ethernet Stat page.

AP Configuration:

Downlink Data Rate: 50%
High Priority Uplink Pertentage: 50%
Total NumUackSlots: 6
UAcks Reserved High: 3
NumDAckSlots: 6
DAcks Reserved High: 3
NumCtlSlots: 6
NumCtlSlots Reserved High: 3
Sustained Uplink Data Rate: 20000
Uplink Burst Allocation 500000
Sustained Downlink Data Rate: 20000
Downlink Burst Allocation: 500000
Max Range: 3
Airlink Security: Encryption Disabled
Authentication Mode: Authentication Disabled
Configuration Source: SM
Bridge Entry Timeout: 1440
Power Control: Normal
Broadcast Repeat Count: 1
Transmit Frame Spreading: Disabled
Encrypt Downlink Broadcast: Disabled

Canopy SM Configuration:
802.3 Link Enable/Disable: Enable
Link Negotiation Speeds:
100 Base T Half Duplex
100 Base T Full Duplex
SM Power UP Mode: Operational Mode
Bridge Entry Timeout: 720
Sustained Upliunk Data Rate: 3500
Sustained Downlink Data Rate: 3500
Uplink Burst Allocation: 500000
Downlink Burst Allocation
Authentication Key: Use Default key
Frame Timing Pulse Gated: Enable
Power Control: Normal

Cisco Trunk Port Setting:
description Trunk Port
switchport trunk native vlan 100
switchport trunk allowed vlan 100,200
switchport mode trunk
switchport nonegotiate
load 30
speed 100

On an individual SM/switch pairing, when both sides of the link are forced to 100FDX and you clear the counters on the Cisco, do CRC errors still accumulate? You will not see any signs of the Canopy stuff as they run LLDP, not CDP.

Yes, the errors continue to accumulate after clearing the counters. I do not believe the Canopy 7.3.6 software version supports LLDP. IOS version 12.2(46)EA for the 2960 switch has support for LLDP but when I activate it, it still does not see the SM.

Switch(config)#lldp run
Switch#sh lldp ?
entry Information for specific neighbor entry
errors LLDP computational errors and overflows
interface LLDP interface status and configuration
neighbors LLDP neighbor entries
traffic LLDP statistics
| Output modifiers

Switch#sh lldp neigh

Capability codes:
® Router, (B) Bridge, (T) Telephone, © DOCSIS Cable Device
(W) WLAN Access Point, § Repeater, (S) Station, (O) Other

Device ID Local Intf Hold-time Capability Port ID

Total entries displayed: 0

Switch#sh lldp entry *

Capability codes:
® Router, (B) Bridge, (T) Telephone, © DOCSIS Cable Device
(W) WLAN Access Point, § Repeater, (S) Station, (O) Other

Total entries displayed: 0
Switch#sh lldp traffic

LLDP traffic statistics:
Total frames out: 3
Total entries aged: 0
Total frames in: 0
Total frames received in error: 0
Total frames discarded: 0
Total TLVs unrecognized: 0

we’ve had problems before with sm’s and cisco equipment. try starting from 10 half forced on both ends and go up from there to see if it resolves the issue. personally the ne we have plugged into cisco we have forced to 10mb half

We have seen this in the past. If I recall, we had to force both the radio and the switch to 100FDX to resolve.

yes, but its not a general rule…
sometimes it gets stable on 10h sometimes on 100f. depends on the software version…but cisco port and SM sould be forced.

It seems logical to force the Cisco switches to 10 HDX considering that I am seeing several switches on one trunk port, the Canopies are logically a hub, not a switch. I went to my switch connecting to the Canopy Cluster Management Module Micro and forced it to 10 HDX. I looked at the AP’s and picked on AP that had only three devices associated and forced the trunk ports and all of those switches to 10 HDX. The CRC errors continued to occur on all four switches. I also added the following command to all four switches:

spanning-tree link-type shared

which had no affect.

I have a project coming up shortly to upgrade that AP Cluster and all the SM’s on that part of the network to Canpoy Software 8.2.7 which I hope will help resolve the issue. In the meantime I will try some other Cisco configuration changes…since I have only two vlans…changing the trunk ports to access ports.

moinavery wrote:
yes, but its not a general rule....
sometimes it gets stable on 10h sometimes on 100f. depends on the software version...but cisco port and SM sould be forced.

This is true. Cisco makes great products but their auto-negotiation sucks, and has sucked since their first switches that they released.

CRC errors are directly associated with the physical interfaces. There won't be any correlation between the switches on the network in terms of CRC errors. Basically, just because you see CRC errors on a switchport on switch A, doesn't mean you will see errors on a switchport on switch B, just because they are tied together via radio.

Now, if the switches were connected back to back via a cable, then yes you would possibly see errors on the interfaces. Since your switches are connected together with an intermediate device (backhaul radio of some type that has an ethernet interface on it) this does not apply.

You should mess around with the speed and duplex configuration on the switch until you find a combination that works without generating errors. I did this last month on one of my sites. I had forgotten to configure it when I originally installed the site.

If you have tried all combinations, it could be a faulty port or radio. It also may be noise on the ethernet.

I have resolved this issue by upgrading all the AP on my cluster to P10 units and switching all the SM’s on the cluster to Hardware scheduling. I have spoken with other network admins at other locations who have reported the same issue effecting the SM’s but not BH units. It is in my opinion that this problem exist when the SM are using software scheduling. I do not believe it was the new hardware so much as it was switching to hardware scheduling that resolved the issue. The latency on the Sm links also decreased from 40 ms to 15 ms on average.

P10 hardware and latest software has better negotiation with cisco than 7.3.6

Aditionaly, I have a couple of ap that survive a near lightning but have to force ethernet to 10F or 10H to function properly…just wanted to share that

error due to duplex mismatch.

What product are you using and have you tried forcing the Ethernet port setting to something other than Auto negotiation?

What switches are you trying to connect?  

As you might note from the date of the previous post, this has not been an issue for quite some time (~7 years or so).

We too have been seeing a lot of CRC errors on our AP's.  I'll give the auto to forced negotiate a try.  

What model products are connected, which software revision are they on?

What are they connected to?