450i generating CRC errors on Cisco routers and on CMM4 Switch

We are in the process of upgrading our existing PMP 400s to PMP 450i.

So far we have about 20 SMs in place with two APs.

We are noticing CRC errors on the Cisco routers attached to the SMs and also on the switch in the CMM4 for the APs.

These were working fine with the PMP 400 but now generate CRC errors with the PMP 450i.

We have tried different power injectors, with and without lightning arrestors, and different speed/duplex settings.

No change in the CRC errors. What else can we try?

Also, as a side note, 1 AP and 1 SM were updated to the 14.1.1 software and the SM is now showing

"24 dBm (target power [41 dBm] exceeded maximum)"  with no way of turning down the power.

Are you actually seeing CRC's or input errors?

It increments the counters at at rate of 27 errors in 90 seconds for 17 connected SMs.

We are getting droped voice packets.

Hi Bernd,

Can you please try disabling CDP (Cisco Discovery Protocol) on your Cisco routers to see if this clears up the issue? We have identified an issue with CDP packets and 450i and are investigating a fix.

Thanks,

-- Josh

1 Like

Some routers, Dlinks in particular, don't like 100 Mbps cable links.  It happens with Canopy equipment, but I've also seen it on PCs wired into Dlink routers.  At the suggestion of Dlink support, I've been setting our SM cable links to Auto 10 full/half.  I've found that it also makes the cable link a lot more stable in general, especially when dealing with longer cable runs.

I doubt that's the issue you're seeing, but it's something worth trying.

Bernd and all,

First, many thanks for your patience as we've worked to track down the root cause of this problem.

We've identified and tested a fix which will be deployed in the next software release. We are also discussing beta/early access options for customers who require this fix earlier, through the normal beta and support channels.

This issue only affects packets using 802.3 frame encoding. In particular, the "EtherType/Length" field for these packets is less than 1536 (0x0600), indicating a length instead of protocol. These packets, when received over Ethernet by a 450i radio, will be sent out of the remote radio (even if the remote radio is not a 450i) with a bad FCS. The packets will then be flagged as a FCS/CRC error by the receiving device.

Although Cisco CDP is the most common case where we've seen this, it is not specific to Cisco switches.

Please note that typical IPv4 traffic, ARP, LLDP, etc., all use Ethernet v2 framing, so they are not affected by this issue.

-- Josh

2 Likes

We had this issue on our very first pair of 450i radios we installed.  The customer was very worried that heir data was being affected, so I just placed a small dumb switch in between the 450i and the Cisco router and all was good.  Somple fix and they were happy.

Hi Josh,

Thanks for your feed back. I'm experiencing the same issue as well.
I have a PMP 450 SM associating to a PMP 450i AP. Both radios have been updated to the latest firmware 14.1.2, but errors are still incrementing on the SM's cisco device. When I have a wireless association to the AP, errors begin to increment. When there is no association, there are no errors.
Please advice.

Wireless Association to AP:
 413 input errors, 413 CRC, 0 frame, 0 overrun, 0 ignored

No wireless association to AP:
 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

Many Thanks,

Rollen.

I am also seeing this on 4 AP's (2 450 & 2 450i) after upgrading to 14.1.2.  In our case the AP's are talking to Mikrotik CCR1036 routers.

Hi Rollen and Craig,

Here is exactly what was happening when the Cisco packets were causing problems. The 450i Ethernet MAC has a configuration register with a bit that can be configured to make it strip the padding and FCS/CRC field on incoming frames with Ethertype less than 0x0600 (1536 bytes). This is IEEE 802.3 framing, which is pretty uncommon, but is used occasionally such as for Cisco CDP packets. We mistakenly had this configured to strip off this information.

Normally, unless we have a reason to regenerate a CRC, such as if we do VLAN remarking, etc., we avoid it in order to maximize throughput. When we do have to generate a CRC, we use hardware acceleration to make it as quick as possible. The vast majority of packets are transmitted over the air interface with their CRC intact, so we just hand them directly to the Ethernet MAC on the other side. This meant that since we didn't know the Cisco CDP packets had no CRC, we just put them on the wire without one, and thus the switch saw CRC errors. The actual bug was on the radio which received the packet from the source Ethernet, not the one which sent it out the Ethernet on the other side of the link, which made it an interesting bug to track down since we naturally assumed the problem was happening on the radio connected to the switch seeing the CRC errors.

I just looked at the 14.1.2 code tag, and the fix still appears to be present, so it is possible this is a different problem. Have you opened a ticket with the support team on this? Internally, we will try to get a lab setup which has the same or very similar hardware (such as the Mikrotik CCR1036) so we can try to reproduce what you are seeing. The more details on your equipment and network setup you are able to share with us, the better.

The reason I'm asking about the support team is that they have a more extensive equipment setup to use than I do here at my desk in development. If they can reproduce it, then we can track down what packet(s) are being affected and why. If you happen to notice things in particular that aren't working, that may also help us narrow it down.

Thanks,

-- Josh

Hi Josh,

I just emailed support with a detailed explanation.

Craig

Hi Craig,
can you please send this email one more time to this email address: ziad.alsaeedi@cambiumnetworks.com

Thanks!

I've upgraded firmware on my radios to latest 14.2 and all errors have cleared.

1 Like