PTP820 9.7 to 10.3 now seeing packet loss

Very strange issue... we updated (3) PTP820 links from firmware 9.7 to 10.3 recently. One of the links is now showing about 3% packet loss over it where before it was perfect. Topology and config is dead simple. Mikrotik CCR's connected directly to the 820's using copper gig-e and in-band mgmt. Radios setup using 1+0 config and a single 80MHz channel and a 500mbps license. We've checked all kinds of stuff:

  • Tried power cycling both radios
  • No CRC's or FCS errors on the 820's or the Mikrotik's ethernet interfaces. Everything running at full gig-FDX
  • Both radios MRMC status show full modulation and full bandwidth @ 550.615mbps
  • Radio statistics not showing any dropped packets, frame error rates of 0 on both radios
  • Performed Mikrotik to Mikrotik BW tests and are seeing what we should be seeing, around 550mbps FDX
  • When we ping from Mikrotik A to radio A, there's no packet loss. When we ping from Mikrotik B to radio B there's no packet loss. When we ping from Mikrotik A to radio B, there's packet loss. When we ping from Mikrotik B to radio A, there's packet loss. When we ping from Mikrotik A to Mikrotik B, there's packet loss.
  • We did notice that on the radio there's some "TX Length Error Frame Count" slowly incrementing, but we checked the other PTP820's we have that aren't showing this issue, and they're slowing incrementing as well:

We've opened up a ticket with Cambium support and are going through the support hoops now, but I thought I'd reach out to the forum here to see if anyone had any ideas. Next step we're thinking is to reset to factory defaults and reconfigure from scratch... if that doesn't work, then roll back to 9.7 and reconfigure from scratch. Unfortunatly, when we try to roll back to 9.7 it says that it's going to wipe out the config and reset to factory defaults. In addition, we didn't think to make a backup of the config when it was running 9.7.

If anyone has any tips/tricks for reconfiguring remotely please let me know. My fear is that when we reset to factory defaults, it's going to default everything to use the MGMT port and we'll have to physically visit the sites to reconfigure the radios.

Thank in advance!

"TX Length Error Frame Count"  is not used - known issue, so dont bother with it.

what you need to do is to look at your switch's statistic - example, start from clean and send 100 packets, make sure that 100 packets was sent and then look at RMON to see if that 100 packets were received and Transmitted out...

Few things to check:

- MRMC Script configuration

     - In an 80MHz wide channel, the highest modulation supported is 1024QAM (profile 8).  The radio will allow you to configure modulation levels higher than that, but it is not a supported configuration and you may see weird behavior (such as packet loss).  This is documented in the release notes (has been there awhile).

- Management Services setup

    - When using in band management, use VLANs.  The management service is a 10/100Mbps service and can be overloaded if there is a mixture of payload and management traffic across the same interface.  I've attached a management guide.

- Frame size

   - The PTP820 product supports jumbo frames, but by default sets the MRU (i.e. MTU) value to 2000 bytes.  This can be increased under Ethernet>General Configuration

- LAG

     - This doesn't seem to be a configuration that you are using, but I have seen some packet loss in a LAG setup using 10.3.

1 Like

@Chris_Fikert wrote:

Few things to check:

- MRMC Script configuration

     - In an 80MHz wide channel, the highest modulation supported is 1024QAM (profile 8).  The radio will allow you to configure modulation levels higher than that, but it is not a supported configuration and you may see weird behavior (such as packet loss).  This is documented in the release notes (has been there awhile).

- Management Services setup

    - When using in band management, use VLANs.  The management service is a 10/100Mbps service and can be overloaded if there is a mixture of payload and management traffic across the same interface.  I've attached a management guide.

- Frame size

   - The PTP820 product supports jumbo frames, but by default sets the MRU (i.e. MTU) value to 2000 bytes.  This can be increased under Ethernet>General Configuration

- LAG

     - This doesn't seem to be a configuration that you are using, but I have seen some packet loss in a LAG setup using 10.3.


Thanks for the suggestions Chris... we believe we've found a bug in how the 10.3 release is handling the 80MHz MRMC script. We finally found an error metric that we can reliably monitor that increments along with the 3-4% packet loss we see over the link. The counter is under the ethernet RMON stats and it's called "RX FCS errors". We've found that when we use the 80MHz script that it will reliably produce this error (although we do not see any other radio related errors, lost frames, defective blocks, etc). Regardless of the top end modulation that's set.. e.g even if we set the highest modulation to 64QAM... it will reliably produce these errors. Moving to any other, smaller sized channel width (we're using 60MHz now) stops the RX FCS from counting and the packet loss disappears.

We've also run into one other user on FaceBook that's having a very similiar issue when moving from 9 to 10.3. I'm still trying to get more details on if he's running an 80MHz channel width, but so far his symptoms sound exactly the same as mine. He rolled back to 9 and now everything is fine.

Cambium can provide a workaround for those looking to downgrade without losing the entire configuration. It requires that you edit the cli_script file within the backup from another 820 in your network with the same part number of the unit to be downgraded and the firmware that the unit will be downgraded to.

We experienced an issue with firmware 10.3 on a new secondary link where the radios were not reaching the same profile/modulation as the primary backhaul even though the conditions and other specifications were about the same. Downgrading the firmware seems to have solved the issue.