cnMatrix critical issue on SFP+ port when configuring via cnMaestro

Products cnMatrix
1

We had one of our main switches (cnMatrix EX-2028) loose connection to our fiberbackbone last friday after syncing config through cnMaestro. It resulted in us having to go to the location (1 hour away) to fix the issue. During the troubleshooting process, we were not able to get the fiberconnection up and running again, som we had to do some temporary fix using a mediaconverter in front of the switch.

This all started when we by a coincident saw a lot of packet errors on the switch. Because we are using cnWave in the area, we have set MTU size to 2000 on all ports (even on the uplink fiberport), and thinking that a too large and possibly mismatching MTU size on the port connecting to our serviceproviders fiberbackbone, we tried setting the uplink port MTU size to 1500 (which we are using at other locations). Then we suddenly lost connection.

When trying to replicate the same issue at the lab, we have found a quite “scary” issue when editing switchports in cnMaestro: when changing MTU size on a SFP+ port, it seems cnMaestro “adds in” configuration of the port speed as well:

image
image1082×412 20.9 KB

this makes the SFP loose link if the SFP is only 1,25G. As I understand, the SFP ports are supposed to auto-detect the speed based on what gbic is inserted - and it indeed does, but whenever you push a config of MTU, you get the “added” line in the config telling the speed set on the same page (see image above). When it is’nt possible to choose speed “Auto” on the SFP ports in cnMaestro, it is easy to make this mistake! I have read somewhere (probably on the forum or elsewhere on cambium-documents) that you are NOT supposed to manually set the speed for the SFP, so this setting should not be available on cnMaestro - at least when the configline gets pushed even if you just change the MTU size and press save.

If you have have pushed this config, it is not possible to set it back to auto-negotiate from cnMaestro (both because you already have lost connection to the switch, as well as that there is’nt any possibility to choose speed “auto” from the dropdown menu)…

We had to manually configure “interface extreme-ethernet 0/2 speed auto-detect” through cli to remove the command in running-config that said “speed 10000”:
image

image

image

I was afraid that cnMaestro would alert “Not in sync - device config changed outside cnMaestro” when changing this setting through cli, and therefore make this error unreversable, but it does’nt, so luckily this fixes the problem.

I think cambium most certainly have to do something about this, as this is a critical issue!
Either:

  • make “Auto-detect” an option on the dropdown menu in port-configuration->physical->Speed
  • remove the option of setting the Speed on SFP ports in port-configuration->physical->Speed

By the way, we are using cnMaestro cloud, and this has been tested on both switch firmware version 4.5-r3 as well as 5.0.2-r4.

Anyone else seen this, and/or do Cambium Networks even know about this?

I would appreciate feedback!

2

Hello,

  1. The switch has a recovery mechanism to automatically restore the connection with cnMaestro. In the event the new configuration causes the switch to lose connection to cnmaestro, the switch will reboot itself in 15 mins using the previous known working configuration.

  2. The speed of the fiber port is automatically set based on the transceiver type encoded in the transceiver. On the EX2028-P, a SFP or SFP+ transceiver can be used on its 4 fiber ports. When a SFP is inserted the fiber port will be configured for 1G. Similarly, a SFP+ transceiver will be set to 10G. That said, there have been reports that switch does not set the ‘wrong’ speed, and this is because the transceiver does not follow the MSA standard by having the wrong port type in its EEPROM. On cnMatrix, you can run the CLI command ‘show interface transceiver’ to display various information inside the transceiver, including the transceiver type.

  3. In some cases, it is required to manually configure the port speed of the fiber port. For example, a SFP+ transceiver is inserted but the other connection end has SFP transceiver. In this case you need to force the speed of the SFP+ to 1G.

  4. MTU size and port speed are not tied together. It looks like a software defect in cnMaestro. We will reproduce and fix it. Please raise a support ticket for this issue.

Thanks!

3

Thanks TamN!

  1. The problem with this, is that you then get the error “not in sync” on cnMaestro, and it’s hard to find what part of the config we tried to push was the cause of the switch going into recovery…
    Then you cannot try to push the config again,as the switch willl go into recovery once again - and you have this “chain of events” going like a loop.

2/3. That is exactly how I thought it was.

4: I’ll go ahead and raise a support ticket referring to this thread.

Thanks