cnMatrix critical issue on SFP+ port when configuring via cnMaestro

Products cnMatrix
1

We had one of our main switches (cnMatrix EX-2028) loose connection to our fiberbackbone last friday after syncing config through cnMaestro. It resulted in us having to go to the location (1 hour away) to fix the issue. During the troubleshooting process, we were not able to get the fiberconnection up and running again, som we had to do some temporary fix using a mediaconverter in front of the switch.

This all started when we by a coincident saw a lot of packet errors on the switch. Because we are using cnWave in the area, we have set MTU size to 2000 on all ports (even on the uplink fiberport), and thinking that a too large and possibly mismatching MTU size on the port connecting to our serviceproviders fiberbackbone, we tried setting the uplink port MTU size to 1500 (which we are using at other locations). Then we suddenly lost connection.

When trying to replicate the same issue at the lab, we have found a quite “scary” issue when editing switchports in cnMaestro: when changing MTU size on a SFP+ port, it seems cnMaestro “adds in” configuration of the port speed as well:

image
image1082×412 20.9 KB

this makes the SFP loose link if the SFP is only 1,25G. As I understand, the SFP ports are supposed to auto-detect the speed based on what gbic is inserted - and it indeed does, but whenever you push a config of MTU, you get the “added” line in the config telling the speed set on the same page (see image above). When it is’nt possible to choose speed “Auto” on the SFP ports in cnMaestro, it is easy to make this mistake! I have read somewhere (probably on the forum or elsewhere on cambium-documents) that you are NOT supposed to manually set the speed for the SFP, so this setting should not be available on cnMaestro - at least when the configline gets pushed even if you just change the MTU size and press save.

If you have have pushed this config, it is not possible to set it back to auto-negotiate from cnMaestro (both because you already have lost connection to the switch, as well as that there is’nt any possibility to choose speed “auto” from the dropdown menu)…

We had to manually configure “interface extreme-ethernet 0/2 speed auto-detect” through cli to remove the command in running-config that said “speed 10000”:
image

image

image

I was afraid that cnMaestro would alert “Not in sync - device config changed outside cnMaestro” when changing this setting through cli, and therefore make this error unreversable, but it does’nt, so luckily this fixes the problem.

I think cambium most certainly have to do something about this, as this is a critical issue!
Either:

  • make “Auto-detect” an option on the dropdown menu in port-configuration->physical->Speed
  • remove the option of setting the Speed on SFP ports in port-configuration->physical->Speed

By the way, we are using cnMaestro cloud, and this has been tested on both switch firmware version 4.5-r3 as well as 5.0.2-r4.

Anyone else seen this, and/or do Cambium Networks even know about this?

I would appreciate feedback!

2

Hello,

  1. The switch has a recovery mechanism to automatically restore the connection with cnMaestro. In the event the new configuration causes the switch to lose connection to cnmaestro, the switch will reboot itself in 15 mins using the previous known working configuration.

  2. The speed of the fiber port is automatically set based on the transceiver type encoded in the transceiver. On the EX2028-P, a SFP or SFP+ transceiver can be used on its 4 fiber ports. When a SFP is inserted the fiber port will be configured for 1G. Similarly, a SFP+ transceiver will be set to 10G. That said, there have been reports that switch does not set the ‘wrong’ speed, and this is because the transceiver does not follow the MSA standard by having the wrong port type in its EEPROM. On cnMatrix, you can run the CLI command ‘show interface transceiver’ to display various information inside the transceiver, including the transceiver type.

  3. In some cases, it is required to manually configure the port speed of the fiber port. For example, a SFP+ transceiver is inserted but the other connection end has SFP transceiver. In this case you need to force the speed of the SFP+ to 1G.

  4. MTU size and port speed are not tied together. It looks like a software defect in cnMaestro. We will reproduce and fix it. Please raise a support ticket for this issue.

Thanks!

3

Thanks TamN!

  1. The problem with this, is that you then get the error “not in sync” on cnMaestro, and it’s hard to find what part of the config we tried to push was the cause of the switch going into recovery…
    Then you cannot try to push the config again,as the switch willl go into recovery once again - and you have this “chain of events” going like a loop.

2/3. That is exactly how I thought it was.

4: I’ll go ahead and raise a support ticket referring to this thread.

Thanks

4

Hi @rambern

By default, the EX2028-P switch’s fiber ports (EX0/1 - 0/4) are set to a speed of 10Gbps in cnMaestro. I believe you are using 1Gig SFP transceivers on these ports; and when you make any changes to the switch ports and sync them with cnMaestro, it will push the default 10Gbps speed to the switchport configuration. Since your SFP transceivers only support 1Gbps, this will cause the device to go offline. To prevent this, you need to manually set the fiber port speeds to 1Gbps in cnMaestro before syncing the configuration.

5

Thanks for your response. If I don’t set the speed manually in cnMaestro at all, the 1Gig SFP works fine (even though the cnMaestro is set to 10Gig by default). BUT, if I ever manually change the speed to 1Gig in cnMaestro, it seems like the sfp won’t work. Then the problem arises, as setting it back to 10Gig in cnMaestro will have this as an “input” in the config pushed (speed 10000). If you never change the port speed in cnMaestro, it will stay at “auto” (the config does not include “speed 10000” even if the cnMaestro setting is set default to 10Gig). But if you at any point have tried setting the port manually, you will not be able to “remove” the speed setting from the cnMaestro config. It will allways be set to either 10Gig or 1Gig. Only way to set it back to auto, is to make a whole new switch group (we use switch groups) to put the switch in, and reset the switch to default before pushing the config from the new switch group.

6

In my lab, I have two switches connected to each other using the 1Gig SFP modules on the 10 Gbps fiber ports of the switches.
When I look at the cnMaestro - Ports config, the default port speed is set to 10 Gbps by default. Both the switches work fine with this default configuration, as you observed.
Now, if I am changing this port speed to 1 Gbps in cnMaestro for the first switch and sync, it still works.
And, in the second switch, if I change the port speed to 1 Gbps, and sync, it still works.
The problem persists only when you sync the switch configuration with the port speed set as 10Gbps, because the SFP inserted supports only 1 Gig.

7

You are correct! It’s an oversight in cnmaestro. Once the speed is manually changed, there is no GUI option to change the speed back to auto-detect mode again. The auto-detect mode would allow the switch to set the port speed to the speed dictated by the transceiver types SFP vs. SFP+. I will raise an internal ticket for this to be addressed in the next cnMaestro release. In the meantime, you can enter the CLI command in the user-defined overrides to set the auto-detect mode. Here is the example of the CLI command:
config terminal
interface ex 0/1
speed auto-detect

In case you are not familiar with the user-defined overrides, please raise a ticket and I will assist you.

1 Like
8

Thanks for your reply!
My sfp will unfortunatly not work if it is manually set to 1Gig in cnMaestro (at least not consistent) - I don’t know why, but it’s the case. I see TamN has sent another answer below, which will resolve the issue for now.

9

Thanks - we will go ahead and set the overrides where needed in the meantime!

Then we will have a fix in cnMaestro in another release - great!

10

I instantly cringed when I read this post. As a rule, I try to never make changes on Fridays (Even Thursdays if I can help it) that affect critical devices. If its going to break, it will break on a friday.

11

Agreed!

We have allways had that rule too, but the last few years we’ve upgraded our network a great bit, and everything has been running very smooth - upgrades and configchanges seems to allways “work” as expected. This has lead us not to be afraid of changing things on fridays.

Most certainly - we will now go back to: “No configuration or upgrades to the backbone or ap’s on fridays!” :rofl: :joy: :rofl:

1 Like
12

6F6YdLo

1 Like
13

Good plan. I have broken plenty of things on a friday and fixed things others broke. It always seems to be something that was a low risk lol

1 Like