SMs did not come back online after upgrade from 4.7.0.1 to 4.7.0.1STA version

Products ePMP
1

Hey all,

This morning we ran some upgrades on one of our sites from the regular 4.7.0.1 version to 4.7.0.1STA. We have done this upgrade on over a thousand devices already, and have not had any issues. However, this morning there were 10 SMs that did not come back online…

We had the customers power cycle their equipment, and it did not bring them back up. We just sent out a tech to one of them, and while he was getting dhcp from the SM properly (so it was not defaulted) he was unable to get into the radio on either the IP we assign them or via the backdoor IP. We had him perform a factory reset on the radio, and he was unable to get to it via the backdoor OR its default 192 address. According to the tech, the SM at their location is boot looping every 15 seconds.

These are all Force 200 5GHz radios. I wouldn’t have been too surprised if a bunch of elevated units had decided to die on us, but I have never had F200s behave like this after a software upgrade, especially when the only difference in the firmware is whether or not it is the STA version. What’s really interesting is that the firmware update job reports like 3 different statuses for these radios, it looks like some got the upgrade successfully and some did not, but none of them came back afterwards.

We already tried rebooting the AP, reverting the firmware on the AP to non-STA 4.7.0.1, changing frequencies on the AP, just about everything we could think of. It really seems to just be an issue with the upgrade procedure on the SMs themselves. At this point we are going to have to replace all of these units, some of which are in trees (so we will have to eat the climber cost as well…)

Any ideas, anybody had this happen before and managed to get the SM back to working order? It would be really crappy to have to replace 10 radios because of a routine software upgrade…

2

What platform did you use to do these updates? cnMaestro on premise or CNUT? We used CNUT just yesterday to perform a large round of updates from 4.7 to 4.7.0.1-STA… mix of a couple hundred AC and N radios, and all of them completed without issue. We did notice that CNUT rebooted many of the radios before applying the update.

I think performing a reboot of ePMP radios just before an firmware push is probably the number 1 most important thing to do at this point judging from mine and other’s experiences posted in the forums here.

3

I would really appreciate it if you could send me the MSNs/ESNs of the affected radios in DM.
We are looking into the issue with the highest priority in scope of the support ticket.

Thank you.

4

Hi Fedor,

I have shared the serials of the radios in the ticket. Appreciate your help, we are trying to get a remote session for Cambium to access the radios locally but there is no cell service in most of this area and the equipment is in a tree so we won’t be able to get a signal to our tower from the ground.

5

We used cnMaestro in this instance. I think you are right, rebooting all the SMs at the very least for good luck might be the best plan moving forward. It is just odd because over 10k+ upgrades we have never had simultaneous failures like this on F200.