3.3 to 3.5 kills radios?

What's the  deal with this?  I've killed 4 modules now and we're going to need to remove  them from the tower to find out what went wrong.

Cambium is a frustrating company to root for.

Does your switch have a watch dog pinging them ? They don’t respond for a while updating and that certainly will cause a problem.

We upgraded about 200 aps to 3.5 without an issue, mix of 3.3 and 2.6.2.1


These are epmp 2000 radios with smart antennas if that makes a difference.  The updates were done via gui since I can't seem to make  cnut work with these.

I also tried upgrading the force 200 at my house which is currently on 3.4, but I get a message telling me to reboot first.  If I reboot first and try again it repeats the same message telling me to reboot.

Any ideas?  I'm scared to death to upgrade anything else on this network and may be stuck with old firmware forever.  The ubiquitis are looking pretty good again.  I'm not even gonna try that elevate crap, I'll just keep both AP's up until one beats the other.  So far it's a tie I guess and when compared to UBNT that is not a good thing.

Watchdog was turned off.  They complete the update and ask for a reboot, they never return after the reboot.  The switch is still showing power being drawn, but it's not pingable by it's IP or the default IP.

The first 3 happened the other day after a tower got destroyed by lightning.  We installed 3 new radios which came with 3.3 out of the box.  I ran an update on all 3 and killed them.  We then took 3 other new ones out there and left them on 3.3.  Today I decided to try one that had no customers on it and same thing.

IDK.  Too close to Friday to deal with this nonsense.  This should be basic stuff that just works.

More info in case it helps Cambium.  I can arp ping it and it's responding on 169.254.1.1, along with every other ap nearby.  So it's kinda alive.  Tomorrow I'll have someone go defualt the thing and see what happens.  I'll try the same on the first 3 that went down as well.  Fun stuff.

I'm also noticing in my neighbors list from an MT device that the modules with 3.3 are showing version  2.6.32.27 whereas the devices that upgraded successfully are showing 3.5.0.  Is it possible these modules are lying to me about which version is actually installed?  Again these came right out of the box this way.  Maybe I'm jumping to many versions on these?

Are those radios connected to a Netonix switch?  If so, is STP enabled on the AP?  I'll explain if your answers are yes.

Yes and yes.

Take a read through this if you have not already:

http://community.cambiumnetworks.com/t5/ePMP-2000-and-1000/Lost-access-to-2000-AP-during-upgrade-to-3-5/m-p/77332#M11545

I am getting convinced there is an issue with the 2000 AP hardware, firmware 3.5, Netonix switches when the AP has STP turned on. I have three 2000 APs and all three of them I lost control of on the Ethernet side when I tried to upgrade to 3.5.  All three are in Netonix switches and all three use STP.  As a result, I have not upgraded those APs.

2 Likes

Interesting.  

I havnt read through the entire thing yet, but decided to turn off STP in the netonix and then update the radios.  I stepped them first to 3.4.1 and then to 3.5 and all updated fine.  I then turned STP back on in the netonix and lost contact with all of them until turning STP off again in the netonix.  I then turned STP off in the radios and on in the netonix and all is well.

Weird, I can live without stp in the radios if thats the fix.

Thanks for the info!

1 Like

@jmay wrote:

I'm also noticing in my neighbors list from an MT device that the modules with 3.3 are showing version  2.6.32.27 whereas the devices that upgraded successfully are showing 3.5.0.  Is it possible these modules are lying to me about which version is actually installed?  Again these came right out of the box this way.  Maybe I'm jumping to many versions on these?


This is a change where the radios are now reporting the ePMP firmware version instead of the embedded linux kernel version.  When they first started sending out LLDP several releases ago they reported the kernel version (and the platform was identified as "linux" as well) but recently versions identify as the firmware version and platform is something like "5G Force 200 (FCC)".

j

2 Likes

Now that we know exactly where the issue is, hopefully Cambium can reproduce it and then fix it...

1 Like

Hi all,

Thank you for investigation on your side.
Previously we have investigated issue with Chadwick, unfortunately we didn’t know it is related to STP configuration on switch and were not able to reproduce it.
Hope we will reproduce issue in the Lab now and fix issue asap.

I will keep you updated.

Thank you.

1 Like

Issue has been reproduced with 3.5 ePMP FW on Netonix Switch WS-12-250-AC.

We will investigate it deeper and revert back to you.

For now it is not clear whether issue is related to ePMP FW or Netonix Switch FW.

Thnak you.

4 Likes

Just to add to this, having stp enabled can cause some serious problems network wide.  I've had a region thats been acting up every couple of days and rebooting several switches brings everything back to life.  I have a combo of mikrotiks used as switches and netonix.  If I disable stp on the epmps my problems appear to go away.  This is fine for me, I really don't need stp on aps, just thought id pass that info along.


@jmay wrote:

Just to add to this, having stp enabled can cause some serious problems network wide.  I've had a region thats been acting up every couple of days and rebooting several switches brings everything back to life.  I have a combo of mikrotiks used as switches and netonix.  If I disable stp on the epmps my problems appear to go away.  This is fine for me, I really don't need stp on aps, just thought id pass that info along.


Thank you for your input.

We are working now on issue localization and fixing.

Thank you.


@jmay wrote:

...

I also tried upgrading the force 200 at my house which is currently on 3.4, but I get a message telling me to reboot first.  If I reboot first and try again it repeats the same message telling me to reboot.

 ...




I'm having this same issue on two of my ePMP 1000s. Did you get a resolution on this?