We're using a pair of ePMP 1000's to backhaul between two towers. Three weeks ago we had a brutal ice storm that brought power outages acrossed the network. The master side of this ePMP link lost power, and the battery backup failed to pick up in between the power outage, and the generator starting.
After the generator came back online the radio came back on, but instead of running 2.4.3 firmware it was running 2.3.4. It reported that 2.4.3 was the inactive bank. Not only that, but the radio had completely defaulted itself to factory settings.
I re-uploaded 2.4.3, and the configuration file. All seemed well for about 8 hours, at which point it flipped back to 2.3.4. Three weeks later, it has defaulted to the older firmware at least 20 times... Other ePMP1000 radios that were plugged into the same outlet are 100% fine. I tried a new power supply, and I have verified there's no issues with the cable.
I tried 2.6, and had no luck.
I'm very seriously thinking that the radio has become a paperweight. Prior to replacing it (it's quite a climb, and the tower is fairly icy) I was curious as to whether anyone may have some useful insight.
After thinkign about it I was remembering the problem with the customer radios wrong.
We have had two APs and several customer radios do this. The last time was an AP that became unreachable. We power cycled it and when it came back up we were able to access it. It had retained all of its settings but the firmware had reverted to v1.something.
The one before that was a customer radio. The customer called and reported he had no internet. We were able to log into his radio and everything looked fine other than no address was being DHCPed out to his router. So we rebooted the radio and... it never came back up. We drove out to the customers house and when we accessed the radio all of the settings were correct but it had reverted to v1.something firmware.
There was at least two other occasions we had this happen with APs and there has been 3 or 4 other customer radios we had to truck roll to re-upgrade the firmware
.
I was able to update the firmware and get it back up and running in every instance. The problem is it keeps defaulting to this old firmware every 24-48 hours. When it does so our customers suffer a break in any data streams for a minute, and then the radio shuts off GPS sync and starts generating its own and we start to self interfere really bad.
I've tried several versions of firmware and had no luck.
Update to 2.6 firmware. Once it's running 2.6, update to 2.6 firmware again and it will flash the secondary (inactive) bank. After that, if it fails the primary and goes to secondary bank it will still be the same version, compatible with the existing config settings.
This doesn't fix the problem, it just makes it so that when the problem recurs it will have less severe impact.
j
Definitely a good idea, one that I feel foolish for overlooking.
It still leaves a bit of a problem though.
After uploading 2.6 into the active, and inactive banks the radio seems to be okay...... Is it possible the older firmware that was moving into the active bank was corrupted and somehow causing the issue?
the AP will move to the inactive bank when it fails to boot multiple times, if you had power flickering on and off then you've got the device thinking it has a problem and changes firmware. when upgrading firmware, be sure to update it twice to avoid strange issues happening like that.
also many want to disable the ability to factory reset via power sequence. if your radio has power issues that coincides with the power procedure you'll factory default. never fun.
The fist time it defaulted to the inactive back, yes. we had power issues. However for three weeks after that power was fine, the radio had a new PSU and it was plugged into the same battery backup our other ePMP 1000 radios are, and it was still defualting to the inactive bank for no apparent reason. Every 8~16 hours.