The issue has been reproduced locally in the lab and the fix is coming in 4.7.0.1.
4.7.0.1 Beta will be released this week.
Thank you.
2 Likes
I have been told by support that downgrading below 4.7.0 is not possible by design. Cambium does not want users to downgrade below 4.7.0 after upgrading. Since all of our 180s, 190s and 200s downgraded just fine and only about 50% of the F300 will not downgrade, that software block is not working well.
None the less, we are not able to wait for 4.7.0.1 and do not run beta software in production so we need to find a solution to downgrade the radios that wouldnāt. I canāt afford any more truck rolls and our customer reputation is suffering from all the issues we have experienced with 4.7.0.
Weāve already tried this and when the radio is in this state, cnMaestro cannot get a TSF dump. After multiple attempts the radio will stop responding to even cnMaestro. We were able to get a log dump, which is how we got the data previously posted.
1 Like
The only thing that seems to (barley) still work is a reboot requestā¦ so thatās what weāve been doing. Once they get in this state, nothing else seems to workā¦ and weāre flirting with disaster.
We have downgraded to 4.7-RC64 from 4.7, this was done because 4.7 has stabilty issues that take time to manifest. SMs just up and stop allowing any login, showing offline in cnMaestro and can not be remotely rebooted. This is most prevalent on F180 SMs but we have seen more than a few F300 SMs do this.
APs do not seem to have the issue but that could be for another reason that we just do not see.
All of our towers are DC direct powered with Packetflux sync Injectors and sync to the Packetflux except for the e3kL radios which get sync from the onboard GPS. Just incase this is important.
Iām losing my mind now. We downgraded everything we could back to 4.6.2. There are ~20 radios that gave us the error that they would not downgrade below 4.6.4 so they are stuck on 4.7.0. Last night, we rebooted a router in our system after a firmware upgrade. That simple act factory reset every 4.7.0 radio that was getting a DHCP management IP address from that router.
I have a list of customers now off line and power cycling the F300 client radio does not bring the radio back online. We are truck rolling for a dozen more customers on a weekend because of this firmware.
We have seen this factory reset issue before on 180 client radios under 4.7.0 and now it hit on on a bunch of F300 client radios. I honestly do not think we will ever upgrade ePMP beyond 4.6.2. I can not afford to lose any more customers as a result of this firmware. Weāve been using ePMP for 7 years now and never had issues like this but there is something very wrong in 4.7.0
2 Likes
Dumb question, but do you have reset via power enabled?
I would also review the commands used (especially if CnMaestro was involved) to see if a factory reset command was send to the multicast addresses.
We turned off reset by power years ago on all radios and it is off on our default template.
We did not issue any commands to the radio. All we did was reboot a Mikrotik CCR2216 router that happened to be the DHCP server for the management IPs for these radios. That is it. All 8 radios reset at the same moment the router came back online. Only radios that were running 4.7.0 reset.
This is not the first time rebooting the management DHCP server has done this to 4.7.0 radios.
Our on-prem cnMaestro does not have any onboarding enabled or automatic software changes enabled. All 8 radios that reset were already onboarded (years ago) and were running normally at the time of the reset.
Iāve been out to two sites so far to re-program the radios.
2 Likes
Third radio we visited had switched to 4.6.4 from the 4.7.0-STA firmware that was running on it. Since this is an STA AP, the SM could no longer connect since 4.6.4 is not STA.
What makes no sense is we never uploaded 4.6.4 to that radio. It is in cnMaestro but is not set as the preferred firmware and is not set to auto upgrade. the preferred firmware is 4.6.2-STA and we have images for all radio models we use in cnMaestro.
4.7.0 keeps doing things we can not explain. Of the 8, so far 2 did a factory reset and 1 did a revert to firmware we never pushed to it - all at the same exact time and all synchronized with a reboot of the management IP router.
Iām not new to running ePMP or a WISP and I canāt for the life of me explain any of this but weāve never had a single issue like this with any firmware prior to 4.7.0
2 Likes
Sorry, I did not get a notification of your other post.
IF cnmaestro had a update pending then this would make sense, but this is sounding like either a hold over in 4.7-STA or something funny is interacting and forcing a web update, which ignores your on-prem server.
I wish we could disable the web update in config though, it is too easy to bit flip a register thus gain an unintended web update.
What improvements exactly?
Oops! That page is private.
Hi Chad have you already tried the new 4.7.0.1-RC6? what were the results? did your issues were fixed?
We are not trying release candidates. We spent considerable time getting everything back to 4.6.1 and I think that is where the network will likely stay for the rest of this hardwareās life.
Does anyone have some field data using 4.7.0.1-RC6
There is a known memory leak in 4.7.0 for the DPI feature, this is not yet confirmed in 4.7.1rc6 but there is issues surrounding DPI in 4.7.1rc6.
We are still lab testing 4.7.1rc6 but it is not going well, odd issues in the lab mean major issues in the field. Funny thing is that we are having a hard time quantifying these issues as they come and go fast enough that we are not seeing the cause and the syslogs are as helpful as ever! Some more verbosity would be nice!
Hi all, so to roll back anything from 4.7 is not possible? I have a few weird things happening with some subscribers but do not know if its software related.