ePMP Software Release 4.7 is now available

is the community’s name at least 8 characters long?

10 characters and both are different from each other.

By default, after upgrade the radio presents you with a user dialog box and promptly disables snmp regardless of the configuration. Most annoying.

Diabling unused users is normal, but why diable a properly configured option? cant the firmware check the config status for this and pop up a dialog box?
Better yet, add proper security measurs such as limiting whoch IP addresses can talk to the radio and adding a login failure blocker like fail2ban?

1 Like

That’s weird, it happend to me ONLY when the community was less than 8 chars or default one.
I just upgraded ~200 SM/AP to 4.7

our experience seems to be different here, but this still does not change the fact that proper security measures that are available for the version of linux these things run need to be implemented instead of forcing some third party policy on us operators. A simple nag-box and refusal to save config until these important settings are changed is sufficient to make sure Cambium is doing the basics of enforcment. Adding proper security abilities to the devices is more important than deciding what my config should be and changing it without my authorization, again a simple pop-up box with a warning and an option to keep the setting enabled and check if the communities are default and over the minimum character limit is all that was/is needed.

I know I am more than bitter about this and will be for a long time. Security isnt just disabling a service, its a whole system!

hi all, in CNmastro it still says 4.6.1 is recommended. Will this be changed soon? ive upgraded a few links to test 4.7…

1 Like

I’m surprised I’m not seeing anyone else with the issues we have seen. We have ~300 radios that were upgraded to 4.7.0STA - which I understand is the same firmware as 4.7.0 but with the additional 5.9 frequency.

Since that upgrade, we have had issues all over the place. None of these issues existed before 4.7.0:

  • Device Name will not remain through a power cycle. We can set the device name, it forces a reboot, device name stays but as soon as that radio is hard power cycled, the device name reverts back to what it was pre 4.7.0. I can duplicate this on 100% of our radios (Force 180, 190, 200, 300-16, 300-25). Have not tested on an AP yet.

  • GPS coordinates do not remain through a power cycle. Same behavior as Device Name. They will remain through reboot commands but as soon as you hard power cycle the radio, the GPS reverts to the data that was there pre 4.7.0 upgrade. Can reproduce on 100% of my client radios.

  • Radios crash with various commands pushed from cnMaestro. Sometimes a reboot command will crash them, sometimes a data vlan ID change will crash them (the two things we have tested). When they crash, they are up but will not associate with a tower. We can log in locally, they act weird but are accessible and will not associate until they are hard rebooted with a power cycle. The reboot button in the GUI never initiates a reboot when in this state. We have seen this on Force 180, 190, 200 and various 300-x client radios.

  • So far, 4 client radios have reverted to a factory default state when they were sent a vlan ID change command from cnMaestro. We logged in locally (after a truck roll) and it was totally reset to factory. We have the power sequence factory reset option turned off on all radios.

  • Radios lose the ability to be logged into. We can ping the IP and the IP shows up in the AP/Monitor list but trying to access it from the GUI fails. It’s like there is nothing at that IP. This is only fixed with a hard power cycle. Then it works again. See this on all SM models. No pattern we can tell when they decide to hang.

  • Every single ePMP 1000 AP we have will stop responding to SNMP requests once the airtime utilization creeps up. I can’t tell you the threshold yet but in the evening, our SNMP status board is red on most of our 1000 APs showing SNMP is no longer responding. Ping times to the AP remain unchanged. We do not see this on 2000 or 3000 series APs but we do see it on Force 180 radios in AP mode.

As an ePMP customer for 7 years, this is the worst upgrade we’ve been through. I am ready to revert everything back to a previous version but I am afraid of how many radios we will lose control of in that process and how many more truck rolls we will have to make.

Here is our vlan ID change code that is crashing things. You can see it is not doing anything special:

{
    "device_props": {
        "dataVLANEnable":	"1",
	"dataVLANVID":	"301",
        "dataVLANVP":	 "6"

    }
}
3 Likes

We had to stop deployment as well for reasons stated above. Sticking /w 4.6.1 for now.

1 Like

I am going to recommend we downgrade all sites to 4.6.1 as well. This current version is costing us customer churn and way too much support time. So far, 5 truck rolls and two cancelled customers as a result of problems directly related to the upgrade to 4.7.0.

1 Like

Just lost another radio with a simple data VLAN change. Never came back online after saving that one change. Yet another truck roll. I am terrified how many radios (and customers) we are going to lose in the downgrade process. I wish we had any idea what was causing this.

1 Like

We are going to have to downgrade as well. We didn’t see any problems with 4.7 for quite a while, but more & more stations stop passing traffic, we can’t log into gui - but thankfully can be rebooted via cnMaestro.

Also unfortunately means we can’t get any support file downloaded from the radios.

Ticket has been opened, no reply yet.

We’ve been working with Cambium support on this and we do have a clue, it may have something to do with the SNMPd having issues. We also only seen this issue with 802.11n clients. We were able to grab some logs from a wedged SM that was still connected to cnMaestro. Our current strategy is to use cnMaestro to reboot all of our ePMP SM’s late at night on a semi-weekly basis. This issue doesn’t appear to happen until past 7 days of uptime, and it only happens to a small % of our SM’s.

Here’s a dump of those logs:

Jan 3 20:06:24 Client CN7744 kernel: [2019276.450000] SM disassociated from AP[00:04:56:22:29:bf] F=5240 11naht20. Reason: 32 (NO ALLOCATION ON AP)
Jan 3 20:07:05 Client CN7744 kernel: [2019317.240000] SM associated with AP[00:04:56:22:29:bf]
Jan 3 20:07:07 Client CN7744 miniupnpd[3126]: ioctl(s, SIOCGIFADDR, …): Cannot assign requested address
Jan 3 20:07:07 Client CN7744 miniupnpd[3126]: Failed to get IP for interface ath0
Jan 12 14:11:15 Client CN7744 kernel: [2775567.930000] SM disassociated from AP[00:04:56:22:29:bf] F=5240 11naht20. Reason: 33 (GPFs MISS)
Jan 12 16:10:09 Client CN7744 kernel: [2782701.510000] SM associated with AP[00:04:56:22:29:bf]
Jan 12 16:10:11 Client CN7744 miniupnpd[3126]: ioctl(s, SIOCGIFADDR, …): Cannot assign requested address
Jan 12 16:10:11 Client CN7744 miniupnpd[3126]: Failed to get IP for interface ath0
Feb 7 12:32:56 Client CN7744 snmpd[7593]: DFS status: N/A
Feb 8 09:24:01 Client CN7744 snmpd: Watchdog:Abnormal SNMPd stop occured, restarting…
Feb 8 10:56:02 Client CN7744 snmpd: Watchdog:Abnormal SNMPd stop occured, restarting…
Feb 8 10:56:03 Client CN7744 snmpd[7661]: DFS status: N/A
Feb 8 11:08:01 Client CN7744 snmpd: Watchdog:Abnormal SNMPd stop occured, restarting…
Feb 8 11:08:02 Client CN7744 snmpd[9997]: DFS status: N/A
Feb 8 11:32:01 Client CN7744 snmpd: Watchdog:Abnormal SNMPd stop occured, restarting…
Feb 8 11:34:01 Client CN7744 snmpd: Watchdog:Abnormal SNMPd stop occured, restarting…
Feb 8 11:34:02 Client CN7744 snmpd[12503]: DFS status: N/A
Feb 8 12:08:01 Client CN7744 snmpd: Watchdog:Abnormal SNMPd stop occured, restarting…
Feb 8 12:08:03 Client CN7744 snmpd[16077]: DFS status: N/A
Feb 8 19:32:02 Client CN7744 snmpd: Watchdog:Abnormal SNMPd stop occured, restarting…
Feb 9 05:26:01 Client CN7744 snmpd: Watchdog:Abnormal SNMPd stop occured, restarting…

1 Like

I may try this tonight, rather than downgrading everything.

Okay, cambium replied to my ticket, said it is a ‘strange issue, and they have not come across any other account’ such as this. :person_shrugging: So, they didn’t make it sound like the have any clue.

They also say they may need access to a device when the problem is present, I’m not sure how to accommodate that, since we can’t SSH or HTTP into them at that time. Plus of course, the customer is without internet, So The priority is usually rebooting the SM and getting them back online. However, we may do a truck roll and see if we can access it from their Ethernet side of things.

For us, I’m pretty sure it has been a mix of ePMP 1000 / Force 200 as well as Force 300. I will triple check with the staff. Virtually all of our 1000/200 clients are 2.4 GHz, almost all of our 5ghz is 3000/300, but there are a few straggling 1000/180/200s in 5ghz out there - not very many. But, I will triple check and make sure. :+1:

1 Like

Maybe reference my ticket: Request #340393 ePMP legacy N clients stop responding on 4.7

1 Like

We mass upgraded our network of about 1500 devices to 4.7
no issues at the beginning but we noticed a lot of disconnected routers (cpe are in bridge mode).
They dont get dhcp anymore.
the cpe are not getting dhcp renewed, but they still answer on their old IP.
we connect them via ssh and reboot them.
downgraded to 4.6.1 and the issue disappeared.

we have noticed that some devices cannot downgrade below 4.6.4 … but we dont have any 4.6.4 around, so they stay on 4.7

We had turned off SNMP on all our SMs but the problem continues long past that so that doesn’t point to an SNMP issue unless turning it off doesn’t really turn it off.

We just lost another SM. All it did was renew it’s management DHCP lease. That was enough to knock it offline and require a power cycle.

It is not a “small number” of devices. It is 100% our network (around 300 devices). So far, we’ve lost around a dozen customers as a result of this. They got tired of their internet just stopping randomly and having to constantly reboot their gear. That did not instill confidence in them with their ISP and they switched.

So there must be something about your configuration that is triggering this issue more than ours. I’d encourage you to work with Cambium support to help them find the issue. I’m currently working with them and I have a debug firmware that they built (4.7.0.1-RC5) that I’ll be deploying tonight in an effort to try to catch the issue.

Please open a ticket and reference ticket #340393

@Fedor do you have any advice for @Au_Wireless?

1 Like

@Au_Wireless,

If you can provide remote access to device when the issue is seen that would help us a lot.
Unfortunately we haven’t succeeded in the issue replication in the lab so far and can rely on customers’ help only.
We do have private firmware with enabled debug prepared that I can share with you, but even having access to device with original 4.7 firmware would be very useful.

Thank you.

We also downgrade all SMs and ePMP2000 APs to 4.6.2, only leaving 4.7.0 on ePMP3000s. Our technicians call 4.7 a total disaster :slight_smile:

We’ve lived quite happily for more than a year with 4.6.2 (few long happening issues aside). 4.7 broke lots of things… SMs got more expensive, software got more unstable snd buggy. Such are the times I suppose :man_shrugging:

@Fedor As much as I’d love to help on this issue, we are moving as many radios to 4.6.1 as fast as possible. This has been such a disaster for us and has angered so many customers, we are moving everything we can as fast as we can. I do not want to leave any devices on 4.7.0 or any beta versions of 4.7 for any longer than absolutely necessary. I am happy to email you configs before we downgrade but even that makes me nervous. It seems every time we touch a radio in any way, it results in a truck roll.