450 APs and SMs Randomly Going "Offline"

I'm seeing PMP 450 APs and SMs on 15.0.2 and 15.0.2.1 randomly going "Offline" once onboarded.  When this happens,  the DNS tool will successfully resolve any legitimate url including cambiumnetworks.com, but will not resolve cloud.cambiumnetworks.com.  Furthermore, when trying to resolve cloud.cambiumnetworks.com none of the Statistics/DNS Statistics page counters increment.  It's as if the unit isn't trying to resolve the cloud url at all.

The only way I've been able to get the units "online" once again is to reboot.  However, in time, other units will fall off.  

Then it's a matter of "Rinse and Repeat"

Anyone else noticing this?

Are you facing this problem in Cloud or On-Premises version?
If cloud can you please send me your cambiumID and list of radio MAC which are having issues via private message.

I am using the cloud Version 1.3.0-r4

I emailed you as requested.

Looking forward to hearing from you soon.

Are you facing any network down time ? We have a known issue when the network is down for long time DNS resolution for cloud.cambiumnetworks.com ( radio will be keep trying to reconnect to cnMaestro even during downtime)  is not working  and other URLS working. We are investigating on this. From the symptoms i believe you also face the same issue. Once reboot everything is back to normal ?

I don't know if it is the same exactly, but I have had a few SMs dissappear when I clicked "approve". They would show up again after a short while with no information (like I hadn't input location, etc). I gave up after 4-5 times of this happening only for the SM to onboard the next morning with no issue. 

Also some mornings (like this one) I find cnMaestro showing me half the netowrk down. All I have to do is click on each one in the tree (some more than once) and they all show as normal again.

Cloud Version 1.3.0-r4

Thanks for the reply not 1337, but this is not the same problem.  My units remain onboarded and visible, but "offline".  The units will subsequently resolve any (legit) url other than cloud.cambiumnetworks.com.  Obviously, without proper resolution, the unit remains "offline" to cnMaestro.  The unit continues to work otherwise.  The management interface, likewise, remains active and usable.  The only way to return the unit to an "online" condition is to reboot the unit.  This is randomly happening with both 450 APs and SMs.

Frankly, I'm surprised that no one else has responded as I'm being told this is a "known" DNS issue.

The problem I am having is that if they go offline, they will not reconnect.  An example would be an AP reset, the AP will not reconnect to cnMaestro (Cloud) and none of the SMs will either.   I have to go in and reboot everyone of them.  Sometimes like right now rebooting them doesn't even do the trick.  The 450s are the only product line that does this.  We have ePMP and cnPilot and they all work great.

Once the unit goes "offline" (Looses contact with cnMaestro):

1) Perform a DNS test to see if the unit resolves cloud.cambium.networks.com - Mine don't

2) Perform a DNS test to see if the unit resolves cambium.networks.com - Mine always do

 

If an AP goes "offline" it doesn't have an effect on the SMs.  An "online" SM remains "online"

If an AP returns to an "online" condition it will not affect the "offline" SMs.  They remain "offline"

If a SM goes "offline" it must be individually rebooted.  An AP reboot will not retore the SM to "Online" status

A reboot of the affected device has not yet failed to reconnect it to cnMaestro 

Just to drive the point home, once either APs or SMs drop to an "offline" condition, they will resolve any url other than cloud.cambiumnetworks.com.

1 Like

Now I seem to be experiencing this.  Randomly the SM (450d)  loses it's DNS and the Router (200p) follows with it.  At first (a few weeks ago) rebootting the affected systems (SM and Router) seemed to resolve it .  Now it's chronic and hard coding the DNS (in the SM and router) seems to fix it ....sometimes.  Very frustrated trying to find the issue here.  

Hi ColdBlueFire,

How the SM and R200P are getting the IP Address . Are they getting through DHCP or static IP is assigned.

If DHCP is used then the DHCP server should give the DNS address also to the devices. If static IP is used then the DNS is needs to be manually configured on the devices .

DNS is mandatory as the devices needs to connect to the cnMAestro Cloud server which is located in the Amazon AWS.

If you are using on Premises version of cnMaestro then the DNS server is not required as far as the devices can communicate with the cnMAestro on Premises server.

Which software version is used in  PMP 450 and cnPilot R200P . Also , please send me the config files of the both to my mail address krishna.reddum@cambiumnetworks.com . I will check the configs to see if they are fine .

If at all going forward you face any connectivity issues with cnMaestro in the device GUI you can go to Tools->DNS test or Ping test and check wheter you are able to reach the cloud.cambiumnetworks.com

Also you can check the Logs->Device Agent Logs to know what exactly is the issue in connecting with the cnMAestro server. Before checking the logs please turn on the debug mode in Configuration->cnMaestro page for the cnMaestro Agent Debug Log Level option

Please refer to the following KB article for more details

http://community.cambiumnetworks.com/t5/PMP-Getting-Started/On-boarding-the-PMP-Devices-to-cnMaestro-Cloud-or-On-Premises/m-p/59728#M53

1 Like

Luis Mondragon wrote:

I'm seeing PMP 450 APs and SMs on 15.0.2 and 15.0.2.1 randomly going "Offline" once onboarded.  

When this happens,  the DNS tool will successfully resolve any legitimate url including cambiumnetworks.com, but will not resolve cloud.cambiumnetworks.com.  Furthermore, when trying to resolve cloud.cambiumnetworks.com none of the Statistics/DNS Statistics page counters increment.  It's as if the unit isn't trying to resolve the cloud url at all.

The only way I've been able to get the units "online" once again is to reboot.  However, in time, other units will fall off.  

Then it's a matter of "Rinse and Repeat"

Anyone else noticing this?  ________________________________________________________________________________________

Perform the acions in bold and compare the results - monitor the counters on the Statistics/DNS Statistics page.  This pertains to PMP 450 AP & SM - Your Milage May Vary

It's as if the units keep some value in cache and therefore will not even perform a DNS query for cloud.cambiumnetworks.com, as evidenced by the lack of counter incrementation.

I finally gave up on the cloud version and moved to OnPremise - 

Happy Hunting

1 Like

It would appear that this issue is still not resolved.

The only solution seems to be to change from Cloud hosted server to On-premises server.  Is this correct?

Hi Luis,

The issue mentioned by you was a device platform issue wtih DNS resolution and was fixed in PMP build 15.1.5 and above . 

Please upgrade to this build and everything works fine in cnMaestro cloud servers.

Thanks,

KR.

Hello

I think we have a same problem. We are using CnMaestro Version 1.6.2-r11 with devices: PMP450 and 450i - sw version: 15.1.2.

The software upgrade from 15.1.2 to 15.1.5 could solve this problem?

Thank you!

Br

István

Since updating all of our 450's to 16.0.1, we haven't had any AP's drop off of cnMaestro cloud version yet... it's been about 22 days now.

I’m having this problem even on version 20.3.1.2. The ap’s (one 450i and one 450 micropop) cannot resolve cloud.cambiumnetworks.com. When I log into the ap’s, they suddenly are able to resolve dns, and pops back online in cnMaestro… It seems very strange… Anyone able to give me some input?