We have a freshly provisioned AP that has 7 Elevated XW UBNT radios hanging off it that's got a locked up webUI:
No part of the UI loads, and when you try to click on one of the categories, this happens;
Both the AP and all connected SMs are on v3.3. There's no output for debug crashlog on the AP device... As this is an in-use device, I'm forced to reboot it to resolve the issue (fortunately the reboot button did work and appears to have fixed this) so I don't know what other info I might be able to retrieve.
I neglected to dump the syslog prior to reboot, my apologies...
Understand that this is a live network and you cannot have it down. In case of this happening again, it may help to get syslog output using a syslog server, if that is possible. Crashlog from CLI will also help, if available. If you are using cnMaestro, getting a Tech Support file from the unit while in taht stats (before rebooting) would help as well.
Different AP unit, FW 3.3, Same AP site, same symptoms, both Connectorized APs with ~8 ePMP Elevate SMs attached and about that many again Force180/Force200s.
I attempted to grab the tech support file from cnMaestro however the AP has lost connectivity to cnMaestro (cnMaestro reports it as down for 4 hours)
AP is still passing traffic. I logged into the console and pulled the syslog (though it doesn't go back to ~12:00 when the issue appears to have started);
May 29 15:59:50 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 15:59:50 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:01:00 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:01:00 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:05:12 MTC-AP22 DEVICE-AGENT[2534]: da_enter_state:DA_STATE_DISCOVERY
May 29 16:05:16 MTC-AP22 json-config[16273]: Error: file already registered: /var/run/json-config.pid
May 29 16:05:16 MTC-AP22 json-config[16273]: Fault to lock process (main): show
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: No Static URL configured
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: No dhcp 43 option configured
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: No DHCP Option 15 DomainName found
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: discover_cns: method 4 cns_url [cloud.cambiumnetworks.com]
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: build_http_get_request: cns_url [cloud.cambiumnetworks.com]
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: build_http_get_request: Serial Number not available
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: build_http_get_request: No cambium_id available
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: getaddrinfo failed with error_code=-2
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: Not able to resolve cloud.cambiumnetworks.com
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: Not able to resolve cloud.cambiumnetworks.com
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: platform_set_cns_connection_state: New CNS_STATE cns_serv_stat Resolving URL(1)
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: OpenConnection to cloud.cambiumnetworks.com:443 failed
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: Discovery of cnMaestro Failed
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: Unable to discover cnMaestro URL (re-discover in 358 seconds)
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: da_enter_state:DA_STATE_IDLE
May 29 16:05:16 MTC-AP22 DEVICE-AGENT[2534]: Attempting (re)connection in 5 minutes
May 29 16:05:37 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:05:37 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:07:09 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:07:09 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:08:12 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:08:12 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:09:21 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:09:21 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:10:04 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:10:05 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:11:15 MTC-AP22 DEVICE-AGENT[2534]: da_enter_state:DA_STATE_DISCOVERY
May 29 16:11:17 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:11:17 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:11:18 MTC-AP22 json-config[16357]: Error: file already registered: /var/run/json-config.pid
May 29 16:11:18 MTC-AP22 json-config[16357]: Fault to lock process (main): show
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: No Static URL configured
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: No dhcp 43 option configured
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: No DHCP Option 15 DomainName found
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: discover_cns: method 4 cns_url [cloud.cambiumnetworks.com]
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: build_http_get_request: cns_url [cloud.cambiumnetworks.com]
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: build_http_get_request: Serial Number not available
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: build_http_get_request: No cambium_id available
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: getaddrinfo failed with error_code=-2
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: Not able to resolve cloud.cambiumnetworks.com
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: Not able to resolve cloud.cambiumnetworks.com
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: platform_set_cns_connection_state: New CNS_STATE cns_serv_stat Resolving URL(1)
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: OpenConnection to cloud.cambiumnetworks.com:443 failed
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: Discovery of cnMaestro Failed
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: Unable to discover cnMaestro URL (re-discover in 311 seconds)
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: da_enter_state:DA_STATE_IDLE
May 29 16:11:18 MTC-AP22 DEVICE-AGENT[2534]: Attempting (re)connection in 5 minutes
May 29 16:14:19 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:14:19 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:15:29 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:15:29 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:16:31 MTC-AP22 DEVICE-AGENT[2534]: da_enter_state:DA_STATE_DISCOVERY
May 29 16:16:34 MTC-AP22 json-config[16429]: Error: file already registered: /var/run/json-config.pid
May 29 16:16:34 MTC-AP22 json-config[16429]: Fault to lock process (main): show
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: No Static URL configured
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: No dhcp 43 option configured
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: No DHCP Option 15 DomainName found
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: discover_cns: method 4 cns_url [cloud.cambiumnetworks.com]
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: build_http_get_request: cns_url [cloud.cambiumnetworks.com]
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: build_http_get_request: Serial Number not available
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: build_http_get_request: No cambium_id available
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: getaddrinfo failed with error_code=-2
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: Not able to resolve cloud.cambiumnetworks.com
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: Not able to resolve cloud.cambiumnetworks.com
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: platform_set_cns_connection_state: New CNS_STATE cns_serv_stat Resolving URL(1)
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: OpenConnection to cloud.cambiumnetworks.com:443 failed
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: Discovery of cnMaestro Failed
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: Unable to discover cnMaestro URL (re-discover in 346 seconds)
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: da_enter_state:DA_STATE_IDLE
May 29 16:16:34 MTC-AP22 DEVICE-AGENT[2534]: Attempting (re)connection in 5 minutes
May 29 16:16:45 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:16:45 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
May 29 16:18:10 MTC-AP22 json-config[2490]: Error: file already registered: /var/run/json-config.pid
May 29 16:18:10 MTC-AP22 json-config[2490]: Fault to lock process (action): cambiumDeviceAgentEnable
There is no output from debug crashlog.
Am I able to generate/download the tech support file from SSH or via any other means? As there's still traffic being passed we've elected to leave the unit alone for now; let me know if there's anything you'd like us to try...
@Fedor Yes, the AP is reachable via SSH (that's how I pulled the syslog from memory) and it is responding to SNMP requests just fine...
At this point it might be better if we could continue this via email? My address is in my profile, or I can PM it to you? I'd like to avoid posting too many details on a public forum ;)
