log(1).txt (29.6 KB)
log(2).txt (18.0 KB)
log(3).txt (3.0 KB)
Hi all, we are having a really frustrating issue with our ePMP 4000 series devices. I have submitted a ticket to Cambium, but I wanted to post it here, too, to see if anyone has some ideas on what we can do or has some of the details that I am asking for.
We are having an extremely frustrating issue happen with 18 APs (they are all ePMP 4500 series APs; either 4500s or 4500Ls) and we really need some guidance on what in the world is happening under the hood.
The symptoms of the issue is: if/when a SM disconnects from an AP and reconnects a couple of things that are odd happen.
- The AP will show a 0/<some good number> for the SM RSSI and 0/<some good number> for the SM SNR when looking at the downlink/uplink stats on the “Monitoring” → “Wireless” page on the AP.
- The “Wireless” monitoring menu on the AP will show either no hostname and no IP address for the SM in question or it will show the correct hostname and distance but it will show the default 192.168.1.200 IP address (note: this seems to be the SMs default IP address from the factory; all of our internal IP addresses are in the 10.0.0.0/8 network) if the SM is configured for DHCP or it will show the correct statically assigned address if the SM is configured for static addressing.
- One of two things will happen next: the SM will disconnect with the error message "Reason: 53 (AP KEEP ALIVE RX STUCK) or the SM will just stay connected in that state with an apparent 0 dBm downlink RSSI and SNR until we notice and try to address the problem.
The last part of number 3 is one of the most frustrating parts of this. There doesn’t seem to be anything that we can do to address it. Rebooting the SM and/or the AP doesn’t work, switching the channel of the AP and SM doesn’t work, turning off all of the other APs in the area doesn’t work, forcing the SMs to disconnect and reconnect doesn’t work, forcing a re-scan on the SM doesn’t work, etc. These SMs, when connected, have a downlink RSSI in the range of -40 dBm to -50 dBm and a downlink SNR in the range of 30 dBm to 40 dBm. We limit the SMs’ uplink RSSI to -64 dBm.
I have tried getting the engineering logs and seeing if there was more information in the logs if I log in to the SM or the AP via SSH, but there doesn’t seem to be anything. I have attached the logs (downloaded via the web GUI) of 3 ePMP 4500 series APs of the 18 ePMP 4500 series ePMP APs that are having this issue. Logs 1 and 3 are from an ePMP 4500L and log 2 is from an ePMP 4500. The screenshots are from the APs that match log 1 and log 3.
The AP in log 1 is running firmware version 5.9.2 (you can even see the update happen in the logs; it was running 5.9.0 before the update), and the other two are running firmware version 5.9.0. The SMs are running matching versions.
I have also included a (redacted) dump of one of the AP’s configs. The other APs have identical configs for any of the relevant config options.
This issue seems to have started somewhere around firmware version 5.7 and has continued through 5.9.2. I was really hoping that this issue had been solved in 5.9.0 because of the release notes at ePMP Software Release 5.9.0 is now available seeming to mention this exact problem in issue number AXG-10189, but that does not seem to be true.
It would be immensely helpful to know why this problem is happening on a technical level so that we can attempt to debug further. I would also very much appreciate any workarounds or any further information you have about this issue. Please orient your answer to be read by a technical audience that understands the details of how firmware for embedded devices is constructed and how it works, software development, C/C++/Rust (whatever your 802.11 drivers are written in), and the implementation details of any part of the 802.11 standard. Those are the technical details that we need and the level of detail we would very much appreciate. Otherwise, this has been a stubborn enough problem to just start binwalking the firmware files to figure out what the heck is going on, myself.
I have attached all of the files mentioned in the ticket to this post except for the config file (.json files aren’t supported on here), so here are the relevant parts of the config in case anything stands out to anyone. Thanks in advance for any and all replies – we’re desperate!
{
"centerFrequency": "5840",
"dataVLANEnable": "0",
"mcastVLANEnable": "0",
"dfsAlternative1CenterFrequency": "0",
"dfsAlternative2CenterFrequency": "0",
"dhcpLanEnable": "0",
"dhcpLanLimit": "10",
"dhcpLanStart": "1",
"dhcpOption82": "0",
"dhcpOption82CircuitIDType": "0",
"dhcpOption82CircuitValue": "Cambium",
"dhcpOption82RemoteIDType": "0",
"dhcpOption82RemoteValue": "Cambium",
"dhcpOption66": "1",
"cambiumSSHServerEnable": "1",
"cambiumSSHServerPort": "22",
"cambiumSSHServerDeprecatedAlgorithms": "0",
"cambiumTelnetServerEnable": "0",
"cambiumTelnetServerPort": "23",
"crashReporterEnable": "0",
"httpPort": "80",
"httpsPort": "443",
"l2FirewallEnable": "0",
"l2WanRemoteAccess": "0",
"l2LanRemoteAccess": "0",
"l2SnmpLanRemoteAccess": "1",
"l2DHCPServersBelowSTA": "0",
"l3FirewallEnable": "0",
"mcPriority": "0",
"mgmtVLANEnable": "0",
"networkBridgeDNSIPAddrPrimary": "10.255.3.1",
"networkBridgeDNSIPAddrSecondary": "10.255.3.2",
"networkBridgeGatewayIP": "10.255.3.1",
"networkBridgeIPAddr": "10.255.5.8",
"networkBridgeIPv6Addr": "",
"networkBridgeIPAddressMode": "1",
"networkBridgeIPv6AddressMode": "2",
"networkBridgeMTU": "1538",
"networkBridgeNetmask": "255.255.0.0",
"networkLanAutoNegotiation": "1",
"networkLanSpeed": "1000",
"networkLanDuplex": "1",
"networkLanEnabled": "1",
"networkLanAdvertise": "47",
"networkLan2Enabled": "1",
"networkLan2AutoNegotiation": "1",
"networkLan2Speed": "10000",
"networkLan2Duplex": "1",
"networkLan2PoEEnabled": "0",
"networkLan2MTU": "1538",
"networkLanIPAddr": "10.1.1.254",
"networkLanIPv6Addr": "",
"networkLanIPAddressMode": "1",
"networkLanIPv6AddressMode": "1",
"networkLanIPv6AddressType": "0",
"networkLanIPv6DHCPServer": "0",
"networkLanMTU": "1538",
"networkLanNetmask": "255.255.255.0",
"networkBroadcastStormEnabled": "0",
"networkBroadcastStormRate": "1000",
"networkLanDefaultIP": "169.254.1.1",
"mgmtIFEnable": "0",
"mgmtIFVLAN": "0",
"mgmtIFIPAddressMode": "2",
"mgmtIFIPv6AddressMode": "2",
"mgmtIFNetmask": "255.255.255.0",
"mgmtIFGateway": "",
"networkMode": "2",
"networkSTP": "0",
"networkPortSecurity": "0",
"networkPortSecurityMax": "5",
"networkPortSecurityAgingTime": "300",
"networkWanGatewayIP": "",
"networkWanIPAddr": "192.168.0.2",
"networkWanIPv6Addr": "",
"networkWanIPv6LocalInterfaceId": "::dead:beef",
"networkWanIPAddressMode": "2",
"networkWanIPv6AddressMode": "2",
"networkWanNetmask": "255.255.255.0",
"networkWanPPPoE": "0",
"networkWanPPPoEAC": "Cambium",
"networkWanPPPoEService": "temp",
"networkWanPPPoEAuth": "0",
"networkWanPPPoEMTU": "1492",
"networkWanPPPoEKeepAlive": "10",
"networkWanPPPoEAttempts": "5",
"networkWanPPPoEMSSClamping": "0",
"networkPPPoEIAEnable": "0",
"networkUPNP": "0",
"networkNATPMP": "0",
"networkLLDP": "1",
"networkLLDPMode": "2",
"networkMACTELNET": "0",
"networkMACTELNETProto": "1",
"mcastGroupLimit": "3",
"portForwardingEntryEnable": "0",
"portForwardingSepMangIPEntryEnable": "0",
"qosEnable": "0",
"wirelessCellSizeUnit": "1",
"wirelessDeviceCountryCode": "US",
"wirelessInterfaceEncryption": "2",
"wirelessInterfaceCipher": "1",
"wirelessInterfaceEncryptionKey": "XXXXXXXXXXXXXXX",
"wirelessInterfaceiFreqReuseMode": "1",
"wirelessInterfaceGuardInterval": "2",
"wirelessInterfaceHTMode": "2",
"wirelessInterface2HTMode": "1",
"wirelessInterfaceMode": "1",
"wirelessInterfaceProtocolMode": "1",
"wirelessInterfacePTPMode": "0",
"wirelessInterfaceScanFrequencyBandwidth": "1",
"wirelessInterfaceScanFrequencyListTwenty": "",
"wirelessInterfaceScanFrequencyListForty": "",
"wirelessInterfaceScanFrequencyListTen": "",
"wirelessInterfaceScanFrequencyListFive": "",
"wirelessInterfaceScanFrequencyListEighty": "",
"wirelessInterfaceScanFrequencyList160M": "",
"wirelessInterfaceSSID": "xxxxxxxxxxxxxxxxxx",
"wirelessInterfaceSyncSource": "1",
"wirelessInterfaceSyncHoldTime": "1000",
"wirelessInterfaceTXPower": "19",
"wirelessInterface2TXPower": "3",
"wirelessInterfaceTDDAntennaGain": "17",
"wirelessInterfaceTDDRatio": "1",
"wirelessInterfaceTPCTRL": "-64",
"wirelessInterfaceTPCMode": "2",
"wirelessInterfaceRateMaxMCS": "255",
"wirelessInterfaceDynamicBW": "1",
"wirelessInterfaceSTAMaxDLMCSLimit": "255",
"wirelessFBCompatibility": "0",
"wirelessMaximumCellSize": "2",
"wirelessMaximumSTA": "60",
"wirelessMIREnable": "0",
"wirelessRadiusTimeout": "5",
"wirelessRadiusRetry": "5",
"wirelessRadiusGUIUserAuth": "0",
"wirelessRadiusIdentityStr": "anonymous",
"wirelessRadiusIdentityRealm": "cambiumnetworks.com",
"wirelessRadiusUsername": "cambium-station",
"wirelessRadiusPassword": "cambium123",
"useMACAddressAsWirelessRadiusUsername": "0",
"wirelessRadiusUseDefaultCertificate": "1",
"wirelessRadiusUsePMP450Certificate": "1",
"wirelessStaIsolate": "0",
"wirelessMulticastEnhanceMode": "0",
"wirelessClientBridgeMode": "0",
"wirelessInterfaceTDDFrameSize": "5000",
"wirelessInterfaceColocState": "0",
"wirelessInterfaceColocSystemSyncSrc": "5",
"wirelessInterface1KeepAliveEnable": "1",
"wirelessInterface2KeepAliveEnable": "0",
"wirelessInterface1KeepAliveEPTPEnable": "1",
"wirelessInterface2KeepAliveEPTPEnable": "0",
"wirelessInterface1KeepAlivePMPEnable": "1",
"wirelessInterface1KeepAliveWLREnable": "1",
"wirelessRadioTransmissionOptimization": "2",
"wirelessAntennaType": "1",
"wirelessTDDBeamformingControl": "1",
"dpiQoSEnable": "0",
"dpiQoSEnable": "0",
"dpiLogLevel": "system",
"dpiLogComponents": "all",
"dcsEnable": "0",
"dcsLogFile": "\/tmp\/log\/dcsd.log",
"dcsFrequencyListTwenty": "",
"dcsFrequencyListForty": "",
"dcsFrequencyListEighty": "",
"dcsFrequencyListOneSixty": "",
"dcsSwitchThreshold": "4",
"dcsSwitchMinHoldTime": "180",
"dcsThroughputDropThreshold": "20",
"dcsBandwidthMask": "1",
"wirelessMIRProfileTable": [
{
"wirelessMIRProfileNumber": "0",
"wirelessMIRProfileDescription": "default",
"wirelessDLMIR": "1000000",
"wirelessULMIR": "1000000"
}
]
}