400C PTP been running great for about a year. Recently had to change from 40Mhz to 80Mhz channel and link started failing and bugsplat errors spamming logs. Updated radios to newest beta RC a few days ago and so far link stable, no bugsplats but now these are spamming the logs of both radios
I feel like this is along the same lines as or related to the bugsplats just different wording. It appears to be saying that the kernal is crashing and it’s unable to get the crash data but it wouldn’t be the first time I completely misunderstood what a log entry was trying to tell me.
My main concern is if I there is something wrong with one or both of these radios of if this is just some glitch caused by the software ?
I would be interested to take a look into the Tech Support File if you could share it.
I would assume that it’s related to crashes that happened before and now the device is trying to send it to the remote server.
Edit: So after thinking about what you said “It’s trying to send it to the remote server”. What remote server ? It’s accessing our onprm cnM, I disabled syslog but the errors continue. Do the 400c radios pop up the “Cambium wants your radio to phone home and report stuff” when first accessed ? I normally click the “do not send anything” but it’s possible I unintentionally allowed it and since the radios do not have access to anything outside 10.#.#.#/ it’s trying to phone home and can’t ? I’m looking though the settings, if I did enable that when it popped up, how do I disable it ?
I sent them in. Ticket# 349043
Thank you for your patience and support.
As per the tech. logs, the configuration, and the RF statistics look good. The other logs also not >showing any suspicious thing which can affect these devices.
And as per the logs, the wireless link is also working well, just we are getting these crash >messages/notifications which might be something due to internal process related, but of course that >should not post on the Syslog.
A similar issue has been fixed in the latest Beta firmware version 5.4.1-RC9, so I would suggest you >upgrade both Force 400C devices to the latest Bate Firmware version 5.4.1-RC9.
I have attached the latest Beta-FW 5.4.1-RC9 for the Force 400 series devices, kindly upgrade both >devices to it and let us know the status.
I’ll upgrade to whatever the current beta RC is when I get time to do it or if the link goes down again.
I’m seing that in the lastes versions, when you make the first conection to the GUI, they request to modify the admin’s password, the SNMP communitys and grant o deny the send crash reports to cambium.
It’s posible that if this option was enabled, despiste if the unit is setting to connect or not to cnMaestro (on premise o cloud) make a side conection to another CN server to send specifics the crash reports?
Just FYI I upgraded to the 5.4.1-RC9 support sent me and it didn’t change anything. The errors continue as before, repeated exactly every 20 minutes by both radios with the exact same time stamps.
We would be happy to take a look deeper in the described issue if you can share Tech Support File from the radio with me.
The daemon on the device is trying to collect the information related to the crash if it happened even if the option “Not send” is set.
I’m aware of this ticket and currently trying to reproduce this issue in my lab. Our team initial assumption was wrong and we still able to see it in RC9.
creproterd - is a daemon reporting crashes and the issue is there.
There is no actual kernel crush itself.
So this link started taking spells dropping due to DFS hits, over and over for hours and then running fine for hours, just in the last couple of days (since upgrading to RC9 not sure if just coincidence or not) so I gave up and went back to the 40Mhz channel it had been on and that it had ran perfectly on for a year or so and now I have all the fun of the “Radar detected” hits with a ton of " AP KEEP ALIVE STUCK" over and over every few minutes…
May 9 04:05:30 BH11S netifd: Network device ‘ath0’ link is down
May 9 04:05:30 BH11S kernel: [359188.065635] wlan: [0:I:ANY] ieee80211_assoc_state_run_event: ieee80211_assoc_state_run_event: vap: 0(0x888e8000) event: 2
May 9 04:05:30 BH11S kernel: [359188.064658] SM disassociated from AP[bc:e6:7c:20:40:07] F=5620 11axahe40plus. Reason: 53 (AP KEEP ALIVE RX STUCK)
May 9 04:04:50 BH11S sminfo-notify: wifi0 PROCFS_CMD=“HWADDR=bc:e6:7c:20:16:09; HTTPS=0; PORT=80; HOSTNAME=BH11S; IPADDR=10.10.11.253; IPv6ADDR=; IPADDR_CNS1=52.5.95.225; IPADDR_CNS2=52.72.154.230; FORCE_PATTERN=-1.-1; CT_PROP=0 ;”
May 9 04:04:45 BH11S kernel: [359142.948443] SM associated with AP[bc:e6:7c:20:40:07]
May 9 04:04:45 BH11S netifd: Network device ‘ath0’ link is up
May 9 04:04:25 BH11S netifd: Network device ‘ath0’ link is down
May 9 04:04:25 BH11S kernel: [359123.345026] wlan: [0:I:ANY] ieee80211_assoc_state_run_event: ieee80211_assoc_state_run_event: vap: 0(0x888e8000) event: 2
May 9 04:04:25 BH11S kernel: [359123.344036] SM disassociated from AP[bc:e6:7c:20:40:07] F=5620 11axahe40plus. Reason: 53 (AP KEEP ALIVE RX STUCK)
May 9 04:03:55 BH11S sminfo-notify: wifi0 PROCFS_CMD=“HWADDR=bc:e6:7c:20:16:09; HTTPS=0; PORT=80; HOSTNAME=BH11S; IPADDR=10.10.11.253; IPv6ADDR=; IPADDR_CNS1=52.5.95.225; IPADDR_CNS2=52.72.154.230; FORCE_PATTERN=-1.-1; CT_PROP=0 ;”
May 9 04:03:50 BH11S kernel: [359088.227993] SM associated with AP[bc:e6:7c:20:40:07]
May 9 04:03:50 BH11S netifd: Network device ‘ath0’ link is up
May 9 04:03:31 BH11S netifd: Network device ‘ath0’ link is down
May 9 04:03:30 BH11S kernel: [359068.634505] wlan: [0:I:ANY] ieee80211_assoc_state_run_event: ieee80211_assoc_state_run_event: vap: 0(0x888e8000) event: 2
May 9 04:03:30 BH11S kernel: [359068.633450] SM disassociated from AP[bc:e6:7c:20:40:07] F=5620 11axahe40plus. Reason: 53 (AP KEEP ALIVE RX STUCK)
May 9 04:03:00 BH11S sminfo-notify: wifi0 PROCFS_CMD=“HWADDR=bc:e6:7c:20:16:09; HTTPS=0; PORT=80; HOSTNAME=BH11S; IPADDR=10.10.11.253; IPv6ADDR=; IPADDR_CNS1=52.5.95.225; IPADDR_CNS2=52.72.154.230; FORCE_PATTERN=-1.-1; CT_PROP=0 ;”
May 9 04:02:55 BH11S kernel: [359033.517036] SM associated with AP[bc:e6:7c:20:40:07]
May 9 04:02:55 BH11S netifd: Network device ‘ath0’ link is up
May 9 03:59:53 BH11S creporterd: kernel process_crash… get_crash_data fail : -1
May 9 03:56:07 BH11S netifd: Network device ‘ath0’ link is down
May 9 03:56:07 BH11S kernel: [358625.130339] wlan: [0:I:ANY] ieee80211_assoc_state_run_event: ieee80211_assoc_state_run_event: vap: 0(0x888e8000) event: 2
May 9 03:56:07 BH11S kernel: [358625.120320] SM disassociated from AP[bc:e6:7c:20:40:07] F=5620 11axahe40plus. Reason: 54 (RADAR DETECTED ON AP)
May 9 03:55:54 BH11S sminfo-notify: wifi0 PROCFS_CMD=“HWADDR=bc:e6:7c:20:16:09; HTTPS=0; PORT=80; HOSTNAME=BH11S; IPADDR=10.10.11.253; IPv6ADDR=; IPADDR_CNS1=52.5.95.225; IPADDR_CNS2=52.72.154.230; FORCE_PATTERN=-1.-1; CT_PROP=0 ;”
May 9 03:55:48 BH11S kernel: [358606.661753] SM associated with AP[bc:e6:7c:20:40:07]
May 9 03:55:48 BH11S netifd: Network device ‘ath0’ link is up
May 9 03:55:29 BH11S netifd: Network device ‘ath0’ link is down
May 9 03:55:29 BH11S kernel: [358587.079985] wlan: [0:I:ANY] ieee80211_assoc_state_run_event: ieee80211_assoc_state_run_event: vap: 0(0x888e8000) event: 2
May 9 03:55:29 BH11S kernel: [358587.079516] SM disassociated from AP[bc:e6:7c:20:40:07] F=5620 11axahe40plus. Reason: 53 (AP KEEP ALIVE RX STUCK)
May 9 03:53:54 BH11S sminfo-notify: wifi0 PROCFS_CMD=“HWADDR=bc:e6:7c:20:16:09; HTTPS=0; PORT=80; HOSTNAME=BH11S; IPADDR=10.10.11.253; IPv6ADDR=; IPADDR_CNS1=52.5.95.225; IPADDR_CNS2=52.72.154.230; FORCE_PATTERN=-1.-1; CT_PROP=0 ;”
May 9 03:53:49 BH11S kernel: [358486.960897] SM associated with AP[bc:e6:7c:20:40:07]
Have you tried swapping which radio is the AP? I dont mean physical swap, just with the current SM setup as an AP.
I remember when the epmp1000 radios had this similar DFS issue that we were seeing out of band RF that was triggering DFS hits and swapping the radios config made the link stable.
Huh… until this F400 I never used ePTP and that never crossed my mind but yeah, it doesn’t matter which end is which with ePTP does it ! I’ll keep that in mind as I continue to troubleshoot this link (Been stable for 5+ hours now waiting for it start throwing up all over itself again).
Actually it does even with ePTP. One will always boot up as the AP and this one needs to be determined as the master since the link synchronization comes from the AP even with the ePTP protocol.
@Douglas_Generous – I think @brubble1 is talking about as the ratio goes. With TDD, you if you want 75%/25% that’s from the AP’s perspective. With ePTP (or flexible frame) the AP/Master is still in charge of scheduling, but I think brubble meant “as far as downlink/uplink performance, it doesn’t matter” as much.
I was thinking in terms of how DFS events are detected. Any sudden increase of the noise floor as detected by the AP (sudden drop in SNR) while using a DFS channel will result in a DFS back-off.
This can be as simple as being too close to another radio using the same DFS channel or a harmonic of another band that is strong enough to trigger the DFS back-off.
Swapping which end that controls the link can help if the source is close to the current AP/master side of the link.
It’ll be nice when 4.7.1 is out and has the seamless channel switching finally. That’ll really REALLY help for optimization and for tweaking and for troubleshooting too.
Was there a resolutioon to this issue? Seeing the same thing in the log on both ends of a F400 CSM PTP link, get-crash data fail every 20 minutes. Crash report and CNMaestro are disabled. Running latest 5.4.1 software. Both units are reporting the error. Other than the annoyance factor, no seeing anything weird with the link and it has been stable.
The message can be seen in the log when the remote crash reporting is disabled.
It will be fixed in the upcoming releases.
No impact on stability or performance.
