AP 450m 21.0 Watchdog reset due to process: canopy error

We had a 450m rebooting due to an error we have never seen before, Has someone else noticed something like this on their APs?

******System Startup****** 
System Reset Exception --  User Initiated Reset 
Software Version : CANOPY 21.0 AP
Software Boot Version : CANOPYBOOT 1.0
Board Type : P14
Device Setting : 5.7GHz MU-MIMO OFDM - Access Point - 0a-00-3e-60-5b-d8 - 5735.0 MHz - 20.0 MHz - 1/16 - CC 176 - 2.5 ms - Other Regulatory/Other
FPGA Version : 0c0379;
03/16/2022 : 02:31:30 UTC : File ucos_ii/src/common/psos_wrap.c : Line 1040 :GPS Date/Time Set;
03/16/2022 : 02:33:41 UTC : File ucos_ii/src/common/psos_wrap.c : Line 1107 :Time Set
03/16/2022 : 02:33:59 UTC : Acquired sync pulse from Timing Port (UGPS/cnPulse)
03/16/2022 : 02:34:09 UTC : File devices/fpga/src/fpga_device_driver.c : Line 191 :Board Temperature: 21 C / 69 F
03/23/2022 : 16:25:37 UTC : Lost sync pulse from Timing Port (UGPS/cnPulse)
03/23/2022 : 16:25:37 UTC : File cmdline/src/gps.c : Line 5857 :Sync Pulse History [1/5/15 min]: Sync%[91.67%/98.34%/99.45%], SNR[33.25/32.40/32.81], Visible[20.00/20.00/19.95], Tracked[15.00/14.50/14.76]
03/23/2022 : 16:25:37 UTC : Generating Sync - Free Run
03/23/2022 : 16:43:53 UTC : Acquired sync pulse from Timing Port (UGPS/cnPulse)
04/01/2022 : 14:17:27 UTC : Lost sync pulse from Timing Port (UGPS/cnPulse)
04/01/2022 : 14:17:27 UTC : File cmdline/src/gps.c : Line 5857 :Sync Pulse History [1/5/15 min]: Sync%[93.34%/98.67%/99.56%], SNR[33.50/34.25/34.85], Visible[17.00/17.00/18.01], Tracked[16.00/15.25/14.85]
04/01/2022 : 14:17:27 UTC : Generating Sync - Free Run
04/02/2022 : 09:51:14 UTC : File mips64/src/stub.c : Line 207 :Fatal Error
04/02/2022 : 09:51:14 UTC : File mips64/src/stub.c : Line 222 :Backtrace:
04/02/2022 : 09:51:14 UTC : /opt/cambium/canopy() [0x100071cc]
04/02/2022 : 09:51:14 UTC : /opt/cambium/canopy() [0x1056f230]
04/02/2022 : 09:51:14 UTC : [0x7fff7018]
04/02/2022 : 09:51:14 UTC : /opt/cambium/canopy() [0x104d869e]
04/02/2022 : 09:51:14 UTC : /opt/cambium/canopy() [0x1042cd74]
04/02/2022 : 09:51:14 UTC : /opt/cambium/canopy() [0x104bba14]
04/02/2022 : 09:51:14 UTC : /opt/cambium/canopy() [0x104bc45c]
04/02/2022 : 09:51:14 UTC : /opt/cambium/canopy() [0x104bcb0c]
04/02/2022 : 09:51:14 UTC : /opt/cambium/canopy() [0x1056bf14]
04/02/2022 : 09:51:14 UTC : /lib32-fp/libpthread.so.0(+0x78d8) [0x7713d8d8]
04/02/2022 : 09:51:14 UTC : /lib32-fp/libc.so.6(+0xe90cc) [0x76e170cc]
01/01/2020 : 00:00:01 UTC : File Startup : Line 0 :
01/01/2020 : 00:00:00 UTC : File ucos_ii/src/common/psos_wrap.c : Line 1107 :Time Set

01/01/2020 : 00:00:01 UTC : 
******System Startup****** 
System Reset Exception --  Watchdog Reset due to process: canopy
Software Version : CANOPY 21.0 AP
Software Boot Version : CANOPYBOOT 1.0
Board Type : P14
Device Setting : 5.7GHz MU-MIMO OFDM - Access Point - 0a-00-3e-60-5b-d8 - 5735.0 MHz - 20.0 MHz - 1/16 - CC 176 - 2.5 ms - Other Regulatory/Other
FPGA Version : 0c0379;
01/01/2020 : 00:00:12 UTC : File ucos_ii/src/common/psos_wrap.c : Line 1040 :GPS Date/Time Set;
04/02/2022 : 09:54:44 UTC : File ucos_ii/src/common/psos_wrap.c : Line 1107 :Time Set
04/02/2022 : 09:54:57 UTC : Acquired sync pulse from Timing Port (UGPS/cnPulse)
04/02/2022 : 09:55:07 UTC : File devices/fpga/src/fpga_device_driver.c : Line 191 :Board Temperature: 18 C / 64 F

I would open a ticket with support. That is giving a trace dump, which is not good.

@MW_WISP,

Thanks for posting. I’ve looked at the crash and I believe the root cause is due to the software not correctly handling a bad GPS serial data stream from the GPS chip through the GPS serial cable to the software. We have had other customers having the same issue which we have already investigated and fixed in 21.1.

We have not yet posted a 21.1 BETA load yet, but when we do it will have this fix included in it. In the meantime I could send you a pre-beta image to run on there if you’re interested.

This has been a longstanding problem not only on 21.0 but all the releases prior as well. So unfortunately there’s no version to downgrade to avoid this particular crash.

How often are you seeing it?

The other workaround would be to suggest looking into your GPS cabling. But I realize that is a tower climb, so not an easy ask. But that would hopefully solve your GPS sync issues as well, as I see you’re not getting 100% sync pulses.

Hi and thanks for your reply. This is the first time we witnessed an AP rebooting due to this error, and since Saturday thankfully it didn’t happen again. We have had other instances of APs without a 100% GPS sync using a cnPulse, but usually this situation resolves itself without taking much time. I’ve read somewhere else in this forum that 21.1 was due to be launched in April, but if this kind of error is prone to reappear we could try a pre-beta image on this AP, just to avoid future rebooting during the weekend:)

1 Like

This is resolved in 21.1 BETA-1.

I highly recommend upgrading the AP that has this reset with this fix.

1 Like

We are Having the Same issue On one of Our 3.x 450M where it is Saying Watchdog Reset due to process: canopy. We upgrade to 21.1 beta and the issue still is happing. We have replaced the 450M and resubmitted the allocation in CBRS. Still it reboots about after 7 to 15 minutes. Any Suggestions?

Have you tried beta 4 yet?

Yes. Unfortunately, it did not resolve the issue.

I’m a little backed up on issues, but I’ll contact you to gather some debug information. Thanks for bringing this up.