On two of our 450M AP 's I have seen a few reboots happen over the past 10 days or so. No event's leading up to the reboot were logged in the Event log. Also no start up log showing the AP's System Startup. Has anyone else noticed their 450M AP's rebooting for no apparent reason? This seems to have started with v 15.1.3. Thus far I have noticed this twice on one 450M and one time on another.
Normally I would expect to see something like this:
******System Startup****** System Reset Exception -- User Initiated Reset Software Version : CANOPY 15.1.3 (Build BETA-7) AP-DES Software Boot Version : CANOPYBOOT 1.0 Board Type : P14 Board Temperature : 0 C / 32 F Device Setting : 5.7GHz MU-MIMO OFDM - Access Point - 0a-00-3e-60-xx-xx - 5740.0 MHz - 30.0 MHz - 1/16 - CC 85 - 2.5 ms - North America/United States FPGA Version : 0b1e75 FPGA Features : DES, Sched, US/ETSI;
Yes, with version 15.1.3 there are many ghost restarts. There is no trace anywhere except in uptime. For now it seems to me that the thing happens on the Medusa AP. Also I have lots of reboots for Fatal Error:
****SYSTEM STARTUP****** System Reset Exception -- User Initiated Reset Software Version : CANOPY 15.1.3 (Build BETA-7) AP-AES Software Boot Version : CANOPYBOOT 1.0 […] 01/16/2018 : 13:27:13 UTC : :NiFreeBuf(): Invalid NiBuf:7609b800 Hdr:c532 on Src:44876 SrcIF:236 Dst:1246 Alc:60136 Len:48473 Cop:128
STAT 62 ( 0%) 0 0 0 254293660 ( 8192/ 3%/46%) Ready 0x8000584 IDLE 63 ( 0%) 0 0 0 938546787 ( 8192/ 3%/46%) Ready 0x8000584 PRI PC ID ---------------- 01/01/2016 : 02:00:00 CEST : :
01/01/2016 : 00:00:00 UTC : :Time Set 01/01/2016 : 02:00:00 CEST : ******System Startup****** System Reset Exception -- Watchdog Reset Software Version : CANOPY 15.1.3 AP-AES Board Type : P12
0x76f0ef58: 00000030 00000000 00000000 00000000 0x76f0ef68: 00000000 02/07/2018 : 10:05:31 UTC : :FatalError() 02/07/2018 : 10:05:32 UTC : :Forced reset; 02/07/2018 : 10:05:33 UTC : ******SYSTEM STARTUP****** System Reset Exception -- Reset due to FatalError Software Version : CANOPY 15.1.3 AP-AES Software Boot Version : CANOPYBOOT 1.0 Board Type : P14 Board Temperature : 0 C / 32 F
Well, I think it's either 15.1.3 or it's the 30Mhz channel. One 450M has been up for 13 months and the other 8 months, and this didn't happen until ~10 days ago. We loaded 15.1.3 on 12/13/17 to one AP and the other on 1/19/18. Looking back at my logs I see another unintended reboot, but there was some errors just before the reboot (but no System Startup entry). Here is the log just before the unintended reboot:
Well I will throw out there it must have something to do with firmware over 30mhz channel, as we do not have any channels greater than 20mhz.
We have 18 radios so far in the field and have seen a few stray reboots. Now I did have one start rebooting last week in pretty quick cycles. We powered it off, got it replaced and now I have it running at the office to see if it starts rebooting.
Could this have something to do with a traffic type that hits the radios?
We now have reports of this fault at 20, 30 and 40MHz. There is a hint in the diags (reported contents of the Nibuff) that it is something to do with QinQ, but we are still looking.
Thank you (and everyone else) for sending a set of diags you collected.
Be assured that finding the cause and a fix is our highest priority, the VP of engineering is keeping a close eye on this one.
I'd also like to report we too are seeing reboots on our 450m AP. We only have one AP up currently. It's been up about 2 weeks and has rebooted 3 times now.
AP is running 15.1.3 with 20mhz and has vlan enabled. 126 total connected SM's. Let me know if you would like more details.
I'd also like to report we too are seeing reboots on our 450m AP. We only have one AP up currently. It's been up about 2 weeks and has rebooted 3 times now.
AP is running 15.1.3 with 20mhz and has vlan enabled. 126 total connected SM's. Let me know if you would like more details.
Hi ericmule,
Thank you for the offer. I think we are getting close to a fix for this. We have been running development/debug builds on a couple of networks. These included a fix and debugging counters for if the condition that we think caused the reboot occured again. Over the last week the counters have incremented and the units have not rebooted. So it's looking promising.
The fix will be in 15.2 official release (which should be less than a month away). If anyone doesn't want to wait for this then 15.1.1. should be stable and not have the reboot issue. If you needed a feature from 15.1.3 (e.g. 30/40MHz or 5ms frame) and can't possibly wait for 15.2 then contact me and I'll see what I can do about getting a closed beta release made available after Easter.
The closed beta should be available now if you contact support (tier 1 should have been informed) or through your RTM. It's not open beta as it has not had as much SIT testing time as usual yet.
This load will have the following
450m reset fix (NiBuff error)
450m GPS coordinate display fix
450b improvements/bugfixes
430 SM first registration fix for moving from 14.x or earlier s/w
I've been experiencing reboots on 2 brand new 450m APs we recently purchased. It happens on 15.1.1, 15.1.3, and 15.1.5. I have only tried 15MHz and 20MHz channels, but it happens with both. The only way I can get the APs to stay up more than 10 minutes is to select "none" as the frequency carrier. Below is a little of the event log. Please advise.
01/01/2016 : 00:00:01 UTC : ******System Startup****** System Reset Exception -- Power-On Reset Software Version : CANOPY 15.1.5 AP-None Software Boot Version : CANOPYBOOT 1.0 Board Type : P14 Board Temperature : 0 C / 32 F Device Setting : 5.7GHz MU-MIMO OFDM - Access Point - 0a-00-3e-60-46-81 - 5795.0 MHz - 15.0 MHz - 1/16 - CC 0 - 5.0 ms - North America/United States FPGA Version : 031576 FPGA Features : DES, Sched, US/ETSI; 01/01/2016 : 00:00:07 UTC : :RFSync: EVENT = 4 STATE = 0 05/03/2018 : 14:58:09 UTC : :Time Set 01/01/2016 : 00:00:01 UTC : : 01/01/2016 : 00:00:00 UTC : :Time Set
01/01/2016 : 00:00:01 UTC : ******System Startup****** System Reset Exception -- Power-On Reset Software Version : CANOPY 15.1.5 AP-None Software Boot Version : CANOPYBOOT 1.0 Board Type : P14 Board Temperature : 0 C / 32 F Device Setting : 5.7GHz MU-MIMO OFDM - Access Point - 0a-00-3e-60-46-81 - 5795.0 MHz - 15.0 MHz - 1/16 - CC 0 - 5.0 ms - North America/United States FPGA Version : 031576 FPGA Features : DES, Sched, US/ETSI; 01/01/2016 : 00:00:07 UTC : :RFSync: EVENT = 4 STATE = 0 05/03/2018 : 15:03:55 UTC : :Time Set 01/01/2016 : 00:00:01 UTC : : 01/01/2016 : 00:00:00 UTC : :Time Set
01/01/2016 : 00:00:01 UTC : ******System Startup****** System Reset Exception -- Power-On Reset Software Version : CANOPY 15.1.5 AP-None Software Boot Version : CANOPYBOOT 1.0 Board Type : P14 Board Temperature : 0 C / 32 F Device Setting : 5.7GHz MU-MIMO OFDM - Access Point - 0a-00-3e-60-46-81 - 5795.0 MHz - 15.0 MHz - 1/16 - CC 0 - 5.0 ms - North America/United States FPGA Version : 031576 FPGA Features : DES, Sched, US/ETSI; 01/01/2016 : 00:00:07 UTC : :RFSync: EVENT = 4 STATE = 0 05/03/2018 : 15:08:09 UTC : :Time Set 01/01/2016 : 00:00:01 UTC : : 01/01/2016 : 00:00:00 UTC : :Time Set
01/01/2016 : 00:00:01 UTC : ******System Startup****** System Reset Exception -- Power-On Reset Software Version : CANOPY 15.1.5 AP-None Software Boot Version : CANOPYBOOT 1.0 Board Type : P14 Board Temperature : 0 C / 32 F Device Setting : 5.7GHz MU-MIMO OFDM - Access Point - 0a-00-3e-60-46-81 - 5795.0 MHz - 15.0 MHz - 1/16 - CC 0 - 5.0 ms - North America/United States FPGA Version : 031576 FPGA Features : DES, Sched, US/ETSI; 01/01/2016 : 00:00:07 UTC : :RFSync: EVENT = 4 STATE = 0 05/03/2018 : 15:13:34 UTC : :Time Set
We’re also experiencing similar issues with our 2 3.6 Medusa APs.
12/31/2019 : 17:00:06 MST : :RFSync: EVENT = 4 STATE = 0
09/23/2021 : 10:39:50 MST : :Time Set
09/23/2021 : 10:40:20 MST : :Board Temperature: 16 C / 60 F
AP would reboot every 30 to 40 mins. Our East AP was working good at first with 10 SMs connected but it started acting up when we start connecting SMs to the 2nd AP. Tried updating the firmware to v20.2.2.1 but no luck. East AP eventually went to Radio-Not-Calibrated mode.