Random rebooting of PMP450i Access points due to memory allocation failure with NTP Server issues on 21.0 and 21.1

Hi,

We are seeing a strange and regular random issue on just our 450i access points. We are seeing random reboots based on below -

07/31/2022 : 19:27:41 AEST : :Resetting due to memory allocation failure
07/31/2022 : 19:27:41 AEST : :Forced reset;
07/31/2022 : 19:27:41 AEST :
System Startup
System Reset Exception – Reset due to FatalError
Software Version : CANOPY 21.1 AP

All our 450i are seeing the same error and rebooting. Whilst they are on 21.1 - we did see this behaviour on previous versions (20.3 onwards).

We are not seeing the same error or reboot issues on our 450m Access points. Has anyone else seen this issue?

Is this on 3GHz/CBRS & 5GHz 450i AP’s? Have you tried the newest beta just on the AP?

@Charlie anything else you can think of other then raising a ticket and submitting diags?

Hi Eric,

The issue is occurring on the 5Ghz 450i units. I had thought it was a GPS issue (all have two GPS inputs - cnPulse on the AUX and main power sync). I had seen this issue before on the 5Ghz 450m units …

I have supported a ticket and submitted diags - but seeing if anyone else had an issue - as it is regularly occurring on all our 5Ghz 450i units.

I will try the beta tonight (was waiting for release - but will test beta).

1 Like

@arobo, would you send me the engineering.cgi?

I hope this is the same issue already fixed in 21.1.0.1 BETA-3:
CPY-18035: AP with unreachable NTP server is running into FatalError reset due to fail to allocate mem, out of small heap in 21.0 or 21.1.

This issue only happens when on of the listed NTP servers is unreachable. It was introduced in 21.0.

We have not released 21.1.0.1 BETA-3 yet, unfortunately. A workaround would be to reconfigure your NTP servers to something that is valid and can be reached, or 0.0.0.0.

Thanks for posting and I hope it this problem. eng.cgi will confirm it for me.

We are planning to release 21.1.0.1 officially in around two weeks time, unless something urgent comes up in the meantime that we need to fix.

1 Like

Hi Charlie, I have already sent in the eng.cgi in a support ticket :slight_smile:

Support ID -315833

I can send to you directly if required. I can try the BETA tonight for you - the NTP servers on the APs are the same as the 450m Access points and they should be reachable (will double check nothing has gone awry). Also will try 0.0.0.0

Thanks always for the support.

1 Like

Thanks for that, I’m a developer, so don’t ever do the “look up what tickets a customer has open” thing. Would have been smart to ask someone to do it for me though. :slight_smile: Oops. Thanks for the ticket number.

Yeah I don’t like the HTML format, next time try to upload the eng.cgi in XML if you would.

Yes, I have confirmed this is the same unreachable NTP server memory leak issue we have fixed in 21.1.0.1 BETA-3. We will be getting that posted soon.

You’re welcome. But it’s sort of like thanking a firefighter for putting out a fire… that they started. hehe :slight_smile:

But I do appreciate you taking the time to post and being willing to try the beta. We need customers like you to make this as well as time allows. :slight_smile:

2 Likes

Hi Charlie,

A bit more info here.

We have updated to the beta - but still having issues with NTP update on the AP. It is very strange as we can test the NTP server and seems to be working fine.

NTP time update to the switch that the APs are connected to using the same NTP server … works fine.

Pinging the NTP server from the AP works fine. Pinging from the NTP to the AP works fine. So network connectivity to both is ok

There is no firewall issues - and as stated other devices are updating their time fine from the same NTP server. But the access points are still stating Server time out on the NTP page of the AP… .We will keep looking. (Happy for suggestions though :slight_smile: )

1 Like

Yeah the beta just fixed a memory leak with the NTP server, not NTP parsing. So at least your AP won’t reset after a long time in this condition.

But we still need to figure out why it’s not updating your time. Lets see if we can reproduce it here.

In the meantime, if you try another NTP server does it work?