Hello. We have 1 ePMP1000 AP (2.4Ghz) with 23 connected clients that has twice now lost all it's clients, and was showing 0 associations when a customer called to report no internet. In both cases, a reboot of the AP by our support staff restored the clients, but 5 days later it did it again a second time. Of course, after we rebooted the AP, it makes it difficult to diagnose what's gone on.
I don't see anywhere in cnMaestro where it can log the AP critical errors or anything like that - is that right?
Since this is 2.4Ghz, this isn't a DFS issue.
Also, we have our GSP Holdoff time on this AP set to 600 seconds, so I guess it is still possible that it's losing GPS I suppose, but it'd have to be out for more than 10 minutes at a time, right? Right now, the AP reports that it can see 13 Satellites, that it's tracked 10 of them - and the Satellite SNR's are all 31 - 49 currently. This tower is wide open to the south and the only other GPS receiving gear on there is a few other ePMP1000 AP's, and a couple old 900Mhz Canopy AP's. So - due to all this, it doesn't really feel like a GPS losing sync problem to me. But again - we can't really see that info or logs anywhere that I can see, after we rebooted the AP.
This AP is currently running 3.3 which it has been for some time without problems, and it does report that it's got the latest GPS firmware as well. There are 3 other 2.4Ghz AP's on this same tower that are running the same firmware, without any issues.
If the AP is losing GPS sync, it will def show it in the logs, but you have to first enable the log. We had a number of AP's that was doing this and we eventually just had to use sync off the ethernet via PacketFlux.
You might want to try updating to 3.5.1 as there were some GPS firmware updates for newer rev2 GPS chips on certain radios.
In case it is GPS issue you will be able to find messages like following:
Feb 11 17:48:43 Belvedere AP - 40 MHz kernel: [618243.560000] GPS Sync Lost. (00:48:43:406152)
Feb 11 17:48:43 Belvedere AP - 40 MHz kernel: [618243.570000] GPS Sync Lost. (00:48:43:410600)
Feb 11 17:48:43 Belvedere AP - 40 MHz kernel: [618243.630000] GPS Sync Lost. (00:48:43:470268)
I would like to advidse to remove external GPS antenna in case it is used on current radio.
Sometimes LTE signal affects GPS receiver and internal GPS antenna is more robust against interference. Thank you.
OK, When I look at it, the SNR on the Satelites, it is usually shown as 30 to 40 SNR, and it seems to work fine for many month's at a time. So, I'm not certain it's GPS losing sync, that was just an idea - a possibility.
Also, with my holdoff time set at 600 seconds, if I'm correct that would mean that the GPS would have to lose connection and then not be able to re-achieve GPS for 10 minutes, correct? It's strange that right after a reboot, everyone immedately re-associates, which means if it is GPS, then a reboot seems to eliminate the interference.
In any event, I'm interested in seeing the logs and what they tell us next time. So, in order to see anything in the logs, do we need to have an external SysLog server setup and log to that? Is there no logs saved in the AP itself?
In any event, I'm interested in seeing the logs and what they tell us next time. So, in order to see anything in the logs, do we need to have an external SysLog server setup and log to that? Is there no logs saved in the AP itself?
Thanks,
I already told you how to do this... it's very simple, no syslog server required. Go to Monitor -> System Log and enable it. It will show all the recent log messages immediatly including the aforementioned examples for a GPS sync drop.
@Eric Ozrelic wrote: I already told you how to do this...
Ok, sorry. Thank you.
All I get are these errors.
Feb 21 18:17:35 RDH-S-2 DEVICE-AGENT[2520]: Accept failed
Feb 21 18:17:35 RDH-S-2 DEVICE-AGENT[2520]: No data read from unix domain socket, nbytes = -1
Feb 21 23:08:37 RDH-S-2 DEVICE-AGENT[2520]: Accept failed
Feb 21 23:08:37 RDH-S-2 DEVICE-AGENT[2520]: No data read from unix domain socket, nbytes = -1
Feb 21 23:18:39 RDH-S-2 DEVICE-AGENT[2520]: Accept failed
Feb 21 23:18:39 RDH-S-2 DEVICE-AGENT[2520]: No data read from unix domain socket, nbytes = -1
Feb 22 11:11:03 RDH-S-2 DEVICE-AGENT[2520]: Accept failed
Feb 22 11:11:03 RDH-S-2 DEVICE-AGENT[2520]: No data read from unix domain socket, nbytes = -1
Feb 22 22:33:37 RDH-S-2 DEVICE-AGENT[2520]: Accept failed
Feb 22 22:33:37 RDH-S-2 DEVICE-AGENT[2520]: No data read from unix domain socket, nbytes = -1
Feb 22 22:33:37 RDH-S-2 DEVICE-AGENT[2520]: Accept failed
Feb 22 22:33:37 RDH-S-2 DEVICE-AGENT[2520]: No data read from unix domain socket, nbytes = -1
Hmmm... did you check the log during or right after (before rebooting the AP) when the client drop occured?
>We have 1 ePMP1000 AP (2.4Ghz) with 23 connected clients that has >twice now lost all it's clients, and was showing 0 associations when a >customer called to report no internet. In both cases, a reboot of the AP >by our support staff restored the clients, but 5 days later it did it again >a second time. Of course, after we rebooted the AP, it makes it difficult >to diagnose what's gone on.
I think all those device agent messages have to do with cnMaestro. I'm not seeing anything GPS related, so perhaps that's not the issue.
>Also, we have our GSP Holdoff time on this AP set to 600 seconds, >so I guess it is still possible that it's losing GPS I suppose, but it'd >have to be out for more than 10 minutes at a time, right? Right now, >the AP reports that it can see 13 Satellites, that it's tracked 10 of them >and the Satellite SNR's are all 31 - 49 currently. This tower is wide >open to the south and the only other GPS receiving gear on there >is a few other ePMP1000 AP's, and a couple old 900Mhz Canopy >AP's. So - due to all this, it doesn't really feel like a GPS losing sync >problem to me. But again - we can't really see that info or logs >anywhere that I can see, after we rebooted the AP.
What framing are you using? Flexible or fixed? 2.5ms or 5ms? and are you actually using sync and back to back frequency reuse?
> Fixed, 5ms, 75/25, 600 seconds GPS Holdoff, 10Mhz width, and 'Back Sector' (althought the oposing "North" sector is configured to be currently 20Mhz away)
Lastly, and I didn't see in the thread if you confirmed this or not... are you running 3.5.1 on the AP and clients?
>This AP is currently running 3.3 which it has been for some time >without problems, and it does report that it's got the latest GPS >firmware as well. There are 3 other 2.4Ghz AP's on this same >tower that are running the same firmware, without any issues.
So - the AP has been up for 11 days now since the last time it lost all clients. It did it twice in 5 days, and not before or after. As mentioned, I don't think it feels like a real GPS interference issue because holdoff is set to 600 seconds and it doesn't self fix in that 10 minutes - yet as soon as we rebooted, it was able to lock onto the Satelites. The support staff that was here both times just rebooted the AP, which restored service - but no one was around to diagnose if the GPS had lost link with the Satellites or what was in the logs. All we see in the logs are for the last 11 days is those previously reported cnMaestro or whatever entries.
We are running 3.3 still because 3.5.1 breaks our Real Time Throughput monitoring software and I understood that a new firmware is coming shortly. So, if the new firmware is going to break the Real Time Throughput tool again, my plan was to wait and get the Extension updated yet again, before we upgrade the firmware.
I know it sounds weird, but I would recommend to disconnect the GPS puck. In this case GPS reciever will use small built-in antenna patch. In most cases GPS is stuck due to high signal level of GSM/LTE transimitters and using lower gain build-in antenna patch helps to workaround the issue.
This probem usually occures when the frequency on which your AP is operating is over utilised in your space.
When you change your frequency, this probem wil stop. It will be good to do a frequech scan on that AP and be able to choose a more free frequech to use to stop the scenario described.