Recently our customers indicated having the following GPS issues across different ePMP products.
ePMP3000/ePMP3000L doesn’t report GPS version and/or satellite information while GPS sync works.
As a workaround, you can reboot the unit multiple times to get it working. We are working on a fix of this problem and it will be addressed in 4.4.2 release.
ePMP3000 is losing GPS sync
In most cases, it is caused by LTE interference. Every unit has both an external and internal GPS antenna. As a workaround, it is recommended to disconnect the external high gain antenna to avoid interference.
ePMP3000L is losing GPS sync
As a workaround, it is recommended to disable Spectrum Analyzer and disconnect the GPS antenna. We are investigating an issue. The goal is to fix it in the upcoming release.
ePMP1000/ePMP2000 doesn’t have GPS working after update to 4.4.1
We have a couple of informal reports on this issue. We were not able to reproduce it so far. If anybody has this issue please mail me at dmitry.moiseev@cambiumnetworks.com
I have been having this problem since 4.3.2.1. I have sent in the support file and they told me the GPS sync isnt actually going down. BUT I can see in the GUI top right beside the globe. The GPS "arrow" turns red / I go to Monitor tab then GPS, and I can see all the satellites being tracked, and do not see any indication of sync loss. It just seems like a GUI bug. The really annoying part is PRTG picks this up and I see about 50 GPS sync up/down sensors all day lol. Upgrading to 4.4.2 didnt fix the issue either.
It is not fixed in 4.4.2, still happening on 3000L.
I wonder what's different from my 3000's and 3000L's, to the ones you guys are having problems with? Maybe just luck? or maybe environment? or maybe something else? I don't seem to have any issue with ours. Over 22 days uptime since the last power inturruption, and I don't see any GPS loss (neither GUI artifact or real GPS loss).
'm just about to upgrade some 3000L APs to 4.4.3, and I thought I'd post a screen shot of one of those 3000L's running 4.4.2 with 34 days of uptime, and with the GPS not missing a beat on any of them.
Just to be clear... I certainly don't doubt that the people reporting issues ARE really having issues... but I wanted to also point out that there are people (like me) who are having no GPS issues at all too. :)
While I have several 3000's and 3000L's setting on the bench I've only messed with one of them so far, a 3000L. Just about a month ago I set up a 3000L and wasn't going to use the puck. On the bench, running v4.4.1 it could not see a single Sat but I assumed probably due to the concrete walls. I put a puck on it but still no sats but again I figured maybe the walls. Everything else worked I took it to a small Micropop (10 clients) and swapped out the GPS 1000 AP that had been there for a year or so.
Normally we sync everything over power as I have not had good experiences with ePMP's onboard GPS. However on the Micropops I don't really have room in the cabinets and right now Sync isn't a big deal with them (it will become one eventually so I was really hoping they had their Sync issues worked out with the 3000's).
Swapped the 1000 who's sync had actually been working fine for over a year (with a puck) with the 3000L. No sync, the 3000L can't see a single Sat. So I've got the puck off the 1000 there , I attach it , still not a single Sat... I reboot, still no sats. Oh well, not super important on this micropop and everyone is online and everything looks to be working and I have a huge list of other things I need to be doing so I leave it.
Couple of days later, just out of curiosity I got a new puck and went out swapped them. Still not a single sat in view. That was a month ago and I haven't messed with that AP since. This morning I decided that site would be ideal to test 4.4.3 on. After seeing your post I went and looked and what do you know it's tracking sats ! It's still running on self generated timing but it's seeing sats now !
But ! I went and looked at cacti to confirm that it didn't start seeing sats until I upgraded it this morning and ... well the graphs show that in the entire time since it was installed a month ago it had never seen a single Sat until, yesterday morning at 8:30am when it just suddenly started seeing 17 to 23 sats. I didn't upgrade the software until this morning and I heven't touched that site in almost a month, no one touched it yesterday, there was no reboots, nothing to explain why it just suddenly started seeing sats yesterday morning...
So I won't be turning sync on until I have to and then I will not be using it's onboard stuff.
... well the graphs show that in the entire time since it was installed a month ago it had never seen a single Sat until, yesterday morning at 8:30am when it just suddenly started seeing 17 to 23 sats. I didn't upgrade the software until this morning and I heven't touched that site in almost a month, no one touched it yesterday, there was no reboots, nothing to explain why it just suddenly started seeing sats yesterday morning...
Yes, but you THOUGHT about this AP and GPS? Maybe it's a Quantum entaglement state... where the mere thought of the Quantum state can flip the state? :D
PS. I like your Cacti Graphs... I'm going to have to get your Cacti Template... I don't want to hijack this thread into a Cacti thread - but I'm not graphing Sat/GPS info - and I should be.
I don't know if this is useful info or not but last night the AP in question appears to have rebooted it itself several times and then dropped all clients and stopped responding to HTTP, SSH, ICMP and SNMP. When I arrived at the site to power cycle the radio all the lights on it were on and ethernet was flashing furiously. Power Cycling the radio fixed it but according to the graphs the radio rebooted itself at least twice in the 10 minutes or so after I power cycled it (that was a bit over 12 hours ago and it has not rebooted since). So this morning I looked at the graphs just out of curiosity. I noticed that while we had not rebooted the radio the day the GPS started seeing Sats , it had rebooted itself several time. It also rebooted itself a few times just before it went dark last night.
Also note in the 2nd pic I attached. I zoomed in so the ethernet traffic is a 5 min average. I don't know if this is symptom of the reboots or a glitch in the graphing but I've noticed this in/out spike that coincides with the radios rebooting themselves on customer radios when we had to replace them because they kept randomly rebooting. I do not see these spikes when I reboot or power cycle the radio, only when the radios are randomly rebooting themselves. However there are also normally a huge number of ethernet port errors and there was none on this AP so maybe the spike is just a glitch in the graph caused by the reboot.
Edit: Note that I only have 4.4.3 in the primary bank and 4.4.1 is still in the secondary. So I am assuming the radio is getting completely booted each time before it reboots again else the firmware would have reverted to 4.4.1 ?
Also note the Force200 backhaul radio this AP is conected to has not rebooted during this time. This MicroPoP is a very simple setup : Force 200PTP --> epmp power supply <----> epmp power supply ---> ePMP 3000L ap.
Tonight this same AP dropped all 9 N clients and left the 1 AC (F300-16) client working. It remained this way for about an hour before I was made aware of the problem. I rebooted the AP and all the N clients (and the AC) came back and it appears to be working for everyone again.
After digging around the forums a bit I see that it has been recommend that both banks on the AP be upgraded to the same version of firmware or it can cause problems ? I'm going to upgrade the inactive bank later tonight but for future reference I would like to know if this is still a thing or if it has been fixed.
After digging around the forums a bit I see that it has been recommend that both banks on the AP be upgraded to the same version of firmware or it can cause problems ? I'm going to upgrade the inactive bank later tonight but for future reference I would like to know if this is still a thing or if it has been fixed.
I was under the impression that this was only an issue with F300-16 radios running older firmware from the factory, and that once it had been brought up to a newer version, like 4.4.2... that moving forward, flashing both banks was no longer necessary.
I was told by Cambium that it was (is?) all AC radios - both banks should always be populated with the same firmware.
...which kinda defeats the purpose of having a fall-back bank. BUT, if that is still a known issue, then we should test AC gear that way. For what it's worth - we are not seeing GPS's going offline, and I don't know why we are not affected like others are. However, I can say that we do always upgrade both banks to the same firmware.
I was told by Cambium that it was (is?) all AC radios - both banks should always be populated with the same firmware.
It is not exactly true. The only thing we do recommend is to update F300-16 twice when you go from the earlier firmware to 4.4.0 or later. And more to it do twice only in case you see any instabilities.
I would appreciate if you can keep us posted with the updates on this specific ePMP3000L.
Dmitry
Dmitry,
Even though since upgrading to 4.4.3 I'm plagued with random AP reboots and client radios being randomly dropped it has not stopped seeing/tracking Sats... HOWEVER, this morning I thought I would turn Sync back on. So I set Sync Source back to GPS and set the Holdoff to 1600. The graphing doesn't show that the radio ever loses sight of the sats , Monitor > GPS shows 18 to 21 Sats Visable/Tracked at all times BUT the little "GPS" symbol at the top right of the GUI randomly turns red ever few minutes and may stay red for a few seconds or a few minutes and the status page will say "Sync Down" at the same time. However the Monitor GPS page never shows it losing any tracked or visible Sats.. The randomly disconnected client radios aren't "all" being kicked like you would expect if the AP lost sync and the random client radios being dropped do not appear to be related to the Radio saying it is losing Sync.
I'm rolling back the 4.4.3 firmware update on the clients and AP tonight but I'll leave Sync on to see what it does.
Good tip -I would suggest setting hold off time 864000 on all sectors. That way you wont lose customers if sync is lost and GPS sync will still stay synced with other sectos close by. (Unless frequency is changed)
Good tip -I would suggest setting hold off time 864000 on all sectors. That way you wont lose customers if sync is lost and GPS sync will still stay synced with other sectos close by. (Unless frequency is changed)
While I didn't spend a huge amount of time on it when I tried to set up alerts to let me know when a sector had lost gps sync, I found that none of them would be triggered until the hold off time run out. That means the radio would lose GPS sync but no alarms would go off because running on holdoff is still having GPS sync as far as the radios are concerned. The only thing I could trigger off of was number of sats visible/tracked but then one bug was the radios losing GPS sync even though it could still see/track all the Sats so that was useless. Our solution was to change over to sync over power and that has solved 99.9% of our sync issues.
According to Cambium the timing starts to drift at/around 1600s so it would seem that after 1600s the sector is going to start interfering with neighbors. So if I set it to a huge number I won't know there is a problem until the customers on several AP's are having problems.. So for me the hold off time is really just useful for short/sporadic losses of GPS sync. We use sync over power on everything that matters , it's not a must have on any of the micropops yet.
The SA being enabled seems to be what caused all my issues. It looks like about 3 days before I upgraded the AP to 4.4.3 I had turned the SA on. Looking at the graphs when I turned the SA on the CPU usage jumped from 5% average to 20% average , the next day the two mystery reboots that caused the radio to start showing it was seeing/tracking Sats, then the day after that I upgraded to 4.4.3 (not noticing the increased CPU usage, the two reboots or the radio now seeing sats) so I assumed 4.4.3 was the cause of the constant client drops and AP rebooting.
I had already rolled the AP back to 4.4.1 and it didn't fix the problems but once I disabled the SA the CPU usage went back to 5%' , no more AP reboots and no more clients dropping/reconnecting for more than two days now.
