ePMP 1000 GPS (lite) AP just lost all GPS

I have a 1000 GPS (lite version) AP that has been runinng for days on GPS sync. Suddenly today, it lost all GPS capability and does not show any tracked satellites. I have the GPS puck attached to the AP and the puck is mounted on the top of the tower with clear view of the sky. A nearby AP has no problem seeing dozens of satellites.

AP is running 3.2.2 with GPS firmware: AXN_3.20_8174

I have not had a chance to re-boot the AP since it was changed to flexible and has clients using it.  But, I'm curious why the GPS just died. The AP is only 2 weeks old and been in service for 8 days. Makes me a bit worried to switch back to GPS if a reboot brings back the capability...

Not sure if it helps tech support but here is the syslog dump from that time:

Feb 11 17:38:54 Belvedere AP - 40 MHz kernel: [617654.640000] No sync pulses have been received for 300 seconds. Sync Holdoff timer has expired. Transmitter shut off.
Feb 11 17:47:38 Belvedere AP - 40 MHz DEVICE-AGENT[2503]: rx_config_notify_cb: Send MSG_CONFIG_CHANGE_NOTIFY to cnMaestro
Feb 11 17:47:40 Belvedere AP - 40 MHz DEVICE-AGENT[2503]: rx_config_notify_cb: Send MSG_CONFIG_CHANGE_NOTIFY to cnMaestro
Feb 11 17:47:40 Belvedere AP - 40 MHz kernel: [618180.680000] GPS Sync Lost. (00:47:40:520871)
Feb 11 17:47:40 Belvedere AP - 40 MHz kernel: [618180.680000] GPS Sync Lost. (00:47:40:525309)
Feb 11 17:47:40 Belvedere AP - 40 MHz kernel: [618180.750000] GPS Sync Lost. (00:47:40:593177)
Feb 11 17:47:44 Belvedere AP - 40 MHz DEVICE-AGENT[2503]: rx_config_notify_cb: Send MSG_CONFIG_CHANGE_NOTIFY to cnMaestro
Feb 11 17:47:45 Belvedere AP - 40 MHz kernel: [618185.500000] GPS Sync Lost. (00:47:45:338827)
Feb 11 17:48:41 Belvedere AP - 40 MHz DEVICE-AGENT[2503]: rx_config_notify_cb: Send MSG_CONFIG_CHANGE_NOTIFY to cnMaestro
Feb 11 17:48:43 Belvedere AP - 40 MHz DEVICE-AGENT[2503]: rx_config_notify_cb: Send MSG_CONFIG_CHANGE_NOTIFY to cnMaestro
Feb 11 17:48:43 Belvedere AP - 40 MHz kernel: [618243.560000] GPS Sync Lost. (00:48:43:406152)
Feb 11 17:48:43 Belvedere AP - 40 MHz kernel: [618243.570000] GPS Sync Lost. (00:48:43:410600)
Feb 11 17:48:43 Belvedere AP - 40 MHz kernel: [618243.630000] GPS Sync Lost. (00:48:43:470268)
Feb 11 17:48:47 Belvedere AP - 40 MHz DEVICE-AGENT[2503]: rx_config_notify_cb: Send MSG_CONFIG_CHANGE_NOTIFY to cnMaestro

I'll say we're having similar issues on one tower with ePMP 1000 (Full) which is the same hardware you're using. They lose GPS sync for hours or days at a time with 0 satellite tracked. They eventually work again and track 8-9 satellites but then they lose sync randomly.

We use GPS sync and frequency reuse so when they lose GPS, they stop transmitting thus we have downtime. I have to turn on flexible mode to get customers working again. This is the only tower so far with the 1000 that lose GPS entirely. Not sure why. We installed the radios and GPS antennas the same as all our other towers so far.

Our 2000 have been fine so far with no loss of satellites yet.

Cambium support has not been very helpful yet. They just say to check the cables and replace the antennas. I have emailed and found other WISP's who have the 1000 who also have GPS issues with them. Cambium has recommended they switch to 2000 AP's and new GPS antennas. Similar issues with GPS randomly losing tracking to all satellites with no solution. So I know we're not the only one.

I wish Cambium would acknowledge multiple WISP's are having the same issue and let us know how they can help us short of replacing our hardware.

Sort of glad to hear I am not alone... Ugh. I'm not upgrading this to a 2000 AP. We use a bunch of TwistPort antennas so it would be a pretty big ordeal to change out.

Am I wrong when I thought there was a built in GPS reciever (in the AP) that could be used instead of the puck?

Luckily, PRTG to the rescue. I can monitor this every 30 seconds for the next few days and see if there are any trends on number of satellites tracked before I go back to using sync:

http://prtg.auwireless.net/public/mapshow.htm?id=2722&mapid=7F91C417-FA45-4D38-A3A4-A00AE43878CA

Can't live without that software!

Cambium has document the issues with GPS (you find them commenting plenty on the forums)

Some of the carrier LTE will cause interference with them. You can use a cmm or packet flux timers to remedy the problem. We experience the same issue only when we colocate with Verizon here. No other carriers. It’s a simple matter of resonance on the GPS that the basic GPS receivers can’t deal with.

Packet flux snyc over poe is the lowest cost issue. If your running only 5ghz and of the snyc aps that also read from glnos can work correctly. (Some new shipping 1000s have them all 2000s do as well)

I have seen behavior like this on ePMP 1000 on two sites. It works, sees 8-9 satellites, than suddenly 0 and after random time (minutes,hours, days) it jumps back to 8-9. On one site the outages were shorter and after we set the Synchronization Holdoff Time to maximum value, it was enough to keep the AP running. On the other site it wasnt enough and after it switched itself off few times, we changed it to Flexible mode.

I am not aware of any LTE or other telephone antennas around the sites, but I believe that the problem is somehow location dependent.

I have the same problem with an ePMP 2000 Lite
The next day of installing it I stopped transmitting the WiFi and lost the GPS, my distributor sent me another spare unit, but I am still waiting for the licenses of the Elevate and for the full version, more than a week ago and still are not the licenses, is Desperate, I have customers without service.

Is it possible this is related to the “lite” version of the APs? I believe it’s the same exact hardware, right?

We started seeing this in November at one site. It has grown to several others since then.

Completely unreliable in GPS mode.

Lite is the same hardware as Full, just a license change is all.

We have NO cell carriers on the tower that is having issues. The closest cell tower is 3.13 miles away and it is Verizon and has 700mhz LTE. But this should be way too far to affect GPS for these radios. 

The only thing on the site we're on besides us is some 800mhz Motorola two-way gear and some 6 GHz backhauls. Nothing else except 2.4 and 5ghz from us on this site.

From what I understand, if you have 800MHz gear there, the 2nd harmonic would be 1.6GHz, which is what the Cambium GPS operates on. I've been told that this could interfere greatly with the GPS since the pucks are passive and don't have much gain at all.

Someone could correct me if I'm wrong though.

1 Like

Well, my AP with the failing GPS is mounted on a customer house as a mini-pop. It is far away from any cell towers. The only other radiation near it is a Mimosa B5 Lite 5GHz backhaul radio and it is shielded from the AP.  The AP is brand new and has the newer GLONAS GPS puck.

I've been tracking it since it failed. It has never gone below 6 tracked satellites since it failed 2 days ago - but I am still running in flexible mode until I can get a handle on the problem.

Have you tried to remove the puck and let the internal antenna take over ? Maybe something wrong with the puck.

Not yet. It's a long climb up to this with a couple ladders...  I want to give my monitoring software a week or so to track it and see if it fails again.

If the AP has a very clear view of the sky, is there a reason not to use the internal GPS chip (and not the puck)? Finding a place to get that puck to stick on a pole mount is a bit challenging sometimes.

The puck is better gain and can be moved to see the sky better, the internal can work fine, just don’t want to have anything over the top of the AP

1 Like

The GPS receiver in the ePMP2000 is the same as the later revision of the 450 AP. More accurately, it's a GNSS receiver (GPS+GLONASS). I don't remember what's in the ePMP1000. Might be GPS only like the first spin of the 450AP. This is what Forrest at PacketFlux is currently using in his products. They're all from GlobalTop and are pretty reliable, but sometimes they just get confused a need a hard power-cycle to get un-stupid. Noise can definitely be an issue. Multi-path is another problem.

I agree with Chris, use sync-over-power if you can. I'm sure Cambium would like you to buy and use a CMM, but a PacketFlux SyncInjector or PowerInjector+Sync module and a SyncPipe Basic is <$300. This is the best approach because you can have the receiver down at the base of the site away from radiating devices. We have very few issues with this configuration.

2 Likes

It happened again. Lost all satellites for exactly 7 minutes. I can now track it with SNMP.  Went from 9 satellites to 0 for seven minutes then right back to 9.

It has happened twice now and there is one thing in common... Both times it happened, it had just started to lightly rain. These are the only two times it has rained at all since the AP went up.  It's been plenty cloudy so I don't think it's a reception issue. I think it is water related.  However, it is still lightly raining and the GPS satellites came back to 9 so if water is causing this, it is not staying bad until it dries off.

The AP is fully exposed. It is connected to an omni with short pigtail cables. The omni is mounted to the top of a J-Bar arm and the AP is attached to the angled part of the J-Bar.  The access door for the cables is facing down (on the 45 degree j-bar arm).  Water can't follow the cables into the AP due to gravity. Water could potentially get in the small hole where the cables enter but could not get more than a few drops before leaking back out due to gravity so I don't think connectors in the radio are getting wet.

Here is a shot the installer took during the install.  The GPS puck is on top of the J-Bar. We install a piece of metal strap across the hole at the top of the J-Bar to give the magnet something to stick to (that is the silver band you see).

We have a couple others installed the same way that have never had an issue - but the rain thing has me wondering.

your GPS puck orientation needs changed, face it upward facing so your coax is coming out of the bottom of the antenna, rather than laying flat.  you will see 10 to 15 increase in GPS SNR from that change.     and likely more sats tracked.   

most of my APs see 4 or 5 sats above 40.   If for some reason your issues are related to weather, that extra SNR would make a difference for you.  or if water is some how bothering your puck, changing its orientation might resolve that as well. 

I'm confused now... When I put the GPS puck in the holder at the top of the Cambium sector, it faces up - or what I call up. The part with the magnet faces earth. You're saying stick it to the pole so the cable goes down and the magnet is at 90 degrees to earth? Does it matter if the large flat part faces south, north, etc?

I will admit, the SNR for GPS on that AP sucks. However, I have 5 other APs in the area, all with pucks in the same orientation as that one and all with great SNR. Maybe a bad puck? Or the antenna connector on the AP is problematic...

If you’ve got other pucks sitting like that, with much better SNR, something is probably wrong with that puck. I’ve noticed that laying them flat, with the large surface facing skyward is not as good snr as the puck being flipped 90 degrees so the flat surface is facing parallel to earth, with the coax exiting the antenna headed downwards to the ground. That may change depending on the part of the world your in, I’m not sure, but try it with one of your APs. The GPS snr seems to update kind slow so wait 20 to 30 seconds to see the change. US Midwest is where I am, and every time I’ve flipped a puck flat, like you have, I’ve lost an average of 10points snr.