ePMP GPS sync leads to madness!

To whom it may concern,
We have several dozen ePMP AP2000 and not a day goes by that at least one does not stop broadcasting due to the loss of GPS signals.
We migrated to the Cambium for the reason that it is presented on the frequency space savings that GPS sync allows.
Migration was not cheap either!
How annoying this problem with GPS sync is to migrate to something else but I can’t all the money I had ended up at Cambium Networks…
Reboot of the device, upgrade or downgrade of the firmware usually restarts it until the next blockage.
There are no rules, rain, sun, fog or wind.
It doesn’t happen by any pattern.
At first I suspected it was up to the GPS antenna. I replaced a few antennas I got with ePMP and connected Ubiquity devices there to work perfectly ,GPS is constant.
As far as I understood it is not up to the antennas! The flaw is in the software or the rest of the GPS-related hardware
The problem has been going on for years.
As far as I have seen on the forums, several topics have already been raised regarding this problem and it is high time to correct it

Kind regards,

2 Likes

Hello,

Could you please let US know GPS FW version running on your ePMP2000s?
This can be checked on Tools->Software Upgrade page in GPS Firmware section.

Thank you.

Would you kindly send over tech support log files to me via DM as well. We have a possible line of sight to something internal and would like to verify this on your setup.

I didn’t mention anywhere that they are AP2000 but you are, it says you know about the general GPS sync problem !
Everything became clear!
Regards.

According to this picture, everything is ok, but GPS sync will crash :frowning:

Looks to me like you mention ePMP 2000s…
Cambium is asking for your help in determining the issue and you want to be a smart alec.

Yes I did now, but I was not in the technical support a few days ago.
I don’t want to be smart but happy!

For what it’s worth, we’ve been running Cambium ePMP2k radios for 4 years and GPS sync isn’t a problem we see very often, except when birds or other critters chew up the gps cable. I’ve noticed this happens more with Cambium pucks than UBNT pucks. There must be something slightly different in the cable jacket chemistry that attracts them.

FYI, you do not need to use the puck if your radio is not inside of an additional metal shielded compartment, like some antennas include.

In most of our deployments, we just do not install the puck and we get good tracking performance without the frustrations of damaged pucks every few months.

Another thing you can check is your GPS timeout parameter. I think it defaults to 120s, but you can set it higher so it will operate through poor conditions. We actually set ours to 7200s, which if they truly lose sync will cause performance issues, but they generally regain sync on their own and operate fine. There might be some specific timeout that works really well with less exposure to causing poor performance problems in the event of an issue that cannot be recovered without intervention, we just didn’t spend the time to figure out the magic number.

GPS detects 10 or 15 satellites with an SNR 36dB or higher, then everything is ok and works perfectly.
When the SNR is 32dB everything sometimes crashes. And why?
In Ubiquity devices, it associates a satellite even when the SNR is 20dB. With the same pack antenna
The SNR margin for satellites is not very ok.
This is really annoying.

Hello zica49,

I have experienced this and l have have a mixture of all ePMP AP models on my network. Please change GPS synchronisation time on GUI to the maximum so that GPS does not sync often. This fixes the problem permanently.

If the has to do with SM’s , please consider raising the SM a bit more. Maybe an extra metre or two and this will be fixed.

I ask Santa to deny gifts to Cambium engineers who do not solve the problem of losing GPS synchronization.

We use packetflux sync system and skip the onboard gps, but we do have a few sites that we have tobuse the onboad gps as the 3000L does not do external sync. If you have the new square pucks and oriented them correctly (not laying flat but stuck to the tower leg with the cable is down) then you shouldnt have too many gps issues, a longer hold time is also a requirement, we found anything more than 7200 is good but dont go too high as these radios have horrible clock drift which is compensated by the gps/sync system.

Thank you for the provided information.
I was interested to see GPS chip model and its FW version.
I can see it’s gen2 GPS chip and it’s running the latest recommended version.
We lived years with no issues reported for that chip.

Do you have any other devices running GPS chip on the same tower?
From the provided screenshot I can see very few satellites tracked(7). AP always needs at least 4 satellites tracked to be synced.

Thank you.

This AP lost four hours of GPS time yesterday
I didn’t touch anything, I came back.
Regards
Read satellite from a minute ago.

On the same tower we have 10 APs-

All are with GPS sync

There are no rules!

It loses GPS sync even though it hears enough satellites with a good signal.

Sometimes one, sometimes three never all.

Setting GPS sync maintenance to maximum time may solve the problem on two APs.

On ten devices, some of which are in the shade and some in the sun, it doesn’t work very well. Due to the difference in temperature, the oscillators start to slide.

An internal oscillator with 10 ppm can hold it for a few minutes, up to half an hour.

After that, the performance on a device that has lost sync due to the temperature slip of the oscillator begins to fall sharply, and causes a problem for the AP and also for others.

We tried all the logical positions of the external GPS antenna. The problem still exists.

Two antennas next to each other one maintains GPS for ubiquiti Airfiber (GPS signal is three years j constant)

Second on the Cambium AP 2000 (record 3 months with GPS holding in one piece).

Scenario…

He calls the client and says I don’t have internet or network service.

We look at its primary AP lost GPS sync, catastrophe and secondary also.

Even worse disaster second secondary ordered a huge number of clients once and 120.

That doesn’t work at all then.

I look at the other APs on the same tower, they hear the satellites and the GPS constantan.

Reboot, firmware change restore it and it works.

Then wait for the time when the clients are asleep.

All those clients who migrated to another AP manually one by one to return to the preferred ap, it’s a lot of time.

Of course everyone is asleep but their frustrations over the disconnection have already poured out on me.

Then a similar scenario is waiting for the AP, which worked ok this time.

And why all this?

The magic solution in writing a firmware that should be done would solve a lot of things.

  1. To take a closer look at that GPS sync.

  2. To add an option in the radio settings: “In so many hours scan the spectrum and search for your primary AP and associate with it” or do it every hour two.

That solves the problem quite a bit.

Perfection would be if it was done like this.

Because all APs on the same tower are an integral part of the same network.

Or dislocated again in the same network are irrelevantly connected one way or another.

scenario.

The AP that returned to the broadcast received GPS or power should be sent over the network identification to other APs. hello here I am.

Other APs forward this to ether subscribers and they know it is primary in function.

Then I could sleep too.

You need to add this as an option that will bring relief not only to me but to many others as well.

We lost patience. It no longer occurs to us to talk on the phone with anyone from the technical support. What is the firmware, what antenna, what do you have nearby, etc.

The problem remained, and every now and then we lose customers due to the loss of GPS sync.

And of course it costs money.

Best regards

^^^ This ^^^ should be done anyway - regardless of GPS issues or no GPS issues.

  • The Preferred AP list should be populated with 1st, 2nd, 3rd choices – with simple click on/off rules (or minutes up/down rules) for how the SM should fall over to the 2nd and 3rd AP’s, and for how/when it should climb back up to the 1st preferred AP.
2 Likes

Yes, that’s right
If one of the APs gets into a problem with bandwidth due to interference or a large number of clients, it rates itself with a rating of 1 to 10.
1 = bad (I have interference)
10 = good (no interference)
Because they are in the same network exchanges via Ethernet ratings and send to subscribers.
Then one would know the state of the APs and whether they work at all.
With some of the beacoms the subscriber gets a report for example that his secondary AP is rated 9 and the primary is currently at 6.
The subscriber is dissociated and switched to secondary.
Leave as an option in the radio setings.
Should every subsriber do that or just some who have been granted elite status.
Then Cambium would have a brutal solution.
But that information should be shared by APs when they are on the same “eternet”.
The subscriber should not go into spectrum analyzer mode but only associate with the one that identified with the best score.
Regards

Well yes, there would be a lot of complicated algorithms which could give awesome results, or could be a nightmare.

BUT, simply put - the SM’s should simply be capable of automatically fall back, and also climb up the “Preferred AP List”. Right now, they can fall down to the 2nd AP, but no way to get them to climb back up the ladder (other than manually "Deregister"ing them from the secondary AP)

So it would be fairly simple to have a checkable option on each secondary / tertiary AP in the list, which says “retry higher AP every XX minutes”. That’s really all that needs to be done to have this feature functional and helpful. Which could help mitigate GPS issues, but also help for all sorts of maintenance, or blown power supply, or upgrades, or whatever the WISP wants.

1 Like

Of course, that would take a big step forward
Snmp is the right solution for this.
Upon activation, the AP sends information to other APs that it is ready to accept back its clients.
APs in the network send beacom clients and they know they can migrate to the preferred ap.
There are no unnecessary attempts to search the spectrum and interrupt the transfer.
Regards