We've noticed that switching to "Generate Sync" has been resolving the issue. If you have cnMaestro here is a JSON template file that can push that out:
To shed some light onto the problem, and a workaround, but not the underlying cause:
It appears that something either in the firmware of the GPS receiver module, or something transmitted by the GPS constellation around 2PM Mountain Standard Time today caused the GPS receiver modules commonly used by Cambium, PacketFlux, and others in recent products to lock up and quit receiving GPS. This seems to be at a minimum affecting certain ePMP radios, certain uGPS receivers (like often used with the CMM5), and certain PacketFlux GPS Receivers. (These all share a similar or identical GPS module)
It appears that power cycling or otherwise causing the GPS receivers to be reset resolves the issue, at least for now.
For units which are getting power from the AUX/uGPS/Timing port, you can usually disable uGPS power on all radios the unit is attached to to get the power to shut off. Then re-enable. Alternatively, rebooting all of the radios at the exact same time will often work as well.
For PacketFlux products, most include a way to cycle the GPS remotely if you have a management device (aka RackInjector or SiteMonitor Base Unit).
To shed some light onto the problem, and a workaround, but not the underlying cause:
It appears that something either in the firmware of the GPS receiver module, or something transmitted by the GPS constellation around 2PM Mountain Standard Time today caused the GPS receiver modules commonly used by Cambium, PacketFlux, and others in recent products to lock up and quit receiving GPS. This seems to be at a minimum affecting certain ePMP radios, certain uGPS receivers (like often used with the CMM5), and certain PacketFlux GPS Receivers. (These all share a similar or identical GPS module)
It appears that power cycling or otherwise causing the GPS receivers to be reset resolves the issue, at least for now.
I can confirm that rebooting all of our packetflux rackinjectors resolved our issues. Furthermore, all of our PMP AP's that had autosync - freerun enabled, and ePMP with long hold off timers had very few issues. I didn't even know we were having issues until I actually looked at the event logs.
Yeah unfortunately I don't have anything like that. I'm not going to send somebody out on New Years to do that, so I'll let them run on Free Run for a couple days, hopefully Cambium can do something to sort this out without needing a physical reboot of the GPS modules.
Does anyone else have to constantly clear their "Idle" SM's every so often to get them back online? We seem to have this issue anytime the connection between the SM and AP drops. With "Sync + Free Run", things seem to be ok for most sectors. Others we have to constantly clear their idle state to get them back online.
Confirmed we are having the same widespread issue in New Zealand - across all platforms using the uGPS.
It appears only on the units that have GLONASS from what I can tell, as all of our older Canopy GPS only uGPS appear to be OK (must say the early GPS only platform is solid, GLONASS editions have been problem after problem it feels).
Our older PMP450s reverted to their onboard GPS, and we've set the rest of the network to AutoSync + Free Run for now.
If rebooting the uGPS is the solution that is OK, but how robust is it likely to be? Is the issue going to reoccur?
Before I send our engineers out with the mission of rebooting our uGPS GLONASS devices we will wait for more information. Hopefully some more information becomes available soon so we can make an informed decision.
I sent technicians out to power cycle UGPS and APs and it did not resolve the problem. The only answer so far for me has been to set the AP to generate sync.
With a GPS power reset on site, we solved the problem. We had a problem with the GPS software version AXN 5.1.1; with software version AXN 3.20 GPS synchronization is fine. We are waiting for the definitive cambium solution.
We see the same here. We have six CMM5 with uGPS GNSS_EXTRF_9600 with AXN_5.1.1 that lost all the satellites, but one with AXN_3.20 that lost about 10 satellites and still working.
All of our CMM4 is working fine.
We also have a PMP450m with uGPS connected that is working. This one have AXN_1.3.
We have powercycled two of the uGPS with AXN_5.1.1 on site and those are working fine now.
One of the AP had to be rebooted for the SM to be stable.
I have one device, it is a prototype micropop omni that has built in GPS that will not stay synced. I've been running it on generate sync since Tuesday. I tried changing back t his morning, but no good. Everything else has been solid.