Anyone noticing issues with GPS Sync today. 3 of my 4 towers all lost sync at about the exact same time at 1pm PST. The towers are in different cities and there's no inclement weather here, it's blue skies all around. The towers all have different radios too, one tower is all 1000, one is 2000 and one is 3000.
I am pretty sure that all of the affected radios are on the same firmware AXN_5.11. The tower that didn't go down the GPS firmware is AXN_1.51_2838. The only thing that makes sense to me would be something to do with the date change but even then GMT 0 is still 3 hours away from the date change,
Ditto - There is another thread on this as well showing it spanning time boundaries to all be about the same time. Mine have come back and apppear to be steady.
I've only rebooted the one. I kicked the holdoff time way up to get the customers back online asap. I probably won't reboot everyhting until after hours. The one I did reboot is still currently up.
It would be nice to see a Cambium response regarding this. I saw a discussion on reddit that people using CMM were also having GPS dropouts at the exact same time. No chatter whatsoever from the UBNT camp about similar issues.
It would be nice to see a Cambium response regarding this. I saw a discussion on reddit that people using CMM were also having GPS dropouts at the exact same time. No chatter whatsoever from the UBNT camp about similar issues.
There was a official Cambium response in another thread:
"We are currently looking into the possible cause of the GPS interference/outage we are seeing with GPS sync, currently there seems to be a GPS interferer or outage affecting the uGPS receivers and PMP450 APs with integrated GPS."
In addition, the owner of Packetflux had this to say:
It seems like something happened around 2PM MST today which caused (at a minimum) certain Globaltop/Sierra Wireless
GPS modules to quit receiving signals from the GPS constellations. Because these modules are pretty well respected as far as the quality of
the timing signal (normally) and relative low cost, they tend to be integrated in a lot of devices, including my products, the official Cambium
Products, and I understand the gear from other manufacturers.
So far, a power cycle/reset of the GPS module seems to solve the problem.
Appears to have affected 2 of my Packetflux (SyncBox Junior) sync injectors also. A little before 3pm CT 12/31/2019 one of our sites lost gps sync. I had to drive out to the site and unplug/plug the SyncBox and sync came back up. Then at around 6pm CT that same day I had another tower do the same thing, had to drive out and do the unplug/plug and it sync came back on.
It doesn't appear that this was anything with the GPS Firmware, or anything limited to just Cambium devices.
On Twitter: @GalileoSats said:
"It seems like something happened around 2PM MST today which caused (at a minimum) certain Globaltop/Sierra Wireless GPS modules to quit receiving signals from the GPS constellations."
We are experiencing network wide GPS failure on our Cambium devices. Symptoms include visible but not tracked satellites and a complete inability to regain synchronization. Switching sync sources does not appear to improve the issue with any measurable longevity.
It all started with one of our sectors manifesting curious GPS dropouts. It soon spread to our entire ePMP network (around Tuesday).
We have 12 tower sites spanning hundreds of miles in Nebraska all exhibiting the same GPS failure. I suspect it to be environmental, or at least external, as we are experiencing the phenomenon across a multitude of firmware and hardware versions as well as varied GPS sync sources. We use Packetflux for the majority of our GPS timing.
As soon as we are investigating the issue it will be great if you can help us as well. Please raise a ticket with our support team which is already instructed what date to collect.
While investigation into the root cause continues with the GPS chipmaker and others, the symptoms and recovery recommendations are contained in the Field Service Bulletin below. While we believe that this was a one time occurance due to a satelite maintenance procedure performed on news year eve, the recovery procedures in the FSB are recommended by Cambium in the slight chance there are still anomolies being seen.
So a bit more information, I did have some ePMP radios that did not go down when the event occured and I think the common denominator was that at least for me any sector with the older round pucks we did not lose GPS. But every sector with the newer square pucks did lose GPS.
As of right now I have one ePMP 2000 radio that seems to have lost its mind. It shows 0 tracked 0 visible and won't display lat / long. This started after the issues on NYE. The radio is on the latest 4.x firmware. Right now I have source set to internal to keep the customers up. My other 2000's that are running 3.1.5 the GPS is working. I'll look at this bulletin and see if it helps me to recover that radio.
So I just looked over the bulletin and it apears the recommended procedure for ePMP is power cycle. For the one radio that still does not have GPS this didn't help. I had the radio powered down last night for a good 10-15 minutes and the GPS still won't come back. At this point I think my only other recourse is to try to downgrade to 3.5.1 or we have to do a climb to replace the puck.