I am re-publishing a Field Service Bulletin from 2020 to highlight the potential for GPS synchronization loss when using Cambium uGPS, cnPulse or ePMP 1000/2000/3000 APs using integrated GPS. While many of you may have upgraded your sync hardware since 2020, if you are using any of these devices that are running firmware version AXN5.1.1 or lower, you may lose synchronization on leap year 2024 (January 1, 2024). Please reference the Field Service Bulletin below for details.
Is there any way to avoid this if running the most current firmware, or are we in damage control mode like in 2020? My co-worker had a ticket #196827 and I believe we never heard back. Previously our PMP450m and PMP450i radios were affected unless they were utilizing a CMM5 on 1.14
I see the ticket you are referring to. Looks like you experienced network wide sync loss in the afternoon of December 31, 2019. There was a lot of dialogue between your team and the Cambium support team which ultimately culminated in power resets of the uGPS units in your network and the support team providing the Field Service Bulletin that I reference in my forum post.
Unfortunately, running the most current firmware on the AP will not fix this issue. If the problem is experienced again, a power reset of the AP (if internal sync used) or a power cycle of the uGPS or cnPulse will be required if using an external GPS sync. Proactively power cycling the unit will not fix the problem. Only if you lose GPS sync should the APs or GPS units be power cycled. I apologize for the inconvenience this may cause you again this leap year.
Thank you for following up Chuck. So did the investigation listed in other threads yield zero results? I anticipated some sort of update over the 3+ year period. Will there be further investigation into a firmware fix, or are we just going to need to prepare if we still run 450’s by the following leap year in 2028? (Long time from now, just curious)
2nd question. Will Cambium be issuing a new FSB for this known issue, and potentially sending out a notice on the mailing lists? Realistically this will cause service disruption for nearly every 450m, 450i and any device running AXN_5.1.1 Seems fairly widespread unless I am missing something.
The investigation found that the third party firmware on the GPS chipset was the source of the problem. It is not easy to upgrade that chipset, unfortunately and hence there was no easy solution identified. I am republishing the FSB from 2020 to the forums and our marketing Director is also splashing this out on all the social media to get the word out. I have also emailed every person who opened a support ticket with us back in 2020 to get the word out as well. To my knowledge the internal sync for the PMP 450 is not affected and therefore there should not be any problems with 450 APs running internal sync.
Thank you for running this to ground for us @CambiumChuck . We will prepare, and plan to upgrade our CMM equipment to an unaffected version prior to 2028.
1 Like
This has affected us today. All our CMM GPS was intermittent showing date / time all over the place. This has caused incidents across our production environment. Is cambium providing a fix for this in the near future to alleviate this happening again?
2 Likes
I mean they’ve got 4 years to fix it lol, but they had 4 years, 4 years ago
1 Like
I’d sure love to avoid this issue again in another 4 years time. We have a number of cnPulse/uGPS running the AXN5.1.1. I’d love a solution from Cambium where we could upgrade the firmware on these ourselves, or send in for free factory upgrade to avoid having to spend thousands to replace with newer units as these are known to exhibit issues every 4 years. How about it @CambiumChuck ? Can we get a solution other than rip/replace these units to avoid this issue again in 4 years time?
2 Likes
31/12/2023 22:00 CET (midnight in Russia) and we’re suffering from GPS loss.
Rebooting uGPS from "AUX port) is easy and packetflux connected to Rackinjectors too.
The problem is 450i or 450m that have uGPS powered from an indipendent power supply.
In that case we need to reboot them manually. Correct?
This years issue actually went fairly smooth for us. I scheduled reboots for our 115 450i’s that were using cnPulse devices. Scheduled the reboots to start at 1:05PM PST, and the last one issued at 2:10PM. Only 1 site (2 AP location) was still experiencing issues so we swapped it to Autorun+FreeSync. Fortunately it self corrected between yesterday and today, and was receiving sync this morning.
Also our 450m’s using CMM modules were unaffected. We updated them to newer models over the last 2 years or so.
Hope everyone is recovered from this. Glad we pushed the issue on here so a FSB got released via email and their facebook to reduce the impact for everyone. Happy New Year All!!
3 Likes
We are glad that the issues has been solved with a simply reboot
1 Like
So, we run the uGPS with CMM5. Is there a way that you can control the power to the uGPS port and perform a reload remotely? Currently we had to go out to each CMM5 and manually pull the uGPS cable on each one individually. Ie, the CMM5 is a power injector that allows you to remotely enable and disable power on specific ports to the PMPs, can we make that also occur to the uGPS?
Kanzi331,
Unfortunately, recovering the uGPS requires a hard power cycle of the uGPS itself. Given that the CMM5 doesn’t provide the power to the uGPS, you must manually power cycle the uGPS itself. Power cycling or rebooting the CMM will not resolve this issue.
Chuck
Can you not send a command from the CMM5 to the Master Injector to drop power on the uGPS port and enabled it again? Much like the Master Injector does for the PMP APs?
Or, are you able to console into the Master Injector from the CMM5 control module cli? And do a reboot of the Master Injector?