you'll need a physical reboot, not via the web interface but physically. We had the same problem
21:00 UTC = Midnight in Russia. And it's a leap year. Happy Y2k20!
Rebooting our sync source solved our issue.
Hi @jaggermifter, unfortunately there is no mechanism to reboot the CMM5 injectors from the controller. In practice a reboot would not be sufficient to reset a connected UGPS or cnPulse module, since power to the injector's UGPS port is maintained during the reboot. If you need to reset the UGPS the only options are to disconnect and then reconnect the cable to the UGPS module, or to power cycle the injector itself.
Still having problems with GPS sync from exactkly 3pm CST New Years Eve Day......
Problem seems to be proimarily in new standalone UGPS SW AXN 5.1.1 Rev 8512 connected directly to Aux port on AP.
Jiust did a power reboot and this did not fix the problem....
Is there anyone at Cambium working on this? If so please update.....
Joel, South Dakota
Did you rebooted UGPS or AP ? You have to reboot the UGPS in order to fix the problem. Please let us know if rebooting UGPS fixes the problem or not. Thanks, Chitrang
@Joel wrote:Still having problems with GPS sync from exactkly 3pm CST New Years Eve Day......
Problem seems to be proimarily in new standalone UGPS SW AXN 5.1.1 Rev 8512 connected directly to Aux port on AP.
Jiust did a power reboot and this did not fix the problem....
Is there anyone at Cambium working on this? If so please update.....
Joel, South Dakota
I have rebooted all my devices that were affected, which fixed things.... also unfortunately discovered a 450AP that won't grab it's own sync from the on-board GPS, but that's likely just a bum unit I'll deal with later....
But I am also curious if this issue will occur again. I think everyone is still reeling from this big of an outage, from something so unexpected, that there hasn't seemed to be much chatter since someone discovered what the heck was going on. So I guess I'd like to ask Cambium(or Packetflux for that matter) if they know whether there is a chance this could happen again, and if so, what can be done to fix that?
Any incite would be appreciated.
Also, it’s unclear if Cambium is working on a solution, or is “reboot the uGPS” the official answer? Some of these are in pretty remote locations, it’s not going to be cheap to drive someone around in the winter and visit all these sites for us. I’m still holding out hope that there will be some kind of fix that can be applied remotely
Yes I did a hard power reboot to the GPS and it did not help initially but now this morning has corrected.....
So when all this happened on New Years Eve Day at 3pm CST, I had my AP's all set to Autosync only, so I had numerous AP's drop at the same time at various locations, causing hundreds of customers to lose service New Years Eve at exactly 3PM.
The quick correction was to set the AP's to Autosync + Freerun and a few I just locked on Generate......Had the GPS problem knocked out all of my GPS modules, my self interference would have been intolerable New Years Eve with heavy load.
It now appears that things are slowly correcting on their own, with a few stubborn GPS units that need to be power rebooted.......
While all this was happening, I watched the GPS units constantly find satellites and sync, and then drop out and lose communication, the problem appeared to me to be signal loss or signal quality issue. So what was the problem? Everyone is talking about a quick fix by power reboot but there's been no discussion of what actually happened here at 3PM New Years Eve Day that affected North America, and if there's been signal interference intentional or non-intentional, or hardware failure, causing it...The timing of New Years Eve surely seems to be suspect but it certainly could just be coincidence...
One initial reply was there was a satellite hardware problem, but I've seen no follow up if this was actually the problem or not.
I totally understand this is a complex problem....I'm hoping there's now some discussion on what actually happened and if it's truly been corrected.
I got a private response from Cambium looking for some technical info on my stubborn AP, so I know they're on it. Thank you!! I'm also interested if the cellular PCS industry saw this also.....
Finally, just wondering if the GPS modules are capable of both GPS and GLONASS as I believe they are, but why a problem with one would also block the other....
Am happy things are getting back to normal, but I'd really appreciate understanding more what actually happened....
3 Likes
Just spoke to a fellow Operator just north of me who's reset power and no correction yet overnight, so it still seems to be hit and miss out here.....
Doesn't appear problem is truly solved yet..and hard power reset not a proven correction either...
@tchristle wrote:Does anyone else have to constantly clear their "Idle" SM's every so often to get them back online? We seem to have this issue anytime the connection between the SM and AP drops. With "Sync + Free Run", things seem to be ok for most sectors. Others we have to constantly clear their idle state to get them back online.
@tchristle,
This need to clear idle SMs bug has been fixed in 16.1.1. You can upgrade to 16.1.1 BETA-3 to get the fix. Please let me know if you run into this problem on 16.1.1 or later releases.
Regarding GPS issue, we are continuing our investigation.
Charlie
Hi Joel,
Sorry for the delay in reply. We've been working to investigate this over the past few days with the help of Forrest at PacketFlux as well as the GPS chip vendor.
Currently, the only recommended course of action is to power cycle the UGPS. As you can see here, and we've heard elsewhere, there are scattered reports of self correction but it is not consistent. Also, the flapping nature of the GPS signal that is occurring with this issue means that AutoSync + FreeRun may have issues where it stays on GPS sync since there are some pulses still getting received. In those cases, going to Generate Sync while you work on power cycling the GPS device may help, depending on the location of the AP and other sites and your frequency plan.
Regarding what happened. We are still waiting on confirmation from the chip vendor, but the time of event is consistent worldwide with midnight Moscow time on Jan 1 2020. Midnight in Moscow, which is 00:00 for GLONASS, is 3PM in central time zone in US. The GLONASS system uses a mechanism for time that tracks the number of weeks since the last leap year and that number rolled over at precisely that moment since 2020 is a leap year. The GPS vendor has not validated this to be the cause yet, but it is very suspect and consistent timeline across all reports. It also explains why only chips that support GLONASS are being affected. Furthermore, it looks to only be affecting devices with firmware AXN_5.1.1, which you can verify on the GPS status screen of the AP or the CMM5.
They are doing some tests and code inspection now to verify a root cause, but again, the only current, and likely future, course of action is to power cycle the GPS unit. The main question coming up will if/when it may happen again so we will be able to answer that once we have a root cause identified.
Thank you for your patience and we apologize for the huge inconvenience this has caused, especially on a holiday timeline.
2 Likes
@Eric Ozrelic wrote:
@loganmc10 wrote:
Is there a way to issue a reboot command to a uGPS module that is connected directly to a PMP450? (powered by it's own PoE injector)If the uGPS is power with it's own PoE, then no... you'll need to power cycle it by hand, or if you have it plugged into a remote reboot device.
But if your uGPS is only powered by the AP/BHM from the RJ12 cable of the radio then you can disable the power by the web GUI: "Aux Port Power to UGPS" to Disabled. Then enable it again. This is the SNMP OID uGPSPower (.88) found in whispApsConfig.
But if the UGPS is powered off of its middle RJ45 connector, then will need to be power cycled separately.
@Neil Capell wrote:I have rebooted all my devices that were affected, which fixed things.... also unfortunately discovered a 450AP that won't grab it's own sync from the on-board GPS, but that's likely just a bum unit I'll deal with later....
Neil,
Not sure if it helps in your case, but there are some telnet commands to disable the on-board GPS: "synconboard on/off". This will prevent your 450 AP from getting sync from the on-board GPS. However, this will not reset the on-board GPS. Unfortunately, you will need to reset your AP to do that.
Similarly there are commands "synctimingport" and "syncpowerport" if you need to force your AutoSync to a specific sync source.
The 450 AP on-board GPS does not have an external antenna and therefore is not the most reliable sync source. Sync over power or aux port is recommend even on 450 AP. FYI the on-board GPS is tried last to get sync from, but once sync is achieved AutoSync stays with that sync source.
Hello,
We have verified with the GPS chip vendor that the issue indeed was the GLONASS days since Leap Year rollover that rolled over at 00:00 Moscow Time since 2020 is a leap year.
By now, most devices may have automatically recovered from the incident, but if any issue is persisting, a reboot is the required recovery mechanism. This can be done via the AP if the AP is providing the power to the UGPS/cnPulse.
It only affects devices with firmware AXN_5.1.1 and has also impacted PacketFlux devices that utilize the same chip and firmware. If you use any of those, please work with PacketFlux on the recovery.
For cnPulse and UGPS, we are working with the GPS chip vendor to further identify firmware root cause and will determine next steps as we go forward. For now, on devices running AXN_5.1.1, we can expect this issue to reoccur on the next Leap Year rollover event which is 00:00 Moscow Time, Jan 1 2024 as well as following leap years.
For further updates on this topic, please follow the thread here: http://community.cambiumnetworks.com/t5/PMP-450/GPS-Sync-Loss/m-p/113517#M8891
1 Like
Bringing this thread back from the dead, because we’re coming up on another Leap Year. @CambiumAaron @Charlie , is there any workaround for this aside from AutoSync+Freerun on the 450 line? Kind of a bummer that its been 3 years with no noted fix
Put this together from a different post, if there is no other solution can you confirm this template would work?
{
"cfgFileString": "Canopy configuration file",
"cfgFileVersion": "1.0",
"userParameters": {
"gpsConfig": {
"gpsInput": 5
}
}
}
Used the inspect element output for the 5 for GPS input
@Charlie @CambiumMatt
Hope someone at Cambium answers this question because I really don’t want to spend Sunday Dec 31st rebooting 130 access points for hours on end…
2 Likes
Hello! Yes this is still an issue that would be present on devices that experienced the issue last time during the 2020 rollover. There is no ability to upgrade the GPS firmware on the UGPS devices or the GPS device on the PMP 450 AP.
Using AutoSync + FreeRun should at least ensure that the AP stays up and transmitting. It may cause its own issues however depending on your network topology and frequency plan to have AP’s self generating sync, but it may be better than the alternative for some time.
If you have PMP 450 APs that were affected by this issue, or UGPS/cnPulse devices that were affected and are powered via the AP itself, I’d plan on monitoring them around 3PM Central Standard Time on New Year’s Eve for the issue and then reboot them which would cause a power cycle to the GPS devices.
For externally powered UGPS or cnPulse, you should do a similar monitoring of the AP using this device at that time and ensure some remote power cycle capability is in place.
This is an issue with the firmware on those devices and the GLONASS leap year condition that happens at midnight Moscow Standard Time (UTD +3, MSK time zone), which is at 3PM Central Standard Time on New Year’s Eve in the US. Depending on where you are, adjust the monitoring time window accordingly. Also, since it is a leap year issue, it is expected to occur on these devices on every upcoming leap year like 2028, 2032, etc, if they are still in service.
There is more information being posted soon, if not already, on support site and on here on the forum with more details about affected firmware versions and devices.
2 Likes
FYI there are other posts on this with more detail here:
and here:
Also, there is a way to schedule the reboots for devices via cnMaestro that maybe you would consider scheduling for shortly after midnight Moscow Standard Time on 1/1/2024.
To schedule reboot of multiple APs via cnMaestro:
Go to “Inventory” screen, click “Apply Filters” button, select “PMP AP” in “Type” to select all PMP APs. Then click “Actions > Schedule Reboot” and schedule the reboot for 5 minutes after new year in Moscow.
2 Likes