GPS outage incoming on New Years Eve/New Years day

Products PMP
1

Tried posting on old threads and did not receive a response. With Jan 1, 2024 a mere 6 weeks away and no resolution for us using PMP gear with the AXN_5.1.1 FW for our GPS, what is the word from cambium?

The last I can see being actively discussed was back in 2020, and radio silence ever sense. While we have a workaround we need to implement on a few hundred 450m/i devices, some sort of update would be nice. 2 years and no noted solution is disappointing. Furthermore re-issuing an old FSB rather than making a new one won’t show up on everyones radar. So please, give us your customers a real update with any progress or road blocks, and issue a new FSB so customers can actively prepare. @RayS @CambiumChuck

2

Also, if there is no official solution, our workaround is to upload a new template to our few hundred 450 AP’s to switch them to Autorun+Freesync. Not a great solution, but better than a potential outage on new years eve, and thousands of pissed off customers.

Below is the code options from an ap, and the code for the template

image

{
          
           "cfgFileString": "Canopy configuration file",

            "cfgFileVersion": "1.0",

            "userParameters": {

                        "gpsConfig": {

                                    "gpsInput": 5

                        }

            }

}```
1 Like
3

Chuck responded to my other comment. Sounds like we’ll be implementing the code above (as long as I didn’t mess it up) as a precautionary measure to avoid outages.

4

Another correction, according to a previous post from Cambium, swapping to Autosync+Freerun during this event has the potential to stick in Autosync if even a small GPS signal remains.

Grasping at straws trying to avoid having to reboot a few hundred devices on new years eve. If needed I can just start a support ticket. But 3+ years and no fix really does not build my confidence to pursue the new line. Left out to dry on this issue if it comes up this leap year.

5

Are you using cnMaestro? If you are, you can just schedule an automated reboot of all the AP’s.

6

Good idea. So the next question would be, what should the extent of the disruption be and at what time should the reboot be issued? For us it would be 1PM PST on 12/31/23, Schedule it for 1:30, 1:05?

My main concern in all this is how little information has come out in the last 4 years, and no real fix aside from buying different sync gear. This is really quite frustrating. Thought I left this kind of “just sweep it under the rug” mentality behind when I stopped using UBNT lol. Also @Eric_Ozrelic they should really just give you a Cambium badge for the forum, recently you seem be the main source for info on here.

1 Like
7

Assuming that all your AP’s are running in autosync + freerun mode… if/when they lose sync, they’ll still keep timing for awhile… eventually the timing on some will start to drift and that’s when you’d start seeing possible self interference issues.

As to our plan… all of our ePMP GPS has been updated, and all of our PacketFlux GPS has been updated. We’re going to just wait till the leap year event happens and just check a sampling of AP’s and we also have some latency monitoring software that we can check to see if anything is causing self interference. If we start having noticeable issues, we’ll issue reboots on a case by case basis… and at worst, the mass reboot… if no issues crop up, we’ll probably just do a timed mass reboot sometime late at night/early morning when it’s least disruptive.

1 Like
8

We’re all 450m and 450i’s here. Some of our sites run CMM5 and those were unaffected last time this came up. Unfortunately there is not an update for the internal GPS FW for the 450’s. That’s the real bummer considering the price point on these radios.

I will forge ahead with the swap to Autorun+Freesync. There was a post describing an issue where if there is even a blip of remaining GPS signal that the AP may stick with Autorun and have issues. We’ll just hope for the best. Still irritating that it appears nothing was done to address this issue for the 450 line.

9

If you’re only using 450m and 450i then you shouldn’t have anything to worry about. They don’t have an internal GPS receiver… only the original 450’s did.

10

Hmm, I may need to double check our infrastructure again, but for the 2019/2020 leap year incident our 450i’s took a GPS hit and dropped out due to loss of sync

11

If the sync source that feeds the 450m/450i has issues with leap year, then yeah, you’d see problems. If you’ve updated the firmware on those sync-injection devices, then you should be good.

12

Eric, was there a fix for packetflux rackinjectors and base II devices? I didn’t see anything specifically stated in the release notes.

13

I was referring to the CMM5 and cnPulse… I don’t believe we had any issues with PacketFlux, but I have a ticket open with them to check.

1 Like
14

This excellent forum contributor may have mentioned it. :slight_smile:

1 Like
15

LOLZ… you caught my past self red handed! I guess we did have issues… for some reason I seem to remember there was a firmware update issued from packetflux to resolve this issue. Anyway, I’ll let you folks know as soon as I hear back regarding my ticket.

16

Some of our CMM5 stuff had issues, and I don’t see an update to go past 5.1.1, so we’re preparing for that. Piecing together details on it because I wasn’t at this job during that time. We had issues on our ePMP gear during that, and I remember working through that fix

17

Here’s the response from Packetflux:

The underlying problem is that the GLONASS (Russian) GNSS system rolls over every 4 years, at midnight Russia time which is a few hours before midnight on new years eve if you’re talking about North America. Last time this occurred, the firmware in the multi-GNSS modules didn’t handle it well.

The short version of what is affected:

The currently shipping syncboxes should NOT have this problem. Our GNSS vendor claims the firmware in the new modules fixes this issue. These are ones which shipped early in 2020 on. (Of course, we haven’t had a rollover so no one knows 100% for sure).

Older syncboxes/syncpipes (the GPS-only ones) don’t have this problem since they don’t have GLONASS. These would be mostly prior to 2018.

So the ones everyone needs to look at from us were the ones which shipped in 2018 and 2019, and maybe a bit into 2020.

At this point, everything gets more complex and convoluted, as depending on your situation the problem may or may not even happen. For instance, any unit connected to a rackinjector running relatively recent firmware shouldn’t have a problem unless the customer has turned GLONASS back on in the settings.

1 Like
18

Is there a firmware I’m missing for the uGPS/cnPulse to fix this or is it literally we need to onsite and unplug all the uGPS/cnPulse to reboot them, as we run AutoSync+FreeRun, so we should be ok at the new year, but will slowly go out of sync I imagine on freerun from what I’m reading.

We’re running from a variety of CMM4/CMM5 and cnMatrix devices, so I don’t think theirs any way to reboot the cnPulse/uGPS units except to unplug them unless we reboot the CMM/cnMatrix, but that would cause an outage for customer traffic I’d rather avoid.

19

I seem to remember from 4 years ago we had to turn AUX port power off and back on to some APs with cnPuls timing. Seems like there was some talk of a firmware update that would be forthcoming for the cnPuls but I don’t even know how to do a firmware update on them. Probably not something you can do unless you have it on the bench, not in service up on a tower.

20

Ya ours are powered not from the AP but from the CMM so can’t really cycle the aux :S