GPS Sync - I'm tired

I did a quick google search and found this:

https://www.sanav.com/sm-76g.html

I am not sure if the ePMP uses a powered gps antenna or not, so I am not saying for sure that this will work, but it is an option.

Cambium, these should be a FRU and we do have a need to replace them.

What packet flux product should i use for ePMP1000 and 2000 gps sync?  We are having trouble with sync disconnecting.

I can not tell you what to use as thats not correct for your network, but I can tell you what I am using and what settings I am using and you can make a decision based on the facts and optinions available.

We are using the SyncBox Jr. Basic with the PowerInjector Plus Sync - Gigabit version (the 10/100 version works but for the same prie might as well get gigabit, you can change the power pinouts inside the box to power other devices too. Just make sure you have it setup to power cambium before you power any cambium radio). We power with a 5A 30vdc PSU on a live UPS ( We custom build our UPS systems as we can then add capacity where needed. A GatorPower supply with a pure sinewave inverter accross a couple big batteries (we use locomotive starting batteries) keeps the system stable) you could forgo the inverter if everything in your tower is 12vdc but you would need to have individual regulators for each device added.

Settings in the radios is as follows:

co-location mode enabled

sync source cmm3

sync source co-located equipment cmm3 (set this even if you do not have another system on the tower. it forces the radio to properly set sync.)

sync hold-off time 1200 (this is the number of seconds to wait before sync loss causes radio stop, 1200 sec is a long time but we found the recommended 200sec to be too short on severe weather days)

do not use cmm4 unless you have cmm4's within radio range of each other.

Placement of the syncbox jr is an art. We have had great success placing them in the cone of silence below sectors and omni antennas, but each antenna seems to have different placement points even from the same manufacture. Cable length is not a big issue as it is a serial link to the syncbox, but we keep it as short as possible. Cat5e STP is recommended but we have one that is just on outdoor Cat5e and shows no issues. Cat6 works but the cost/benefit is not there. Just make sure you follow the instructions and they work great. For towers with a lot of radios, the rack solution may be more practical but you can dasy-chain many powerinjectors together and they just work.

sorry for the word bloat and hope this helps.

1 Like

Have you tried removing the external GPS antennas and just letting it use the internals? That helped our side.

We did on a test AP but the antennas we are using place the radio inside a metal box for both additional weather protection (which is needed here) and better RF immunity (an added bonus that allows us to place APs closer together. We found that we needed to have >2m of back to back space before the metal box due to RF injection into the radio. With the metal box this was reduced to just over 1m. Which reduces wind loading torque on the tower.) This left us having to use the external GPS antenna or provide sync through the cable. We chose to use the cable method as it allowed us to synced to a single source and provide power from our DC power bank directly. We do have the GPS antennas installed just in case we have a sync module failure, but we haven’t had one actually fail yet. I still have a couple of older sync pipes in the system and working perfectly. On the other hand I have lost CNN micros(whole unit), the CNN micro GPS antennas, CNN micro GPS board.


@Douglas Generous wrote:
We chose to use the cable method as it allowed us to synced to a single source and provide power from our DC power bank directly. We do have the GPS antennas installed just in case we have a sync module failure, but we haven't had one actually fail yet. 



Two days ago one of our AP's lost GPS sync. It was a tower being synced over power via Packetflux so my first thought was that the AP's sync setting must have been left on GPS and I would just need to log in an change it over to CMM.  I was pretty surprised when I found it was already on CMM.  The AP was only losing GPS sync for a minute or two so I thought I would just up the Sync Holdoff time to 1600 like we had originally been doing before moving everything to Packetflux.  The AP responded by increasing the time the GPS was down...  So I swapped it over to the GPS setting hoping that would keep it going until I could drive out to the tower.

So at this point I'm thinking I have a bad port on the packetflux injector since there was 3 other devices on that injector that were working fine.  I arrive at the tower and swap the offending AP with another AP on the injector.  I wait a while for the other AP to fail but it dosent. I head back to the office thinking that if it fails I'll just change it over to it's GPS puc and I'll know I need to go out and replace the injector.  It doesn't fail though, all the rest of the day it just chugs right along.  So in the wee hours of the morning I log in and change the original AP back over to CMM (it was still in a new port on the injector) and within about 20 minutes it starts losing GPS again....  

Just today, had another AP on another tower getting its sync via power from a packetflux injector suddenly lose gps sync. So for right now I have changed it back to using it's puc and set the hold off to 1600 seconds... so far it's working ok.

So yeah, for sure keep those pucs connected. I'm not sure what we are going to do if it turns out the ePMPs are going to continue to fail no matter how you try to sync them.

We used Canopy gear with packetflux for years and years before ePMP came along and I don't remember ever having a problem with Sync. It wasn't something we even bothered to monitor because it never failed.  Now with the ePMP we are having to put a lot of time and effort into not only keeping GPS sync working but monitoring GPS sync and sending alerts when it malfunctions.

1 Like

First, sorry for all the spelling errors of the last post. New phone and it is still trying to change my spelling to “correct” spellings an abbreviations.

You say on cmm. But there are two cmm settings. This must be the same on all APs on a tower and any AP facing that tower. Ensure both sync settings are set. We set both to cmm3 even if there is no colocated equipment on the tower. We do still have some canopy gear on a couple of our towers as it still works great and the endpoints it services does not require the bandwidth of the epmp system. So we chose to keep the settings regardless to ensure system stability.

The old canopy gear did not need the timming pulse as often as the epmp seems to. So you might have an issue that may require a climb to solve. A thing to try is make sure your offending AP is on the 3.5 firmware and if you are going to use the gps puck make sure you upgrade the gps firmware from the 3.5.1 firmware package as the older gps firmware has a bug that causes the serial link from the gps card to the radio to freeze up and only a full power cycle will fix it.
Another thing to look at is which LPUs are you using? Some LPUs will cause issues with the sync pulse. Some dont until they start having problems and then do. I have lost several LPUs without any lightning involved and the sync lost issue usually is the first indicator. I found the cambium ss600 series LPUs to not be friendly to the sync pulse and the transector replacements are not any better. The Ubiquity eth-sp does play nice and is good for gigabit, though it does shorten your cable limits. We found that out the hard way when a cable run of 328ft with the eth-sp at tower bottom would not work. Shortened to 300 ft and it worked flawlessly.

Another group of thoughts: you only have one AP doing this? Is the AP protected in a box or is it exposed like on the cambium antenna?
What else is on that tower? Especially the leg that the cables come up on and cross bar they come over to the radio on.
If all other radios sync fine and swapping ports does not change anything then it will be either cable (physical or induced into) or the radio (again physical or induced into). The radio looks for a pulse that rises before the end of one second and falls at the beginning of the one second. I cannot remember the duration but its short. Very short. Short enough that too much capacitance in a cable could make it not work.
Check your cable grounding. It must be grounded only on the bottom and ensure that the radio case bond is to the tower. The only LPU should be at the bottom of the tower, just before entering the shelter. Ignore this if local/federal electrical codes require different bonding to be followed. Always follow the regulations required. Maybe try a test without the LPU to eliminate it as the offender.
Also if you are using the newer gigabit injector, make sure that the power pins inside are set correctly and that your power supply is stable and not fluctuating. I use a 30vdc psu from an Xplornet satellite modem for my supplies that are not on a 24v battery bank. Since it is a switching power supply, it can cause fluctuations on the 30vdc output and this caused some sync issues until I place a filter and a zener clamping diode on it. For battery systems I use a linear regulator to ensure the charging voltage does not effect the radios.

Cable impedance does matter. Since impedance is hard to measure without the correct meter you can do other tests that will give you an idea about the cable. Make sure your cable has the correct resistance accross each pair and that the capacitance on each pair is the same. A simple cable tester is not sufficient. You need a cable analyzer to correctly test your cables. If the pulse is attenuated too much then it cannot get through.

Most of the above I expect you to have tried. But any idea is better than no ideas. . . .
Speaking of an idea thats not much better: set your offending AP to cmm sync and once it drops sync power cycle it. Not a software reboot but a real power cycle. We use to see canopy gear loose power sync and it was caused by the sync circuitry locking up. A power cycle later and it was working again.

The ePMP radios dont just loose sync on the cable. Unless your tower was struck by lightning or had a very large power surge, the radios are fairly reliable. Both lightning or a surge will cause more than one radio to be having issues so we can rule that out.

"You say on cmm. But there are two cmm settings. This must be the same on all APs "

While this isn't relevant to the problem being discussed this is interesting if true. I know that sometimes a firmware update can change the sync in a way that all radios have to be running version X or later but I had never heard that sync sources have to be the same. Can anyone in the know commen on this ?

Totally off topic but still pertinent:
From my experience, the sync sources do matter. I have a tower with 3 APs and two force 110s all synced and running a mix of 3.4, 3.5 and 3.5.1
Due to some business client requirements, upgrading firmware to all match has not been able to be done yet (scheduled maintenance window is forthcoming but must be coordinated with our client so it wont cause contractual issues). We were seeing some interference on this tower and could not see any sources not on the tower. We took a spectrum analyzer and found that one AP was not in sync with the others. We found that the sync settings on this AP were not 100% the same. The co-lo sync was not the same so we changed it to match and the tower hasnt had an issue since.

Side note: we tested the ePMP platform on a tower with canopy equipment and found that the cmm3 settings to be most compatible. Even in different RF bands the RF injection on the receivers was too much if either system tried to transmit while the other was receiving. I also qualify this as we could not provide verticle separation on this tower so we had to ensure the timming was correct. This included matching the RF frame size (2.5 vs 5ms) with the existing system.
The ePMP User Guide also has some important information regarding CMM4s mixed with CMM3 based timming (the packetflux system is CMM3 compatible timming).

Back towards on topic: Do not get me wrong, I like the onboard GPS function. This has saved us from delaying a new site install while waiting for packetflux gear to arrive. But we found that we can not trust it 100% either. The APs and backhauls we have on 3.5.1 that we have tested using the on board GPS have been fairly stable with minor issues regarding not enough visible satellites that we traced to local wimax equipment on close frequencies causing desensitization. The packetflux unit does not have this issue, mostly from installed location. We install the packetflux GPS receiver only high enough to ensure that we have a good sight to the south and as much east/west horizon as we can without placing it so high that it can pickup interference from other systems. Most of the time we only need to install at roof height on the tower shelter.

Brubble,

I was rereading the thread and I never asked which sync injector are you using? I had an older sync injector that was 10/100 only and had some sync issues with it. Switched to the sync injector plus (gigabit version), set the jumpers correctly and its been perfect ever since.

1 Like

@Douglas Generous wrote:
Brubble,

I was rereading the thread and I never asked which sync injector are you using? I had an older sync injector that was 10/100 only and had some sync issues with it. Switched to the sync injector plus (gigabit version), set the jumpers correctly and its been perfect ever since.

I've had horrible problems getting our packet flux hardware to sync EPMP...  you ran into this as well with older models?  I'm wondering if that's my problem.   the sync injectors i have are at least 5 years old, the EPMP radios just will not read sync from them.  they work fine on the pmp100 hardware we had on it.   if this was your experience i'll order 1 of the new units and try out.  We've been having problems with birds eating the sync cables on a site recently... and of course there is a huge colo-deck right over our gear, the extra gain from the external antenna is needed, the internal antenna can't get over 15db snr and only tracks 3 or 4 at a time from the path obstructions. 

Has anyone used a high gain GPS\GLONASS antenna and connected it to an epmp for a higher gain signal?  Would this work provided it uses the correct frequencies?  What about one high gain antenna with multiple (4-5) epmp's using that antenna for it's gps signal?  Any ideas?  I'm having a hard time getting some of my epmps to track more than 4-6 satellites even though SNR is 25-35.  Some satellites have SNR of 15-30 but they won't "track".  I am wondering if a higher gain antenna would help because it can see some satellites but with 0 SNR.

Matt,

Questions:
1) is this an epmp 1000 or 2000 system?
2) what firmware are you running?
3) are your gps pucks flat(magnet down, cable out the side) or are they virticle with the cable out the bottom?
There is a known gps loss issue that is resolved with a gps firmware update. Doesnt solve number of tracks though.

To answer your questions:
You have a high gain gps antenna, you are getting 15 or better snr on 15 or more satellites then its not an antenna issue unless the gps antenna is in line with your service antenna. Having satellites with 0snr is a good thing. It means you can see a large constellation group. The system only needs 2 satellites to provide timing and 3 for a 3d location fix. So having only 4 tracked is not a bad thing and infact can be a good thing as it takes cpu time to process those tracks which takes away from pushing packets. I have a lone ap using gps right now and it sees 28 satellites and is only tracking 5 on average. Still maintains sync with the rest of the network.

Adding multiple radios to a single receiver antenna would reduce the snr by the factor of the number of receivers attached. It could be done but it does not work well.

A few different injectors and revisions.  The newest gigabit injectors only work with Rev H or later GPS sources and the older gigabit version (the middle one) only worked with C or later.  If you are like us and have a lot of the PVC pip (Sync Pipe) gps sources from the Motorola Canopy days then the new gigabit injectors may not work with some of the old ones.  Some will though, the sync pipe in the picture is Rev H while the new style in the gray box is a Rev I .  

We replaced a ePMP 1000 a few weeks ago that kept losing sync. So far the new radio is doing just fine.

Just out of curiosity we set up the bad radio (we will call it APe35G-test) and a brand new radio (we will call it new-test-gps)  right next to each other with their pucks laying side by side (in the lab, not on a tower).  We did some testing. 

Below is the old bad AP ------- new good AP.

In the first line the numbers  2  4  11   mean  2 (Have gps sync)  4 (tracking 4 sats)  11 (can see 11)   while the new radio  has sync  and is tracking 18 and see 22.     On down you see a 1 instead of a 2, that means it does not have sync.  We found the radios require 4 tracked sats to have sync, if 3 or less are tracked it loses sync.  I believe the "3" in the status position means the radio is trying to aquire sync.

21:47:44 APe35G-test 2 4 11 -------- new-test-gps 2 18 22
21:48:14 APe35G-test 2 4 11 -------- new-test-gps 2 18 22
21:48:44 APe35G-test 2 4 11 -------- new-test-gps 2 17 22
21:49:14 APe35G-test 3 3 10 -------- new-test-gps 2 19 22
21:49:45 APe35G-test 1 3 10 -------- new-test-gps 2 18 22
21:50:15 APe35G-test 1 3 10 -------- new-test-gps 2 18 22
21:50:45 APe35G-test 1 0 10 -------- new-test-gps 2 16 22
21:51:15 APe35G-test 1 3 10 -------- new-test-gps 2 16 22
21:51:45 APe35G-test 1 0 1 -------- new-test-gps 2 18 22
21:52:15 APe35G-test 1 0 2 -------- new-test-gps 2 17 22
21:52:46 APe35G-test 1 0 2 -------- new-test-gps 2 18 22
21:53:16 APe35G-test 1 0 2 -------- new-test-gps 2 18 22
21:53:46 APe35G-test 1 0 2 -------- new-test-gps 2 18 22
21:54:16 APe35G-test 1 0 2 -------- new-test-gps 2 19 22
21:54:46 APe35G-test 1 0 2 -------- new-test-gps 2 18 22
21:55:17 APe35G-test 1 0 2 -------- new-test-gps 2 16 22
21:55:47 APe35G-test 1 0 1 -------- new-test-gps 2 17 22
21:56:17 APe35G-test 1 0 1 -------- new-test-gps 2 17 22

We set these two radios up and left them so we could figure out how the traps work hoping to get a heads up one when AP's were losing sats but it turns out the radios do not send a lost sync trap until the holdoff time expires. There are very few traps triggered relevant to the GPS status of the radios it turns out. It would be really nice if it triggered a trap when the radio actually loses tracking instead of it waiting unitl the holdoff time runs out.

Note that we put a new (square and round) GPS puck on the old radio and it made no difference at all. The radio will not see any sats at all if it doesn't have a puck attached.   

Also, the new radio has lost gps sync on several occasions (we have the default 30sec holdoff time set on it so we have a better chance of catching it when it loses sync).  And each time the bad radio who's puck is literally right next to it did not lose sync at the same time.  This tells us it is not an interference issue because the bad radio is 1/2 blind already, any interference that would knock out the good radio should kill the bad radio . 

This is almost certainly something goofy going on with the ePMP 1000's GPS sync (haven't tested a 2000 yet and the ones we have in the field are set to 1600 secs holdoff so there is now way to know if they are losing sats if they get them back before the 1600 secs runs out)

We can't run any firmware after 3.4.1 on any live AP's because of problems with all the 3.5 stuff.  However I'm going to put the newest 3.5.whatever  on these two test radios and see if it makes any difference.