AP Disconnecting SM's

Products ePMP
1

Hello,

We have one ePMP 1000 Sync'd Connectorized 120 AP (that we know of), that is intermittently disconnecting all SM's. The period of time dropped has been getting worse. When I first noticed the drops, it seemed to be four or five minutes before the SM would reconnect to the AP. Now (a few weeks later), the downtime of the SM's seems to be around twentyfive to thirty minutes. 

I've noticed a couple wierd things in the log files that have me concerned, but today's log file has a little more immediate information, as I noticed it go down while I was on the internet at home, so I was able to snag a log file as soon as it came back up. 

Things that concern me, within the AP logfiles:

"Not received PONG for the las PING"

What is this referring to, and could it have any adverse affect on the stability of our SM'd connection to the AP? 

"GPS Sync lost."

"GPS Sync restored."

I'm sure there are any number of reasons that this could happen, but if someone could give me a good starting point for what to look into.... This is happening to all of our 6 sync'd ePMP AP's. Same question as above. 

Similar to the two above, but I am hoping that this last one gives someone an idea of what is going on, who has a solution. This is the portion of the log file, just before the SM reconnected to the AP. This outage lasted around 25 minutes. 

"Feb 21 10:30:07 MPHEPMPNE1 DEVICE-AGENT[1826]: Missed 3 consecutive PONGS, disconnecting with server ping_loss=3
Feb 21 10:30:07 MPHEPMPNE1 DEVICE-AGENT[1826]: callback_websocket: LWS_CALLBACK_CLOSED
Feb 21 10:30:07 MPHEPMPNE1 DEVICE-AGENT[1826]: DA_STATE_DISCONNECTED
Feb 21 10:30:21 MPHEPMPNE1 kernel: GPS Sync Restored. (17:30:21:963809)
Feb 21 10:30:23 MPHEPMPNE1 DEVICE-AGENT[1826]: EINPROGRESS in connect()
Feb 21 10:30:23 MPHEPMPNE1 DEVICE-AGENT[1826]: SSL_ERROR_WANT_READ try again
Feb 21 10:30:23 MPHEPMPNE1 DEVICE-AGENT[1826]: SSL_ERROR_WANT_READ try again
Feb 21 10:30:23 MPHEPMPNE1 DEVICE-AGENT[1826]: Server certificate is verified and it is valid
Feb 21 10:30:23 MPHEPMPNE1 DEVICE-AGENT[1826]: Received Headers : "HTTP/1.1 302 Found
Feb 21 10:30:24 MPHEPMPNE1 DEVICE-AGENT[1826]: callback_websocket: LWS_CALLBACK_CLIENT_ESTABLISHED
Feb 21 10:30:53 MPHEPMPNE1 DEVICE-AGENT[1826]: handle_cns_msg: MSG_REGISTER_SUCCESS received"

Since I am at home, and didn't have access to my network, I could not login to the AP, as soon as I noticed my connection drop, so I was unable to get log file information at the time of the drop. 

I can inlude any other information upon request. Any help solving this mystery is highly appreciated. 

-Shannon

2

Hi Shannon,

AP drops all registered on itself SMs because of GPS sync lost.

It is expected behavior in case when GPS Sync is enabled.

AP will not continue transmittion till GPS Sync is down.

Please check whether you have the latest GPS Firmware installed on AP Devices.

Do you use additional GPS antennas on APs?

Other log messages you have attached to post are messages from cnMaestro Device Agent.

cnMaestro DA is not able to establish connection to cnMaestro server when SM is disconnected from AP.

Thank you.

1 Like
3

Hi, 

LIke Fedor said, the primarily reason appears to be due to the AP losing GPS sync. Please check under Monitor->GPS to make sure the radio is tracking at least 4 satellites. If not, you may want to connect the external GPS puck antenna or even provide sync over power using a CMM or third party sync source similar to it. 

Thanks,

Sriram

1 Like
4

Fedor and Sriram,

Thank you both for your responses. It's nice to see that Cambium has such a great support platform through the forums. 

So, our GPS firmware is up to date, and the AP in question, as well as the adjacent AP, is tracking 7 satellites. I don't recall exactly, but I believe that we are using the external puck GPS antennas on all of our AP's. 

What else could cause GPS to lose sync?

By "losing sync," does it mean that it is losing sync with the other AP's, or with the satellites?

Attaching screenshots of our firmware version and satellites being tracked. 

-Shannon

5

Also, I noticed that the GPS Time is set to Greenwich Mean Time (GMT), which is different than our NTP Time Zone, set at MST (Mountain Standard Time). Would this have any affect on the GPS functionality?

-Shannon

1 Like
6

In this context 'losing sync' means that the synchronization pulse is lost.  In your case that pulse would be generated by the internal GPS, though it could come from a CMM, SyncPipe, etc.  Usually when you lose sync it's due to an insufficient number of satellites visible, which is usually a transient problem.

The overall constellation of GPS satellites are in constant motion on different paths, so from one minute to the next satellites may pass below the horizon or pop up above it, or behind a signal-blocking water tank, etc.  (that last only if you're mounted ON the tank, on the rail rather than the peak)  Seeing that you're tracking 7 satellites doesn't really help much, unless that's specifically during the time that it was dropping sessions and complaining about lost sync.

We've had cases in the past where we'd typically run 5-7 satellites tracked, but occasionally drop to 2-3 and lose sync for a few minutes.  Repositioning the GPS antenna usually resolved those.  (We also had a few instances where very large birds were perching on the gear and blocking the GPS signal)

Oh, and the GPS will always report GMT (or UTC) time, that's the only timezone that GPS satellites speak.

j

2 Likes
7
@shannon wrote:

Also, I noticed that the GPS Time is set to Greenwich Mean Time (GMT), which is different than our NTP Time Zone, set at MST (Mountain Standard Time). Would this have any affect on the GPS functionality?

-Shannon

Hi Shannon, 

The GPS (GMT) time is what the GPS system provides to us. So we're just displaying it as is without doing any conversion. Setting your NTP time to your time zone has no impact on GPS sync. 

Losing sync means the GPS receiver on the AP is no longer tracking the minimum required 4 satellites. It has nothing to do with another AP. There is no communication between APs with regards to sync and therein lies the beauty of GPS sync. Every AP receives timing via satellites independently and since they are receiving timing from the same source (GPS satellite system), they are synchronized in time. 

Now, regarding your question about why an AP would lose sync. There could be several reasons. One could be that there is no clear view of the sky for the GPS puck antenna (if installed) or the radio itself (if the puck antenna is not installed). Second could be that there is a strong interferer that is wiping out the GPS receiver that operates at 1.6 GHz. Something like a UMTS repeater or LTE system could cause this problem. Do you know if there are telco radio/antennas installed on the same tower or nearby towers? Better positioning of the GPS puck antennas may help. Alternately you could look into providing sync over power using a CMM4 or a third party sync source such as a PacketFlux SyncInjector. 

EDIT: Joel beat me to this response!

Thanks,

Sriram

1 Like
8

Joel and Sriram,

Thanks for your prompt responses. I really appreciate it. 

The site in  question is on a mountainside, with several other telecom companies operating on different towers in the vacinity. ATT being the most major. We have plenty of other RF AP's on this same tower (Conopy and Ubiquiti), but I'm not sure which might have GPS antennas...

As for the location of the puck antennas, and their view of the sky, all of our AP's puck antennas were set on top of the AP, with a clear view of most of the sky. However, there are areas of the sky that are not viewable, due to the high horizon line of the mountain the tower is on, and any obstructions caused by other equipment and the tower itself. I will have to see what we can do to possibly reposition the GPS pucks. 

As I understand it, the CMM3 (micro) is a network attached antenna(?). CMM4 is a rack mounted device (switched or unswitched), that controls the timing for all APs that are connected to it(?). Is the CMM4 only capable of controlling the timing for the devices at the tower site it is installed? I mean to say, it cannot control timing for remote AP's...

Thanks,

Shannon

9

CMMmicro is an 8-port remote managed Power and Sync injector for many Cambium APs (including ePMP), power injection is 30v passive with 10/100 ethernet.

CMM IV is also 8-port remote managed, but supports both 30v and 56v, (latter for PMP320, PMP450i with special cable/adapter) and may include a 14-port hardened switch - 8 ports for the APs, one for the CMM board itself, three 10/100 and two gigabit.  (AP ports are 10/100) It's available in rackmount form factor or built into a weathertight cabinet, with or without the hardened switch.

Both include onboard GPS, with external GPS antenna.

If you're going to have several ePMP APs at a single location, a CMM is a better choice than the midspan POE injectors that come with the radios - you get remote power-cycling when an AP locks up. 

One possible shortcoming with both is that they don't support gigabit for the APs. (to be fair, they've been established products for quite a few years now) For our use, we're sticking to 20MHz channel widths for RF flexibility, (plus I'd rather add another 20MHz AP than run one at 40MHz) which is under 100Mbps anyway, so no panic.  If you run 40MHz wide channels you could exceed that 100Mbps easily.  Overall throughput could be a concern especially on the CMMmicro, where the entire AP cluster it controls is limited to 100Mbps aggregate - CMM4 you can feed your BH into a gigabit port, so each of the 8 APs could pull 100Mbps simultaneously.

j

3 Likes
10

Thanks again to everyone who has contributed.

We are looking into possible GPS solutions, but I have a couple other questions in the interim. 

How long does the GPS Sync have to be down, before the SM's are disconnected from the AP? 

I notice that there are many logged entries of the GPS sync going down for about a minute, and then being restored, but the SM's don't drop during this time frame. For example:

Feb 22 12:46:05 MPHEPMPNE1 kernel: GPS Sync Lost. (19:46:05:529161)
Feb 22 12:46:32 MPHEPMPNE1 kernel: GPS Sync Restored. (19:46:32:32752)
Feb 22 12:48:18 MPHEPMPNE1 kernel: GPS Sync Lost. (19:48:18:529254)
Feb 22 13:01:44 MPHEPMPNE1 kernel: GPS Sync Restored. (20:01:44:7306)
Feb 22 13:19:03 MPHEPMPNE1 kernel: GPS Sync Lost. (20:19:03:509007)
Feb 22 13:19:09 MPHEPMPNE1 kernel: GPS Sync Restored. (20:19:09:13558)
Feb 22 13:44:15 MPHEPMPNE1 kernel: GPS Sync Lost. (20:44:15:529028)
Feb 22 13:45:23 MPHEPMPNE1 kernel: GPS Sync Restored. (20:45:23:30433)
Feb 22 13:57:38 MPHEPMPNE1 kernel: GPS Sync Lost. (20:57:38:529010)
Feb 22 14:02:34 MPHEPMPNE1 kernel: GPS Sync Restored. (21:02:34:28895)
Feb 22 14:05:42 MPHEPMPNE1 kernel: GPS Sync Lost. (21:05:42:529009)
Feb 22 14:06:08 MPHEPMPNE1 kernel: GPS Sync Restored. (21:06:08:29105)
Feb 22 14:07:32 MPHEPMPNE1 kernel: GPS Sync Lost. (21:07:32:519005)
Feb 22 14:08:08 MPHEPMPNE1 kernel: GPS Sync Restored. (21:08:08:27157)
Feb 22 16:35:36 MPHEPMPNE1 kernel: GPS Sync Lost. (23:35:36:529012)
Feb 22 16:36:04 MPHEPMPNE1 kernel: GPS Sync Restored. (23:36:04:29039)

I'm just trying to narrow down the problem, whether that is poor LOS to satellites, interference, or the absolute need of CMM or third party solutions.

That being said, if there is interference from another tower, or our own equipment, how will that affect a CMM or third party system? 

Is there a way to tell whether or not this is an interference issue?

Thanks again,

-Shannon

11

If the SMs don't drop during the time period in which GPS sync is lost, then there can be two reasons for it:

  1. The parameter, Sync Holdoff Timer under Configuration->Radio is set to greater than 1 minute or however long GPS sync is down. The Sync Holdoff timer allows the AP to keep transmitting (and therefore not drop the SMs) for that time period. If sync is restored in the time period, the AP continues transmitting. The default is 30 seconds so unless you've changed it, this is unlikely the reason.
  2. You have set the AP to Flexible mode. GPS sync is irrelavant in this mode.

One way to know interference is causing GPS sync loss is to check the SNR of the visible and tracked satellites under Monitor->GPS. If the SNR dips low (<20), then you know there is interference. However, this can also mean the LOS to sky is poor and you are losing satellites as the constelation moves. But interference is likely the cause. Using a CMM or third party sync injector may help because you can position those boxes anywhere you want (All you need is a cable run from the box to the AP) so it gives you greater flexibility. But first, I would try repositioning the GPS puck antenna. 

Good Luck!

Thanks,

Sriram

1 Like
12

Hello All,

After we finally got a window in the weather to climb up to our tower site and assess the situation with the GPS puck antennas, I was able to determine that another AP was likely blocking the view to the sky for a long enough period to drop our SM's. I repositione d the pucks by only a few inches, and our clients (including my own radio at home) have not seen a disruption in service in over two weeks! 

Big shout out to Sri, Fedor, and J, for all your help and prompt responses.

-Shannon

3 Likes