I don't know if this is useful info or not but last night the AP in question appears to have rebooted it itself several times and then dropped all clients and stopped responding to HTTP, SSH, ICMP and SNMP. When I arrived at the site to power cycle the radio all the lights on it were on and ethernet was flashing furiously. Power Cycling the radio fixed it but according to the graphs the radio rebooted itself at least twice in the 10 minutes or so after I power cycled it (that was a bit over 12 hours ago and it has not rebooted since). So this morning I looked at the graphs just out of curiosity. I noticed that while we had not rebooted the radio the day the GPS started seeing Sats , it had rebooted itself several time. It also rebooted itself a few times just before it went dark last night.
Also note in the 2nd pic I attached. I zoomed in so the ethernet traffic is a 5 min average. I don't know if this is symptom of the reboots or a glitch in the graphing but I've noticed this in/out spike that coincides with the radios rebooting themselves on customer radios when we had to replace them because they kept randomly rebooting. I do not see these spikes when I reboot or power cycle the radio, only when the radios are randomly rebooting themselves. However there are also normally a huge number of ethernet port errors and there was none on this AP so maybe the spike is just a glitch in the graph caused by the reboot.
Edit: Note that I only have 4.4.3 in the primary bank and 4.4.1 is still in the secondary. So I am assuming the radio is getting completely booted each time before it reboots again else the firmware would have reverted to 4.4.1 ?
Also note the Force200 backhaul radio this AP is conected to has not rebooted during this time. This MicroPoP is a very simple setup : Force 200PTP --> epmp power supply <----> epmp power supply ---> ePMP 3000L ap.
Tonight this same AP dropped all 9 N clients and left the 1 AC (F300-16) client working. It remained this way for about an hour before I was made aware of the problem. I rebooted the AP and all the N clients (and the AC) came back and it appears to be working for everyone again.
After digging around the forums a bit I see that it has been recommend that both banks on the AP be upgraded to the same version of firmware or it can cause problems ? I'm going to upgrade the inactive bank later tonight but for future reference I would like to know if this is still a thing or if it has been fixed.
After digging around the forums a bit I see that it has been recommend that both banks on the AP be upgraded to the same version of firmware or it can cause problems ? I'm going to upgrade the inactive bank later tonight but for future reference I would like to know if this is still a thing or if it has been fixed.
I was under the impression that this was only an issue with F300-16 radios running older firmware from the factory, and that once it had been brought up to a newer version, like 4.4.2... that moving forward, flashing both banks was no longer necessary.
I was told by Cambium that it was (is?) all AC radios - both banks should always be populated with the same firmware.
...which kinda defeats the purpose of having a fall-back bank. BUT, if that is still a known issue, then we should test AC gear that way. For what it's worth - we are not seeing GPS's going offline, and I don't know why we are not affected like others are. However, I can say that we do always upgrade both banks to the same firmware.
I was told by Cambium that it was (is?) all AC radios - both banks should always be populated with the same firmware.
It is not exactly true. The only thing we do recommend is to update F300-16 twice when you go from the earlier firmware to 4.4.0 or later. And more to it do twice only in case you see any instabilities.
I would appreciate if you can keep us posted with the updates on this specific ePMP3000L.
Dmitry
Dmitry,
Even though since upgrading to 4.4.3 I'm plagued with random AP reboots and client radios being randomly dropped it has not stopped seeing/tracking Sats... HOWEVER, this morning I thought I would turn Sync back on. So I set Sync Source back to GPS and set the Holdoff to 1600. The graphing doesn't show that the radio ever loses sight of the sats , Monitor > GPS shows 18 to 21 Sats Visable/Tracked at all times BUT the little "GPS" symbol at the top right of the GUI randomly turns red ever few minutes and may stay red for a few seconds or a few minutes and the status page will say "Sync Down" at the same time. However the Monitor GPS page never shows it losing any tracked or visible Sats.. The randomly disconnected client radios aren't "all" being kicked like you would expect if the AP lost sync and the random client radios being dropped do not appear to be related to the Radio saying it is losing Sync.
I'm rolling back the 4.4.3 firmware update on the clients and AP tonight but I'll leave Sync on to see what it does.
Good tip -I would suggest setting hold off time 864000 on all sectors. That way you wont lose customers if sync is lost and GPS sync will still stay synced with other sectos close by. (Unless frequency is changed)
Good tip -I would suggest setting hold off time 864000 on all sectors. That way you wont lose customers if sync is lost and GPS sync will still stay synced with other sectos close by. (Unless frequency is changed)
While I didn't spend a huge amount of time on it when I tried to set up alerts to let me know when a sector had lost gps sync, I found that none of them would be triggered until the hold off time run out. That means the radio would lose GPS sync but no alarms would go off because running on holdoff is still having GPS sync as far as the radios are concerned. The only thing I could trigger off of was number of sats visible/tracked but then one bug was the radios losing GPS sync even though it could still see/track all the Sats so that was useless. Our solution was to change over to sync over power and that has solved 99.9% of our sync issues.
According to Cambium the timing starts to drift at/around 1600s so it would seem that after 1600s the sector is going to start interfering with neighbors. So if I set it to a huge number I won't know there is a problem until the customers on several AP's are having problems.. So for me the hold off time is really just useful for short/sporadic losses of GPS sync. We use sync over power on everything that matters , it's not a must have on any of the micropops yet.
The SA being enabled seems to be what caused all my issues. It looks like about 3 days before I upgraded the AP to 4.4.3 I had turned the SA on. Looking at the graphs when I turned the SA on the CPU usage jumped from 5% average to 20% average , the next day the two mystery reboots that caused the radio to start showing it was seeing/tracking Sats, then the day after that I upgraded to 4.4.3 (not noticing the increased CPU usage, the two reboots or the radio now seeing sats) so I assumed 4.4.3 was the cause of the constant client drops and AP rebooting.
I had already rolled the AP back to 4.4.1 and it didn't fix the problems but once I disabled the SA the CPU usage went back to 5%' , no more AP reboots and no more clients dropping/reconnecting for more than two days now.
The SA being enabled seems to be what caused all my issues.
Yet another good reason why a simple 802.11 site survey would also be good to have, in addition to the SA. Enabling the SA is a duck shoot in a way.
Also - since it is recommended to 'disable the SA when not in use', I think that enabling the SA should always have a timed option by default, meaning it should self-shut off by default, unless we specifically tell it to remain on forever.
In other words - if it's recommended to "disable when not in use", then 'On Forever' should NOT be the default setting on a SA.. So, I think when we go to enable the SA, the GUI should say something like "Enable Timed SA for 5m/10m/15m/30m/60m/Forever* (Warning: may cause pon"f'ar)"
At the very least... if the SA is 'on', there should be some indicator on the main status screen (like a SA logo up by the Green Globe and the Green GPS/Location icons... maybe a 'SA Running' icon up there?
Regarding the 3000 & 3000L GPS instability which some people are reporting... I just thought I'd post that our 3000 & 3000L have been very stable. I'm just about to upgrade a few AP's to the latest 4.5beta, and I thought I'd post these two images, showing GPS uptimes since our last upgrade/reboot.
39 Days since we rebooted this 3000 AP - with no GSP Issues in the last 39 days.
67 Days since we rebooted this 3000L AP - with no GSP Issues in the last 67 days.
These are both going to 4.5-RC54 today, so the uptime will get reset - but I thought I'd post that not everyone is having GPS weirdness, and that (at least for me) the GPS has been rock solid on our 3000 and 3000L APs.
Is that the standard GPS puck that ships with the ePMP3000?
It’s got a relatively short cable - can I pop an extender cable on it? Any cabling distance limit?
As I’d like to install the puck on the roof of the building the cnMatrix would be installed in. The actual ePMP radios are installed on the tower next to the building 25M up.
Yes it is. There were 3 generations of them. I’m talking about the last one.
Yes you can add an extender. But actual distance will depend on cable quality and connections losses. I suggest to experiment with available materials. 25 meters is significant distance.
We have a water tower where we’re colocated with 3 GSM operators. You can imagine what level LTE signals are there. Considering there’s lots of metal constructions, antennas human high, etc, the harmonics from LTE totally destroys GPS signal. It didn’t catch any satellite for even 1s there. We use 3rd gen GPS pucks and have placed them under the water tank some 5 meters below the roofing. We use 10 meter extenders with H155 coaxial cable, and so far this tower did’t have any GPS problems. Watch for very good characteristics of coax shield. Lower grade cables get easily penetrated.
