We recently swapped out a UBNT Airmax 5GHz PtP link that had been operating fine for a couple years, with ePMP radios in place between two sites. 7 mile shot, using 2’ dishes. 20MHz channel width. 5.675GHz. Around -61dB on both sides. SnR of around 35dB on both sides. MCS 14/15 on both sides. Speed tests normally show around 70-80mbps DL/UL.
We’ve had two issues now. The first issue happened 2 hours after the link had been running flawlessly. We had about 30mbps of aggregate traffic flowing through, and pings were great, 20ms avg using 1450byte ICMP. All of a sudden pings dropped, and no traffic would pass.
We logged into both sides and there was no RF link. Unfortunately the tech didn’t grab the log files and just rebooted both sides. Everything came back normally after the reboot.
Today, after working fine for almost 14 hours, we started noticing sporadic packet loss via the RF link to the remote side. Not high ping times, just every 5-6th packet dropped. Very little traffic was making it through. This time we grabbed the log files off of both the AP and SM. We first tried rebooting the SM/far side and it was still having intermittent packet loss. We then rebooted the AP side and everything was fine after that.
This is a sample of the AP log that was repeated over and over:
Sep 24 17:28:14 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPS Sync Lost. (00:28:14:932555)
Sep 24 17:28:17 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: VAP device ath0 created
Sep 24 17:31:40 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPS Sync Lost. (00:31:40:272572)
Sep 24 17:31:42 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: VAP device ath0 created
Sep 25 09:51:18 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 25 09:53:04 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 25 11:10:01 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 snmpd: Watchdog:Abnormal SNMPd stop occured, restarting…
Sep 25 12:07:58 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 25 13:08:40 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 25 13:09:53 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 25 13:15:31 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 25 13:38:31 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 25 13:39:19 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 25 13:41:13 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 25 16:11:16 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 25 17:21:44 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN
This is a sample of the SM log, that was very quiet:
Dec 31 17:00:24 GRIZZLY-(EPMP-5GHZ-GREY-BH)-CN2944 kernel: VAP device ath0 created
Sep 24 17:22:42 GRIZZLY-(EPMP-5GHZ-GREY-BH)-CN2944 crond[1534]: time disparity of 23526742 minutes detected
We believe this may be a problem with the AP. Next time we’re going to just reboot the AP and see if it fixes the issue.
It happened again today. This time instead of rebooting, we just changed frequencies to the alternate frequency of 5810. Linked right up and started passing traffic.
Checked logs on the AP and more of the same:
: tdd_gpf_send tdd_maptx_add_gpf error
Sep 27 00:00:07 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 27 00:03:17 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 27 00:26:48 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 27 01:02:19 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 27 01:03:24 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 27 01:12:10 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 27 01:41:19 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Sep 27 01:46:29 GREY-(EPMP-5GHZ-GRIZZLY-BH)-CN2942 kernel: GPF: tdd_gpf_send tdd_maptx_add_gpf error
Eric,
This is an issue we are currently investigating. Those error messages may be misleading as they don’t happen often enough (every 10-20 minutes or so) to cause traffic to stop. So this is something we have to dig into and figure out what’s causing your link to stop working. I will keep you posted.
Eric,
We need to get to the bottom of this. Please send me the radio configuration files (from Tools->Backup/Restore) from both the AP and the SM and send them to me @ sriram@cambiumnetworks.com.
I would like to continue troubleshooting this issue with you offline.
Thanks,
Sri
I sent you some details, thanks!
I am also having this problem on multiple AP's and SM's in PTMP mode. I have sent an email to cambium_sri asking for help.
Did you find a solution to your problem?
Thanks,
was this problem fixed?
Hi,
Yes. Please upgrade to the latest 2.3.4 release.
Thanks,
Sriram
Hi Sri.
We have a costumer here with the same problem. I try 2.4.3, 2.4.1 and 2.4.2 and the problem persists.
We can access the radio but the traffic from wireless interface "stop" and only returns after reboot.
@Zucchi wrote:
Hi Sri.
We have a costumer here with the same problem. I try 2.4.3, 2.4.1 and 2.4.2 and the problem persists.
We can access the radio but the traffic from wireless interface "stop" and only returns after reboot.
Hi Zucchi,
When you say traffic from wireless interface stops, are you talking about downlink or uplink traffic?
Thanks,
Sriram
Rebooting an old post but the same issue has cropped on one of my APs. I get the following in our SYSLOG:
GPF: tdd_gpf_send tdd_maptx_add_gpf error
Does not happen frequently but they come in bursts of a handful. This is on an ePMP 2000 AP running 3.1. It is running in flexible mode on a 20 Mhz channel width in PtMP mode.
Hi,
Have you seen this iisue on your link once or is it reproducible all the time?
Thank you.
Only a couple of times since the 3.1 upgrade. Never seen it before.
We have already faced with issue closely related to current one and fix has been already added to next release 3.2, which will be available soon.
Thank you.
Our old friend has returned. I have a new 1000 GPS AP running 3.2.2 and my log is filling up with this:
Feb 13 17:48:50 Belvedere AP - 40 MHz kernel: [791050.390000] GPF: tdd_gpf_send tdd_maptx_add_gpf error
It's been running for 2 weeks now. At first, it lost GPS sync (reported in another thread) and today, it is hitting my syslog with tons of these GPF errors. It is running in a non-DFS channel at 40 Mhz and in flexible mode due to the GPS failure.
After taking one of many 1000 GPS APs to firmware 3.5, I am seeing:
GPF: tdd_gpf_send tdd_maptx_add_gpf error
This AP does not have GPS turned on. About the same time frame as I see this error message, we loose wireless link to one of the 7 SMs on that AP - but just briefly. The AP reports the SM as disconnecting but it pops right back again a few seconds later.
What does this error message really mean?
I have a 1000 GPS AP running 3.5.2 and it is litterally barfing this into the syslog almost constantly:
GPF: tdd_gpf_send tdd_maptx_add_gpf error
I'm getting hundreds of these at a time and it will last for an hour or so - mostly in the eveing.
I'm also getting plenty of those for some time now on most of our sectors. The version is 4.3.2 everywhere.. Can anyone please explain what it means?
- Clipboard016.png (33 KB)
