Dropping to 16 QAM and staying there until rebooted

Could you collect diagnostics files and email them in to Customer Support please?

Will do.  Is there a case number I should reference?

Hi,

You can call the support center with the details on the issue and they will open the case to assit you further.

Regards,

Sanjay Kumar

This issue is still not resolved, and I haven't gotten a response from support since November 23rd.

I've had to restart at least one of the impacted BH pairs every day for the last three, and we're losing customers because of the reduced capacity.

I've worked in call center support, so I try to be understanding about the limitations of support staff, but it's becoming increasingly difficult when we're bleeding here.

Hi Ted,

Sorry that we've not reached a resolution sooner. I have a copy of field diags that you provided last July. We don't have a complete explanation for the odd behaviour that you're experiencing. However we can make some suggestions that might help to unlock the puzzle. I'll contact you directly to avoid publishing details of your network.

Mark

1 Like

I wanted to post a follow up in case anybody else comes across this problem.

The issue is being triggered by weather radar interference every five minutes.  Because the 650s cover a wider band, they don't have noise filters to catch that like the 600s and previous models did.  Every now and again, the BHs fail to go back to a higher modulation once the interference stops.

Mark has been able to replicate the issue, and I've been told they're hoping to have it fixed in the 01-50 release.

Hi

we are experiencing a very similar situation, with the only difference that sometimes rebooting doesn't solve the issue.

I've the suspect that a strong signal, even in adjacent band, may overload radio or let the master behave badly.

have you got any news for the software release?

thanks

Ted,

I cannot think of any legitimate reason that would explain this behavior on your two links. 

To help us investigate, it would be great if you were able to get the Field Diags from all 4 units to us. 

Thank you!

Antony, you should talk to Mark Thomas about it.  As I said above, he's replicated the issue, and it's on his todo list as it's a firmware issue.  The problem is that photo radar noise is driving the modulation down every five minutes, and for whatever reason the firmware is sometimes deciding to stay at the modulation rather than going back up to 64 or 256 QAM in the five minutes when the noise isn't present.

BlackTC, if you're seeing the same radar interference, you'll see a lot of variance in your vector error min/max, and you'll see regular dips in vector error in the diagnostic plotter.  Weather radar in my area is on a five minute cycle, so I see sharp dips every five minutes.

If that's the case, the only thing you can do until the new firmware is ready is to put on external antennae and put in a filter for the frequency that photo radar runs on in your area.  Because the PTP650 works across such a wide spectrum, it doesn't have the same filters the previous PTP products had, so you have to add them yourself.

I haven't communicated with Mark Thomas in a while, but he's the guy who would know how the firmware is coming along.

Ted,

My mistake, apologises for the confusion!

Mark has indeed confirmed we have all the diags.  We will have somebody looking into this right away.

Hi,

The characteristic of what I'm inclined to call the "Ted Stewart Problem" is the interference is intermittent and at a very high level, occurring every five or ten minutes. Following a reboot, the link reacts to the interference by stepping down modulation mode, and then increasing modulation mode once the interference has passed. All of this is as we expect. The odd behavior is that after a few days of this activity, the link drops to a low modulation mode and doesn't recover. A reboot puts everything back as it should be.

Note that the interference is not in the band that the link is operating in.

The reduction in modulation mode in the presence of interference is of course expected and useful behavior.

BlackTC: Do you see exactly the same behavior?

Mark

I seem to have an uncanny knack for finding and reporting issues nobody else notices, so dubbing it after me might lead to some confusion with the PMP devs.  :-p

Hi Mark

i really do not know, but it seems to me similar enough in the sensitivity to strong signal that are interfering even from different frequencies:

we had a very strong intermittent signal from 5603 to 5678 Mhz, slave side saw peak signal stronger than -40dBm with a minimum of -70dBm. we were transmitting and receiving at 5555 MHz, so quite far away from signal, and there was no increase in peak signal in the frequency we were using or in the nearby frequencies. master dropped modulation to 16QAM and even QPSK, and there is no way to have an usable modulation untill the strong signal stopped.

we tried even to change temporarily band, and moved to 5.8GHz, with a completely clear spectrum, but master was only transmitting at QPSK.

in both cases, 5.4Ghz and 5.8Ghz, the slave was transmitting at 256QAM.

in the same time, we had the same behaviour in another ptp650 link that saw the same strong signal, even here at more than -40dbM; here we were transmitting at  5713Mhz

we had also a ptp500 link and a ptp600 link in the same sites, they saw this same signal, same frequencies with the strong peak, but since they were using different frequencies, as is expected, just goes on with the same modulation ever. ptp600 were using 5590Mhz, so really near, but manage to have very minimal impact ( just a little drop in average transport )

Hope it helps in finding and resolving the problem.

Mark

I had the same issue on a PTP 650 link  (release 50650-01-42) running in the 5.1- 5.2 freq about 6 or so different times  in a period of 3 weeks (March - April 2017) and a reboot kicked the QAM  back up. I do not believe it is interference causing the modulatipon  to drop and not recover. 

i just have to watch to see if it pops up again as I think it is a software flaw. I believe you have access to my email and can email me if you like

Thanks

Robert

Hi Robert,

Thanks for the information. We're actively investigating this issue at the moment. If you notice your link is stuck at a low modulation mode in the absence of interference, please collect the field diagnostics before rebooting it. This will help us a lot.

Mark

An update:

Adaptive modulation responds to interference by reducing the modulation mode. If the interference is intermittent, we expect the modulation mode to recover when the interference is removed. The problem that Ted Stewart identified is sustained high levels of intermittent interference can cause the link to become "stuck" at 16QAM. You can "unstick" the link by rebooting an ODU, or just by dropping and re-establishing the link.

We are now working with Mr. Stewart to evaluate some firmware changes that address the "stuck at 16QAM" problem. Initial results look promising. We're very grateful to him for helping us with this.

We'll provide updates in a forthcoming PTP 650 release to address this problem.

Hi Thomas,

have you got any news for our problem? I mean a strong signal in a different channel far enough from channel in use makes the modulation drops to minimum.

thanks

Could you collect field diagnostics from the link that is operating at low modulation mode please?

I don't see any mention of this issue in the 01-44 release notes, so I wanted to touch base and see how this issue is coming along.

Any progress, and is there anything I can do on my end to help?

Have a lovely day, all!

Hi Ted,

We should have been in touch sooner.

As you know, we eventually managed to reproduce the problem. The link responds to interference by dropping modulation mode as it should, but eventually sticks at 16QAM in the absence of interference. The problem is in the FEC decoder stage of the receiver. The link can be reset by re-establishing the link. We only seen the problem in your links that are exposed to high levels of interference from a radar station.

We provided some beta firmware that worked well in our lab testing and you kindly installed this in a standby link. From what we understand, this has improved the situation. However, the change is not 100% effective and we continue to work on this problem. We hoped to provide a fix in 650-01-44 but this fix missed the boat.

We'll be in touch with some more beta firmware as soon as we have a better solution. In the mean time, some more field diagnostics  from links with the earlier beta firmware would be very useful.

Thanks, Mark

1 Like