All SMs dropped to MCS1 and would not maintain connection

Had a bad day yesterday and I'm not sure the root cause. Maybe someone has seen this and has advice.

I climbed my tower to replace the shielded Ethernet cable to the AP. We co-locate with FM and I was getting issues with the AP's Ethernet connection re-negotiating between 100BT and 1000BT as well as errors on the switch for that AP.  I cut and crimped a brand new cable and put 6 100Mhz ferrite's on each end to help combat the FM noise.  The AP was down for about 30 seconds as I replaced the cable.  

When it came back up, all 10 SMs that were assigned to it had their uplink MCS drop to 1 and all were having connection problems. They are a mix of Force 110s and 180s on a 20 Mhz channel that was and still is clean.

I was unable to get at any of the SMs over the wireless link. I rebooted the AP twice and each time, had the same problem with upload MCS.

I then changed frequencies from 5.2 to 5.8 (not as clean) but that allowed me to get at the SMs and migrate them off to a hot spare AP we have.

When I used the Monitor -> Wireless in an SM, it would show me the troubled AP but I got a GFS error note and signal strength was in the dirt. 

I put the AP in spectrum analyzer mode to see if anything changed but I've never been successful getting that to work with Java on my Mac.  I do have a Mimosa B5 at this same tower and using its spectrum tool, the old 5.2 channel was clean.

I went back to it and started slowly migrating SMs back to it and now they are working just fine.  No reason to explain what happened but it was not working for about 3 hours.  It looked like interference at the AP but it started when I replaced the cable and never showed on a spectrum analyzer.

I started fearing the ferrite's had something to do with it but I had also installed them on another AP right next to this one at the same time and it had no issues at all.  The new cable and ferrite's did seem to solve the Ethernet error - which is a silver lining.

 Anyone seen this or have any thoughts?


@Au Wireless wrote:

Had a bad day yesterday and I'm not sure the root cause. Maybe someone has seen this and has advice.

I climbed my tower to replace the shielded Ethernet cable to the AP. We co-locate with FM and I was getting issues with the AP's Ethernet connection re-negotiating between 100BT and 1000BT as well as errors on the switch for that AP.  I cut and crimped a brand new cable and put 6 100Mhz ferrite's on each end to help combat the FM noise.  The AP was down for about 30 seconds as I replaced the cable.  

When it came back up, all 10 SMs that were assigned to it had their uplink MCS drop to 1 and all were having connection problems. They are a mix of Force 110s and 180s on a 20 Mhz channel that was and still is clean.

I was unable to get at any of the SMs over the wireless link. I rebooted the AP twice and each time, had the same problem with upload MCS.

I then changed frequencies from 5.2 to 5.8 (not as clean) but that allowed me to get at the SMs and migrate them off to a hot spare AP we have.

When I used the Monitor -> Wireless in an SM, it would show me the troubled AP but I got a GFS error note and signal strength was in the dirt. 

I put the AP in spectrum analyzer mode to see if anything changed but I've never been successful getting that to work with Java on my Mac.  I do have a Mimosa B5 at this same tower and using its spectrum tool, the old 5.2 channel was clean.

I went back to it and started slowly migrating SMs back to it and now they are working just fine.  No reason to explain what happened but it was not working for about 3 hours.  It looked like interference at the AP but it started when I replaced the cable and never showed on a spectrum analyzer.

I started fearing the ferrite's had something to do with it but I had also installed them on another AP right next to this one at the same time and it had no issues at all.  The new cable and ferrite's did seem to solve the Ethernet error - which is a silver lining.

 Anyone seen this or have any thoughts?


Chad,

This is a bizarre one. I have not heard of anything similar in the last 24 months from the world over. I can't pinpoint anything other than that you were seeing GPF errors which has to do with the SMs not getting responses back from the AP. Typically happens if the UL interference becomes really high and AP never responds to the SM. I can't help but also tie it back to the FM tower and the timing of your change. Maybe other folks who have deployed near FM towers can provide a theory better than mine. 

Sakid

2 Likes