Odd Issue after CPE lightning struck

Robert_Haas · March 2, 2020, 8:36pm

Odd issue - This same issue occurred before with the same customer once before.

They get hit by lightning and something happens to their radio which causes the AP's and CPE's on that segment to have packetloss - even on the data VLAN - which each AP has its own Data VLAN but all AP and CPE share the same management VLAN.

There are 8 AP's on the management vlan, including ~150 CPE. When this occurs it affects all of the AP on that management VLAN.

Once I narrowed it down to their CPE and blocked it from AP the problem immediately clears. Looking at the log file for the AP they are connected to it is filled with:

--
Mar 2 11:30:55 10.100.3.70 kernel: 1 2020-03-02T12:30:55-05:00 MBT-SE kernel - - [meta sequenceId="18694"] [14903.540000] Adding WDS entry for 00:04:56:f9:59:85, through ni=00:04:56:60:91:97

Mar 2 11:30:56 10.100.3.70 kernel: 1 2020-03-02T12:30:56-05:00 MBT-SE kernel - - [meta sequenceId="18697"] [14904.800000] Adding WDS entry for 00:04:56:e0:c1:7d, through ni=00:04:56:60:91:97

Mar 2 11:30:57 10.100.3.70 kernel: 1 2020-03-02T12:30:57-05:00 MBT-SE kernel - - [meta sequenceId="18698"] [14905.770000] Adding WDS entry for 00:04:56:d2:1c:86, through ni=00:04:56:60:91:97

Mar 2 11:30:57 10.100.3.70 kernel: 1 2020-03-02T12:30:57-05:00 MBT-SE kernel - - [meta sequenceId="18699"] [14905.780000] Adding WDS entry for 00:04:56:ec:13:0f, through ni=00:04:56:60:91:97
--
Where 00:04:56:60:91:97 is the customers CPE and the other MAC addresses are CPE connected to other AP's.
I have SM isolation enabled on all the AP's (which obviously they have layer 2 connectivity across the management vlan to other AP's and CPE).. This is the second time this has occurred with the same customer, we replaced the CPE the previous time as well.

It's almost as if it is causing a loop on the management network.

Cheree_King · March 2, 2020, 9:02pm

Was the SM in Bridge mode? I have had a similar issue happen a couple times when the SM was in Bridge mode, but it only affected the other SMs on the one AP the SM hit by lightning was connected to. It created a loop on only the one AP the bad SM was connected to, even though I had others on the same management and data VLAN. I have not seen this on an SM that was in NAT mode. And thinking back, I think they were both the old F110s. Once I disabled the ethernet port, the loop stopped.

Robert_Haas · March 2, 2020, 10:11pm

Yes, the majority of our SM are in bridge mode. Just a few are in NAT mode.
I think it’s odd it’s the same customer twice now.

Anthony_SBB · March 2, 2020, 11:48pm

If it seems to be ONLY that customer when it happens... Try changing their connection to a NAT'd network (if you can) and see if that solves it. Seems it's spewing some unwanted traffic back into your network when it happens, and if that IS the case... Moving them to a NAT'd connection should solve that!

Douglas_Generous · March 5, 2020, 7:43pm

I had an OOW Force190 do something similar to this but we couldnt say it was lightning. Then the radio died before we could figure it out so we never did figure it out.

Do you have lightning protection in place? is the radio protected from direct/indirect strikes by having something metal at least 3ft above the radio? Is there an LPU on the cable? As much as I dislike Ubiquity, their Eth-SP is cheap enough to almost handout like candy and is fully compatible with the epmp equipment and has similar ratings to the Cambium LPU. An LPU also protects the radio from surges coming from inside the premesis and a bad ethernet transformer would do wierd things to the ethernet side of the connection which would be seen as many connections being added to the bridge table.

I agree with Anthony_SBB, change the network mode to see if it can be stopped or simply try disabling the ethernet port. The radio will still recieve power, just not process packets. If this stops the problem then try enabling again and see if that radio's logs tell you anything regarding speed/link flapping. What happens when a known good device is plugged directly? Has the power supply been changed yet? Gigabit inline power injection requires some fancy electronics, if they are damaged then the signal can be reflected back to the radio which in bridge mode would just pass what it gets to the AP. In short there is a lot that can be wrong seperately and enven more combinations of things that creat a mess to sort out.

Dmitry_Moiseev · March 6, 2020, 2:50pm

Hi,

In my experience I've seen devices that are mirroring data from the Ethernet port after lightning strike. Network acts like it has a network loop. Limiting broadcast domain by changing the mode as adviced should help.

Thanks,

Dmitry

Robert_Haas · March 9, 2020, 4:19pm

The SM does have a LPU, which both times tested good after the strike.

Both times, the radio was functioning on the RF side of the SM but the ethernet side was non-functioning. I can't remember the first incidence but I think the router behind the SM was dead. This time there was still a physical connection but no data passing on the ethernet side and the router was fine after testing.
I just find it odd that the SM has entered this failure mode. The first time this happened it took a few hours to figure out what exactly was happening, at least this time I had a inkling on where to start.

Douglas_Generous · March 9, 2020, 10:09pm

I have had lpu's test good and still be an issue. Tvs diodes at dc vs at 33.3Mhz are two different animals, it never hurts to swap in a known good one.

Link but no data is usually a blown coupling capacitor on the chip side of the isolating transformer. I have fixed many of canopy fsk units that took a graze and this has been fairly accurate.

Have you determined if its the radio being struck or if its coming in on the power? If you think the radio is being struck, get a galvanized rod and bolt it to the radio mount then run a 10awg or larger grounding lead to the house ground, no water or gas pipes, if you cant find the ground the service meter is very well grounded. Clamp to the box.

If you think its coming in from the power side, a good ups will usually provide enough isolation. Or you can use a bathroom razor isolation transformer in a non metal box.

Robert_Haas · March 11, 2020, 7:56pm

The path - whether on the power or through the radio I'm not sure about. I've gone ahead an nat'd that customers radio since we are entering storm season just to see if this happens again.

Skullzaflare · March 11, 2020, 8:06pm

Make sure in the AP you have "SM isolation" enabled, it doesnt fully prevent the "loop" issue, but it significantly reduces it to barely noticeable levels