Network flooded with SYN_SENT traffic this morning origination from cnPilot r200 and 201s

Products Residential Wi-Fi
1

We woke up this morning to “Packet Loss” alarms from our Grafana instance. Internal routers were being overrun with traffic like this:
tcp 6 77 SYN_SENT src=162.218.199.62 dst=116.0.0.0 sport=32133 dport=28699 packets=1 bytes=60 [UNREPLIED] src=116.0.0.0 dst=162.218.199.62 sport=28699 dport=32133 packets=0 bytes=0 mark=0 use=1
tcp 6 76 SYN_SENT src=162.218.199.177 dst=73.0.0.0 sport=5233 dport=60120 packets=1 bytes=52 [UNREPLIED] src=73.0.0.0 dst=162.218.199.177 sport=60120 dport=5233 packets=0 bytes=0 mark=0 use=1
tcp 6 67 SYN_SENT src=162.218.194.200 dst=116.0.0.0 sport=54620 dport=21803 packets=1 bytes=60 [UNREPLIED] src=116.0.0.0 dst=162.218.194.200 sport=21803 dport=54620 packets=0 bytes=0 mark=0 use=1
tcp 6 55 SYN_SENT src=162.218.196.189 dst=68.0.0.0 sport=63160 dport=35927 packets=1 bytes=60 [UNREPLIED] src=68.0.0.0 dst=162.218.196.189 sport=35927 dport=63160 packets=0 bytes=0 mark=0 use=1
tcp 6 43 SYN_SENT src=162.218.194.200 dst=99.0.0.0 sport=30246 dport=65173 packets=2 bytes=120 [UNREPLIED] src=99.0.0.0 dst=162.218.194.200 sport=65173 dport=30246 packets=0 bytes=0 mark=0 use=1

Note the odd destination IP addresses. Routers are all running 4.7.1-B3. A reboot of the router stops the broadcast. See attached packet captures from the WAN port and internal ports of a representative router. We have approximately 500 cnPilot routers deployed. This traffic was coming from a small subset (approx. 22) randomly dispersed over our 20 tower sites and includes both r200 and r201.cnPilot Capture Data.zip (746.1 KB)

Thoughts?

Todd Wilson

1 Like
2

Hi Todd,
We are looking at your ticket. Can you eliminate clients from being the source of the SYN, by disconnecting all LAN & WiFi clients and then doing a packet capture from the WAN side on an affected device?

Also can you share the output of “cat /proc/net/ip_conntrack”.

From the techdump it did not seem that the CPU was too busy. Did you also observe any performance/connectivity problem ?

Are the units protected with safe admin passwords and controlled access from WAN side (limited allowed IPs that can access the device)? In general these would help.

ashutosh

3

Archive.zip (1.9 MB)
trace.cap = capture of router with ongoing issue
trace-2.cap = same router with all wifi and hard-wired devices disconnected
trace-3.cap = same router with all wifi and hard-wired devices disconnected after reboot
cnPilot r200P-0F6C71_1600442350985.gz.zip (24.8 KB)
Engineering file from same.
CPU load on this device was at or near 100%. The devices have a complex password but are configured with Public IP.

4

An update. Our firm has made the decision to abandon the cnPilot product line. This is due to three major issues that we faced this year: 1) necessity for customers to reboot routers several times per day to make wifi work, 2) 50% and 100% CPU bug, and 3) this botnet.

1 Like