Why CnMatrix Ex2052-P CPU load suddenly goes to 100%?

Products cnMatrix
1

We didn’t make any changes in configuration, and CPu load suddenly goes upto 100% and started flapping, then we need to reboot the switch and CPU load came back down to 7%. (Varies between 5-15%)

2

I had same problem with one of the E2052 last week.
After reboot it’s working normal.
Huge problem was to connect to the switch with gui, ssh.
90% ping lost, dropped ssh session.
After try #197 reboot was done.

3

Thanks Aman and Pshemo for reporting the high cpu utilization on cnMatrix. Engineering team is currently investigating it. The questions below would help our investigation.

  • What software version did you observe the high cpu usage? Had you seen this symptom on the same switch with previous version?
  • Was the symptom seen on multiple switches in the same network?
  • Do you run any mib tool or scanner such as Auvik to query the switch in your network?
  • Was switch accessible via SSH or serial console during this time? If so, please run the commands ‘show os stacktace’ and ‘show os cpu top’ (run this command a couple times), ‘show cpu-counter’. Please post the output of these commands.
4

We’ve had this happen and have gone over extensive troubleshooting with support with no resolution yet. The ticket 341091 is still open. Only way to resolve it appears to be a reboot.

1 Like
5

Hi Tam,

we are running on 4.5-r3, yes we also faced same type of CPU load increasing in previous version. couple of months ago, almost all switches CPU load was high- at that time we need to reboot all switches and CPU load went back to normal condition.
We are not running any tool/scanner in our network.
we haven’t tried to access switch via SSH or console during that period.

6

  1. Soft 4.4-r3, no updateed from 6 months

  2. First time on this device. No observed on others in this network

  3. During high load ssh was able to connect in 197 try ! Ping Was 3000-5000, and 90% lost

4.No Auvik or other staff

  1. Command result
    ex2052-2p# show os cpu top
    Mem: 300268K used, 211616K free, 80784K shrd, 0K buff, 81048K cached
    CPU: 100% usr 0.0% sys 0.0% nic 0.0% idle 0.0% io 0.0% irq 0.0% sirq
    Load average: 0.00 0.00 0.00 1/108 2034
    PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
    2032 2030 0 R 2452 0.4 0 0.0 /usr/bin/top -n2 -d2
    1153 1077 0 S 1028m205.7 0 0.0 {CFA} /usr/bin/ISS.exe
    1147 1077 0 S 1028m205.7 0 0.0 {TMR#} /usr/bin/ISS.exe
    1160 1077 0 S 1028m205.7 0 0.0 {AstT} /usr/bin/ISS.exe
    1163 1077 0 S 1028m205.7 0 0.0 {PoeT} /usr/bin/ISS.exe
    1197 1077 0 S 1028m205.7 0 0.0 {AA} /usr/bin/ISS.exe
    1198 1077 0 S 1028m205.7 0 0.0 {WSIF} /usr/bin/ISS.exe
    1196 1077 0 S 1028m205.7 0 0.0 {LLDP} /usr/bin/ISS.exe
    1171 1077 0 S 1028m205.7 0 0.0 {LNXT} /usr/bin/ISS.exe
    1180 1077 0 S 1028m205.7 0 0.0 {RAD} /usr/bin/ISS.exe
    1166 1077 0 S 1028m205.7 0 0.0 {SnpT} /usr/bin/ISS.exe
    1145 1077 0 S 1028m205.7 0 0.0 {ev} /usr/bin/ISS.exe
    1203 1077 0 S 1028m205.7 0 0.0 {EVLS} /usr/bin/ISS.exe
    1179 1077 0 S 1028m205.7 0 0.0 {OSPF} /usr/bin/ISS.exe
    1202 1077 0 S 1028m205.7 0 0.0 {DAL} /usr/bin/ISS.exe
    1193 1077 0 S 1028m205.7 0 0.0 {RMN2} /usr/bin/ISS.exe
    1156 1077 0 S 1028m205.7 0 0.0 {LNXS} /usr/bin/ISS.exe
    1164 1077 0 S 1028m205.7 0 0.0 {VLAN} /usr/bin/ISS.exe
    1146 1077 0 S 1028m205.7 0 0.0 {intTask.73} /usr/bin/ISS.exe
    1158 1077 0 S 1028m205.7 0 0.0 {L2DS} /usr/bin/ISS.exe
    1175 1077 0 S 1028m205.7 0 0.0 {ARP} /usr/bin/ISS.exe
    1200 1077 0 S 1028m205.7 0 0.0 {SNT} /usr/bin/ISS.exe
    1144 1077 0 S 1028m205.7 0 0.0 {intTask.72} /usr/bin/ISS.exe
    1333 1077 0 S 1028m205.7 0 0.0 {MSR} /usr/bin/ISS.exe
    1174 1077 0 S 1028m205.7 0 0.0 {RTM} /usr/bin/ISS.exe
    1168 1077 0 S 1028m205.7 0 0.0 {RT6} /usr/bin/ISS.exe
    1191 1077 0 S 1028m205.7 0 0.0 {HRT3} /usr/bin/ISS.exe
    2029 1077 0 S 1028m205.7 0 0.0 {C301} /usr/bin/ISS.exe
    1150 1077 0 S 1028m205.7 0 0.0 {CLIC} /usr/bin/ISS.exe
    1173 1077 0 S 1028m205.7 0 0.0 {LXMC} /usr/bin/ISS.exe
    1178 1077 0 S 1028m205.7 0 0.0 {SAT} /usr/bin/ISS.exe
    1167 1077 0 S 1028m205.7 0 0.0 {QOS} /usr/bin/ISS.exe
    1162 1077 0 S 1028m205.7 0 0.0 {LaTT} /usr/bin/ISS.exe
    1161 1077 0 S 1028m205.7 0 0.0 {PIf} /usr/bin/ISS.exe
    1186 1077 0 S 1028m205.7 0 0.0 {D6CL} /usr/bin/ISS.exe
    1188 1077 0 S 1028m205.7 0 0.0 {HST} /usr/bin/ISS.exe
    1165 1077 0 S 1028m205.7 0 0.0 {FLU} /usr/bin/ISS.exe
    1190 1077 0 S 1028m205.7 0 0.0 {HRT2} /usr/bin/ISS.exe
    1184 1077 0 S 1028m205.7 0 0.0 {DHC} /usr/bin/ISS.exe
    1187 1077 0 S 1028m205.7 0 0.0 {IGMP} /usr/bin/ISS.exe
    1157 1077 0 S 1028m205.7 0 0.0 {IPDB} /usr/bin/ISS.exe
    1169 1077 0 S 1028m205.7 0 0.0 {LI6T} /usr/bin/ISS.exe
    1177 1077 0 S 1028m205.7 0 0.0 {RIP} /usr/bin/ISS.exe
    1189 1077 0 S 1028m205.7 0 0.0 {HRT1} /usr/bin/ISS.exe
    1192 1077 0 S 1028m205.7 0 0.0 {RMON} /usr/bin/ISS.exe
    1149 1077 0 S 1028m205.7 0 0.0 {SMT} /usr/bin/ISS.exe
    1152 1077 0 S 1028m205.7 0 0.0 {VcmT} /usr/bin/ISS.exe
    1182 1077 0 S 1028m205.7 0 0.0 {DHS} /usr/bin/ISS.exe
    1183 1077 0 S 1028m205.7 0 0.0 {DHRL} /usr/bin/ISS.exe
    1201 1077 0 S 1028m205.7 0 0.0 {DNSR} /usr/bin/ISS.exe
    1151 1077 0 S 1028m205.7 0 0.0 {PKTT} /usr/bin/ISS.exe
    1159 1077 0 S 1028m205.7 0 0.0 {CFAT} /usr/bin/ISS.exe
    1170 1077 0 S 1028m205.7 0 0.0 {PNG6} /usr/bin/ISS.exe
    1176 1077 0 S 1028m205.7 0 0.0 {PNG} /usr/bin/ISS.exe
    1181 1077 0 S 1028m205.7 0 0.0 {TACT} /usr/bin/ISS.exe
    1185 1077 0 S 1028m205.7 0 0.0 {DCS} /usr/bin/ISS.exe
    1194 1077 0 S 1028m205.7 0 0.0 {CTS} /usr/bin/ISS.exe
    1195 1077 0 S 1028m205.7 0 0.0 {SSH} /usr/bin/ISS.exe
    1199 1077 0 S 1028m205.7 0 0.0 {cnAg} /usr/bin/ISS.exe
    1204 1077 0 S 1028m205.7 0 0.0 {ExtL} /usr/bin/ISS.exe
    1205 1077 0 S 1028m205.7 0 0.0 {RLD} /usr/bin/ISS.exe
    1707 1137 0 S 18484 3.6 0 0.0 /usr/bin/device-agent
    1135 1077 0 S 2664 0.5 0 0.0 /usr/bin/udevd
    1141 1135 0 S 2660 0.5 0 0.0 /usr/bin/udevd
    1441 1135 0 S 2660 0.5 0 0.0 /usr/bin/udevd
    2034 2030 0 S 2452 0.4 0 0.0 head -c-2
    2030 1138 0 S 2448 0.4 0 0.0 sh -c export LINES=100;export COLU
    MNS=100;echo -n ‘H’|
    2033 2030 0 S 2448 0.4 0 0.0 tail -n+2
    1 0 0 S 2364 0.4 0 0.0 init
    1077 1 0 S 2364 0.4 0 0.0 {init.sh} /bin/sh /etc/init.sh
    1127 1077 0 S 2364 0.4 0 0.0 sh /etc/pushbutton.sh
    1137 1077 0 S 2364 0.4 0 0.0 sh /etc/cda.sh
    1126 1077 0 S < 2168 0.4 0 0.0 /usr/bin/swWdog
    1131 1127 0 S 2148 0.4 0 0.0 /usr/bin/pushbutton
    7 2 0 SW 0 0.0 0 0.0 [rcu_sched]
    221 2 0 SW 0 0.0 0 0.0 [kworker/0:1]
    3 2 0 SW 0 0.0 0 0.0 [ksoftirqd/0]
    994 2 0 SW 0 0.0 0 0.0 [irq/31-f10d0000]
    431 2 0 SW 0 0.0 0 0.0 [kworker/u2:3]
    16 2 0 SW 0 0.0 0 0.0 [kworker/u2:2]
    10 2 0 SW 0 0.0 0 0.0 [kdevtmpfs]
    2 0 0 SW 0 0.0 0 0.0 [kthreadd]
    5 2 0 SW< 0 0.0 0 0.0 [kworker/0:0H]
    8 2 0 SW 0 0.0 0 0.0 [rcu_bh]
    9 2 0 SW 0 0.0 0 0.0 [migration/0]
    11 2 0 SW< 0 0.0 0 0.0 [perf]
    14 2 0 SW< 0 0.0 0 0.0 [writeback]
    219 2 0 SW< 0 0.0 0 0.0 [crypto]
    220 2 0 SW< 0 0.0 0 0.0 [bioset]
    223 2 0 SW< 0 0.0 0 0.0 [kblockd]
    231 2 0 SW< 0 0.0 0 0.0 [ata_sff]
    338 2 0 SW< 0 0.0 0 0.0 [rpciod]
    351 2 0 SW 0 0.0 0 0.0 [kswapd0]
    352 2 0 SW< 0 0.0 0 0.0 [vmstat]
    353 2 0 SW 0 0.0 0 0.0 [fsnotify_mark]
    354 2 0 SW< 0 0.0 0 0.0 [nfsiod]

    CPU Total Receive Counters
    Unicast Packets : 98736
    Multicast Packets : 593738
    Broadcast Packets : 108045
    ARP Packets : 92883
    IGMP Packets : 190037
    IP Multicast Packets : 55149
    STP Packets : 238895
    LLDP Packets : 109657
    DHCP Packets : 101
    Other Packets : 113797

1 Like