Why CnMatrix Ex2052-P CPU load suddenly goes to 100%?

We didn’t make any changes in configuration, and CPu load suddenly goes upto 100% and started flapping, then we need to reboot the switch and CPU load came back down to 7%. (Varies between 5-15%)

I had same problem with one of the E2052 last week.
After reboot it’s working normal.
Huge problem was to connect to the switch with gui, ssh.
90% ping lost, dropped ssh session.
After try #197 reboot was done.

Thanks Aman and Pshemo for reporting the high cpu utilization on cnMatrix. Engineering team is currently investigating it. The questions below would help our investigation.

  • What software version did you observe the high cpu usage? Had you seen this symptom on the same switch with previous version?
  • Was the symptom seen on multiple switches in the same network?
  • Do you run any mib tool or scanner such as Auvik to query the switch in your network?
  • Was switch accessible via SSH or serial console during this time? If so, please run the commands ‘show os stacktace’ and ‘show os cpu top’ (run this command a couple times), ‘show cpu-counter’. Please post the output of these commands.

We’ve had this happen and have gone over extensive troubleshooting with support with no resolution yet. The ticket 341091 is still open. Only way to resolve it appears to be a reboot.

1 Like

Hi Tam,

we are running on 4.5-r3, yes we also faced same type of CPU load increasing in previous version. couple of months ago, almost all switches CPU load was high- at that time we need to reboot all switches and CPU load went back to normal condition.
We are not running any tool/scanner in our network.
we haven’t tried to access switch via SSH or console during that period.

  1. Soft 4.4-r3, no updateed from 6 months

  2. First time on this device. No observed on others in this network

  3. During high load ssh was able to connect in 197 try ! Ping Was 3000-5000, and 90% lost

4.No Auvik or other staff

  1. Command result
    ex2052-2p# show os cpu top
    Mem: 300268K used, 211616K free, 80784K shrd, 0K buff, 81048K cached
    CPU: 100% usr 0.0% sys 0.0% nic 0.0% idle 0.0% io 0.0% irq 0.0% sirq
    Load average: 0.00 0.00 0.00 1/108 2034
    PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
    2032 2030 0 R 2452 0.4 0 0.0 /usr/bin/top -n2 -d2
    1153 1077 0 S 1028m205.7 0 0.0 {CFA} /usr/bin/ISS.exe
    1147 1077 0 S 1028m205.7 0 0.0 {TMR#} /usr/bin/ISS.exe
    1160 1077 0 S 1028m205.7 0 0.0 {AstT} /usr/bin/ISS.exe
    1163 1077 0 S 1028m205.7 0 0.0 {PoeT} /usr/bin/ISS.exe
    1197 1077 0 S 1028m205.7 0 0.0 {AA} /usr/bin/ISS.exe
    1198 1077 0 S 1028m205.7 0 0.0 {WSIF} /usr/bin/ISS.exe
    1196 1077 0 S 1028m205.7 0 0.0 {LLDP} /usr/bin/ISS.exe
    1171 1077 0 S 1028m205.7 0 0.0 {LNXT} /usr/bin/ISS.exe
    1180 1077 0 S 1028m205.7 0 0.0 {RAD} /usr/bin/ISS.exe
    1166 1077 0 S 1028m205.7 0 0.0 {SnpT} /usr/bin/ISS.exe
    1145 1077 0 S 1028m205.7 0 0.0 {ev} /usr/bin/ISS.exe
    1203 1077 0 S 1028m205.7 0 0.0 {EVLS} /usr/bin/ISS.exe
    1179 1077 0 S 1028m205.7 0 0.0 {OSPF} /usr/bin/ISS.exe
    1202 1077 0 S 1028m205.7 0 0.0 {DAL} /usr/bin/ISS.exe
    1193 1077 0 S 1028m205.7 0 0.0 {RMN2} /usr/bin/ISS.exe
    1156 1077 0 S 1028m205.7 0 0.0 {LNXS} /usr/bin/ISS.exe
    1164 1077 0 S 1028m205.7 0 0.0 {VLAN} /usr/bin/ISS.exe
    1146 1077 0 S 1028m205.7 0 0.0 {intTask.73} /usr/bin/ISS.exe
    1158 1077 0 S 1028m205.7 0 0.0 {L2DS} /usr/bin/ISS.exe
    1175 1077 0 S 1028m205.7 0 0.0 {ARP} /usr/bin/ISS.exe
    1200 1077 0 S 1028m205.7 0 0.0 {SNT} /usr/bin/ISS.exe
    1144 1077 0 S 1028m205.7 0 0.0 {intTask.72} /usr/bin/ISS.exe
    1333 1077 0 S 1028m205.7 0 0.0 {MSR} /usr/bin/ISS.exe
    1174 1077 0 S 1028m205.7 0 0.0 {RTM} /usr/bin/ISS.exe
    1168 1077 0 S 1028m205.7 0 0.0 {RT6} /usr/bin/ISS.exe
    1191 1077 0 S 1028m205.7 0 0.0 {HRT3} /usr/bin/ISS.exe
    2029 1077 0 S 1028m205.7 0 0.0 {C301} /usr/bin/ISS.exe
    1150 1077 0 S 1028m205.7 0 0.0 {CLIC} /usr/bin/ISS.exe
    1173 1077 0 S 1028m205.7 0 0.0 {LXMC} /usr/bin/ISS.exe
    1178 1077 0 S 1028m205.7 0 0.0 {SAT} /usr/bin/ISS.exe
    1167 1077 0 S 1028m205.7 0 0.0 {QOS} /usr/bin/ISS.exe
    1162 1077 0 S 1028m205.7 0 0.0 {LaTT} /usr/bin/ISS.exe
    1161 1077 0 S 1028m205.7 0 0.0 {PIf} /usr/bin/ISS.exe
    1186 1077 0 S 1028m205.7 0 0.0 {D6CL} /usr/bin/ISS.exe
    1188 1077 0 S 1028m205.7 0 0.0 {HST} /usr/bin/ISS.exe
    1165 1077 0 S 1028m205.7 0 0.0 {FLU} /usr/bin/ISS.exe
    1190 1077 0 S 1028m205.7 0 0.0 {HRT2} /usr/bin/ISS.exe
    1184 1077 0 S 1028m205.7 0 0.0 {DHC} /usr/bin/ISS.exe
    1187 1077 0 S 1028m205.7 0 0.0 {IGMP} /usr/bin/ISS.exe
    1157 1077 0 S 1028m205.7 0 0.0 {IPDB} /usr/bin/ISS.exe
    1169 1077 0 S 1028m205.7 0 0.0 {LI6T} /usr/bin/ISS.exe
    1177 1077 0 S 1028m205.7 0 0.0 {RIP} /usr/bin/ISS.exe
    1189 1077 0 S 1028m205.7 0 0.0 {HRT1} /usr/bin/ISS.exe
    1192 1077 0 S 1028m205.7 0 0.0 {RMON} /usr/bin/ISS.exe
    1149 1077 0 S 1028m205.7 0 0.0 {SMT} /usr/bin/ISS.exe
    1152 1077 0 S 1028m205.7 0 0.0 {VcmT} /usr/bin/ISS.exe
    1182 1077 0 S 1028m205.7 0 0.0 {DHS} /usr/bin/ISS.exe
    1183 1077 0 S 1028m205.7 0 0.0 {DHRL} /usr/bin/ISS.exe
    1201 1077 0 S 1028m205.7 0 0.0 {DNSR} /usr/bin/ISS.exe
    1151 1077 0 S 1028m205.7 0 0.0 {PKTT} /usr/bin/ISS.exe
    1159 1077 0 S 1028m205.7 0 0.0 {CFAT} /usr/bin/ISS.exe
    1170 1077 0 S 1028m205.7 0 0.0 {PNG6} /usr/bin/ISS.exe
    1176 1077 0 S 1028m205.7 0 0.0 {PNG} /usr/bin/ISS.exe
    1181 1077 0 S 1028m205.7 0 0.0 {TACT} /usr/bin/ISS.exe
    1185 1077 0 S 1028m205.7 0 0.0 {DCS} /usr/bin/ISS.exe
    1194 1077 0 S 1028m205.7 0 0.0 {CTS} /usr/bin/ISS.exe
    1195 1077 0 S 1028m205.7 0 0.0 {SSH} /usr/bin/ISS.exe
    1199 1077 0 S 1028m205.7 0 0.0 {cnAg} /usr/bin/ISS.exe
    1204 1077 0 S 1028m205.7 0 0.0 {ExtL} /usr/bin/ISS.exe
    1205 1077 0 S 1028m205.7 0 0.0 {RLD} /usr/bin/ISS.exe
    1707 1137 0 S 18484 3.6 0 0.0 /usr/bin/device-agent
    1135 1077 0 S 2664 0.5 0 0.0 /usr/bin/udevd
    1141 1135 0 S 2660 0.5 0 0.0 /usr/bin/udevd
    1441 1135 0 S 2660 0.5 0 0.0 /usr/bin/udevd
    2034 2030 0 S 2452 0.4 0 0.0 head -c-2
    2030 1138 0 S 2448 0.4 0 0.0 sh -c export LINES=100;export COLU
    MNS=100;echo -n ‘H’|
    2033 2030 0 S 2448 0.4 0 0.0 tail -n+2
    1 0 0 S 2364 0.4 0 0.0 init
    1077 1 0 S 2364 0.4 0 0.0 {init.sh} /bin/sh /etc/init.sh
    1127 1077 0 S 2364 0.4 0 0.0 sh /etc/pushbutton.sh
    1137 1077 0 S 2364 0.4 0 0.0 sh /etc/cda.sh
    1126 1077 0 S < 2168 0.4 0 0.0 /usr/bin/swWdog
    1131 1127 0 S 2148 0.4 0 0.0 /usr/bin/pushbutton
    7 2 0 SW 0 0.0 0 0.0 [rcu_sched]
    221 2 0 SW 0 0.0 0 0.0 [kworker/0:1]
    3 2 0 SW 0 0.0 0 0.0 [ksoftirqd/0]
    994 2 0 SW 0 0.0 0 0.0 [irq/31-f10d0000]
    431 2 0 SW 0 0.0 0 0.0 [kworker/u2:3]
    16 2 0 SW 0 0.0 0 0.0 [kworker/u2:2]
    10 2 0 SW 0 0.0 0 0.0 [kdevtmpfs]
    2 0 0 SW 0 0.0 0 0.0 [kthreadd]
    5 2 0 SW< 0 0.0 0 0.0 [kworker/0:0H]
    8 2 0 SW 0 0.0 0 0.0 [rcu_bh]
    9 2 0 SW 0 0.0 0 0.0 [migration/0]
    11 2 0 SW< 0 0.0 0 0.0 [perf]
    14 2 0 SW< 0 0.0 0 0.0 [writeback]
    219 2 0 SW< 0 0.0 0 0.0 [crypto]
    220 2 0 SW< 0 0.0 0 0.0 [bioset]
    223 2 0 SW< 0 0.0 0 0.0 [kblockd]
    231 2 0 SW< 0 0.0 0 0.0 [ata_sff]
    338 2 0 SW< 0 0.0 0 0.0 [rpciod]
    351 2 0 SW 0 0.0 0 0.0 [kswapd0]
    352 2 0 SW< 0 0.0 0 0.0 [vmstat]
    353 2 0 SW 0 0.0 0 0.0 [fsnotify_mark]
    354 2 0 SW< 0 0.0 0 0.0 [nfsiod]

    CPU Total Receive Counters
    Unicast Packets : 98736
    Multicast Packets : 593738
    Broadcast Packets : 108045
    ARP Packets : 92883
    IGMP Packets : 190037
    IP Multicast Packets : 55149
    STP Packets : 238895
    LLDP Packets : 109657
    DHCP Packets : 101
    Other Packets : 113797

1 Like