We didn’t make any changes in configuration, and CPu load suddenly goes upto 100% and started flapping, then we need to reboot the switch and CPU load came back down to 7%. (Varies between 5-15%)
I had same problem with one of the E2052 last week.
After reboot it’s working normal.
Huge problem was to connect to the switch with gui, ssh.
90% ping lost, dropped ssh session.
After try #197 reboot was done.
Thanks Aman and Pshemo for reporting the high cpu utilization on cnMatrix. Engineering team is currently investigating it. The questions below would help our investigation.
- What software version did you observe the high cpu usage? Had you seen this symptom on the same switch with previous version?
- Was the symptom seen on multiple switches in the same network?
- Do you run any mib tool or scanner such as Auvik to query the switch in your network?
- Was switch accessible via SSH or serial console during this time? If so, please run the commands ‘show os stacktace’ and ‘show os cpu top’ (run this command a couple times), ‘show cpu-counter’. Please post the output of these commands.
We’ve had this happen and have gone over extensive troubleshooting with support with no resolution yet. The ticket 341091 is still open. Only way to resolve it appears to be a reboot.
Hi Tam,
we are running on 4.5-r3, yes we also faced same type of CPU load increasing in previous version. couple of months ago, almost all switches CPU load was high- at that time we need to reboot all switches and CPU load went back to normal condition.
We are not running any tool/scanner in our network.
we haven’t tried to access switch via SSH or console during that period.
-
Soft 4.4-r3, no updateed from 6 months
-
First time on this device. No observed on others in this network
-
During high load ssh was able to connect in 197 try ! Ping Was 3000-5000, and 90% lost
4.No Auvik or other staff
-
Command result
ex2052-2p# show os cpu top
Mem: 300268K used, 211616K free, 80784K shrd, 0K buff, 81048K cached
CPU: 100% usr 0.0% sys 0.0% nic 0.0% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 0.00 0.00 0.00 1/108 2034
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
2032 2030 0 R 2452 0.4 0 0.0 /usr/bin/top -n2 -d2
1153 1077 0 S 1028m205.7 0 0.0 {CFA} /usr/bin/ISS.exe
1147 1077 0 S 1028m205.7 0 0.0 {TMR#} /usr/bin/ISS.exe
1160 1077 0 S 1028m205.7 0 0.0 {AstT} /usr/bin/ISS.exe
1163 1077 0 S 1028m205.7 0 0.0 {PoeT} /usr/bin/ISS.exe
1197 1077 0 S 1028m205.7 0 0.0 {AA} /usr/bin/ISS.exe
1198 1077 0 S 1028m205.7 0 0.0 {WSIF} /usr/bin/ISS.exe
1196 1077 0 S 1028m205.7 0 0.0 {LLDP} /usr/bin/ISS.exe
1171 1077 0 S 1028m205.7 0 0.0 {LNXT} /usr/bin/ISS.exe
1180 1077 0 S 1028m205.7 0 0.0 {RAD} /usr/bin/ISS.exe
1166 1077 0 S 1028m205.7 0 0.0 {SnpT} /usr/bin/ISS.exe
1145 1077 0 S 1028m205.7 0 0.0 {ev} /usr/bin/ISS.exe
1203 1077 0 S 1028m205.7 0 0.0 {EVLS} /usr/bin/ISS.exe
1179 1077 0 S 1028m205.7 0 0.0 {OSPF} /usr/bin/ISS.exe
1202 1077 0 S 1028m205.7 0 0.0 {DAL} /usr/bin/ISS.exe
1193 1077 0 S 1028m205.7 0 0.0 {RMN2} /usr/bin/ISS.exe
1156 1077 0 S 1028m205.7 0 0.0 {LNXS} /usr/bin/ISS.exe
1164 1077 0 S 1028m205.7 0 0.0 {VLAN} /usr/bin/ISS.exe
1146 1077 0 S 1028m205.7 0 0.0 {intTask.73} /usr/bin/ISS.exe
1158 1077 0 S 1028m205.7 0 0.0 {L2DS} /usr/bin/ISS.exe
1175 1077 0 S 1028m205.7 0 0.0 {ARP} /usr/bin/ISS.exe
1200 1077 0 S 1028m205.7 0 0.0 {SNT} /usr/bin/ISS.exe
1144 1077 0 S 1028m205.7 0 0.0 {intTask.72} /usr/bin/ISS.exe
1333 1077 0 S 1028m205.7 0 0.0 {MSR} /usr/bin/ISS.exe
1174 1077 0 S 1028m205.7 0 0.0 {RTM} /usr/bin/ISS.exe
1168 1077 0 S 1028m205.7 0 0.0 {RT6} /usr/bin/ISS.exe
1191 1077 0 S 1028m205.7 0 0.0 {HRT3} /usr/bin/ISS.exe
2029 1077 0 S 1028m205.7 0 0.0 {C301} /usr/bin/ISS.exe
1150 1077 0 S 1028m205.7 0 0.0 {CLIC} /usr/bin/ISS.exe
1173 1077 0 S 1028m205.7 0 0.0 {LXMC} /usr/bin/ISS.exe
1178 1077 0 S 1028m205.7 0 0.0 {SAT} /usr/bin/ISS.exe
1167 1077 0 S 1028m205.7 0 0.0 {QOS} /usr/bin/ISS.exe
1162 1077 0 S 1028m205.7 0 0.0 {LaTT} /usr/bin/ISS.exe
1161 1077 0 S 1028m205.7 0 0.0 {PIf} /usr/bin/ISS.exe
1186 1077 0 S 1028m205.7 0 0.0 {D6CL} /usr/bin/ISS.exe
1188 1077 0 S 1028m205.7 0 0.0 {HST} /usr/bin/ISS.exe
1165 1077 0 S 1028m205.7 0 0.0 {FLU} /usr/bin/ISS.exe
1190 1077 0 S 1028m205.7 0 0.0 {HRT2} /usr/bin/ISS.exe
1184 1077 0 S 1028m205.7 0 0.0 {DHC} /usr/bin/ISS.exe
1187 1077 0 S 1028m205.7 0 0.0 {IGMP} /usr/bin/ISS.exe
1157 1077 0 S 1028m205.7 0 0.0 {IPDB} /usr/bin/ISS.exe
1169 1077 0 S 1028m205.7 0 0.0 {LI6T} /usr/bin/ISS.exe
1177 1077 0 S 1028m205.7 0 0.0 {RIP} /usr/bin/ISS.exe
1189 1077 0 S 1028m205.7 0 0.0 {HRT1} /usr/bin/ISS.exe
1192 1077 0 S 1028m205.7 0 0.0 {RMON} /usr/bin/ISS.exe
1149 1077 0 S 1028m205.7 0 0.0 {SMT} /usr/bin/ISS.exe
1152 1077 0 S 1028m205.7 0 0.0 {VcmT} /usr/bin/ISS.exe
1182 1077 0 S 1028m205.7 0 0.0 {DHS} /usr/bin/ISS.exe
1183 1077 0 S 1028m205.7 0 0.0 {DHRL} /usr/bin/ISS.exe
1201 1077 0 S 1028m205.7 0 0.0 {DNSR} /usr/bin/ISS.exe
1151 1077 0 S 1028m205.7 0 0.0 {PKTT} /usr/bin/ISS.exe
1159 1077 0 S 1028m205.7 0 0.0 {CFAT} /usr/bin/ISS.exe
1170 1077 0 S 1028m205.7 0 0.0 {PNG6} /usr/bin/ISS.exe
1176 1077 0 S 1028m205.7 0 0.0 {PNG} /usr/bin/ISS.exe
1181 1077 0 S 1028m205.7 0 0.0 {TACT} /usr/bin/ISS.exe
1185 1077 0 S 1028m205.7 0 0.0 {DCS} /usr/bin/ISS.exe
1194 1077 0 S 1028m205.7 0 0.0 {CTS} /usr/bin/ISS.exe
1195 1077 0 S 1028m205.7 0 0.0 {SSH} /usr/bin/ISS.exe
1199 1077 0 S 1028m205.7 0 0.0 {cnAg} /usr/bin/ISS.exe
1204 1077 0 S 1028m205.7 0 0.0 {ExtL} /usr/bin/ISS.exe
1205 1077 0 S 1028m205.7 0 0.0 {RLD} /usr/bin/ISS.exe
1707 1137 0 S 18484 3.6 0 0.0 /usr/bin/device-agent
1135 1077 0 S 2664 0.5 0 0.0 /usr/bin/udevd
1141 1135 0 S 2660 0.5 0 0.0 /usr/bin/udevd
1441 1135 0 S 2660 0.5 0 0.0 /usr/bin/udevd
2034 2030 0 S 2452 0.4 0 0.0 head -c-2
2030 1138 0 S 2448 0.4 0 0.0 sh -c export LINES=100;export COLU
MNS=100;echo -n ‘H’|
2033 2030 0 S 2448 0.4 0 0.0 tail -n+2
1 0 0 S 2364 0.4 0 0.0 init
1077 1 0 S 2364 0.4 0 0.0 {init.sh} /bin/sh /etc/init.sh
1127 1077 0 S 2364 0.4 0 0.0 sh /etc/pushbutton.sh
1137 1077 0 S 2364 0.4 0 0.0 sh /etc/cda.sh
1126 1077 0 S < 2168 0.4 0 0.0 /usr/bin/swWdog
1131 1127 0 S 2148 0.4 0 0.0 /usr/bin/pushbutton
7 2 0 SW 0 0.0 0 0.0 [rcu_sched]
221 2 0 SW 0 0.0 0 0.0 [kworker/0:1]
3 2 0 SW 0 0.0 0 0.0 [ksoftirqd/0]
994 2 0 SW 0 0.0 0 0.0 [irq/31-f10d0000]
431 2 0 SW 0 0.0 0 0.0 [kworker/u2:3]
16 2 0 SW 0 0.0 0 0.0 [kworker/u2:2]
10 2 0 SW 0 0.0 0 0.0 [kdevtmpfs]
2 0 0 SW 0 0.0 0 0.0 [kthreadd]
5 2 0 SW< 0 0.0 0 0.0 [kworker/0:0H]
8 2 0 SW 0 0.0 0 0.0 [rcu_bh]
9 2 0 SW 0 0.0 0 0.0 [migration/0]
11 2 0 SW< 0 0.0 0 0.0 [perf]
14 2 0 SW< 0 0.0 0 0.0 [writeback]
219 2 0 SW< 0 0.0 0 0.0 [crypto]
220 2 0 SW< 0 0.0 0 0.0 [bioset]
223 2 0 SW< 0 0.0 0 0.0 [kblockd]
231 2 0 SW< 0 0.0 0 0.0 [ata_sff]
338 2 0 SW< 0 0.0 0 0.0 [rpciod]
351 2 0 SW 0 0.0 0 0.0 [kswapd0]
352 2 0 SW< 0 0.0 0 0.0 [vmstat]
353 2 0 SW 0 0.0 0 0.0 [fsnotify_mark]
354 2 0 SW< 0 0.0 0 0.0 [nfsiod]CPU Total Receive Counters
Unicast Packets : 98736
Multicast Packets : 593738
Broadcast Packets : 108045
ARP Packets : 92883
IGMP Packets : 190037
IP Multicast Packets : 55149
STP Packets : 238895
LLDP Packets : 109657
DHCP Packets : 101
Other Packets : 113797