Why CnMatrix Ex2052-P CPU load suddenly goes to 100%?

Aman_Chanpura · March 30, 2023, 6:52am

We didn’t make any changes in configuration, and CPu load suddenly goes upto 100% and started flapping, then we need to reboot the switch and CPU load came back down to 7%. (Varies between 5-15%)

PFR · March 30, 2023, 11:15am

I had same problem with one of the E2052 last week.
After reboot it’s working normal.
Huge problem was to connect to the switch with gui, ssh.
90% ping lost, dropped ssh session.
After try #197 reboot was done.

TamN · March 30, 2023, 1:19pm

Thanks Aman and Pshemo for reporting the high cpu utilization on cnMatrix. Engineering team is currently investigating it. The questions below would help our investigation.

What software version did you observe the high cpu usage? Had you seen this symptom on the same switch with previous version?
Was the symptom seen on multiple switches in the same network?
Do you run any mib tool or scanner such as Auvik to query the switch in your network?
Was switch accessible via SSH or serial console during this time? If so, please run the commands ‘show os stacktace’ and ‘show os cpu top’ (run this command a couple times), ‘show cpu-counter’. Please post the output of these commands.

Eric_Ozrelic · March 30, 2023, 4:34pm

We’ve had this happen and have gone over extensive troubleshooting with support with no resolution yet. The ticket 341091 is still open. Only way to resolve it appears to be a reboot.

Aman_Chanpura · March 30, 2023, 10:39pm

Hi Tam,

we are running on 4.5-r3, yes we also faced same type of CPU load increasing in previous version. couple of months ago, almost all switches CPU load was high- at that time we need to reboot all switches and CPU load went back to normal condition.
We are not running any tool/scanner in our network.
we haven’t tried to access switch via SSH or console during that period.

PFR · March 31, 2023, 4:25am

Soft 4.4-r3, no updateed from 6 months
First time on this device. No observed on others in this network
During high load ssh was able to connect in 197 try ! Ping Was 3000-5000, and 90% lost

4.No Auvik or other staff

Command result
ex2052-2p# show os cpu top
Mem: 300268K used, 211616K free, 80784K shrd, 0K buff, 81048K cached
CPU: 100% usr 0.0% sys 0.0% nic 0.0% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 0.00 0.00 0.00 1/108 2034
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
2032 2030 0 R 2452 0.4 0 0.0 /usr/bin/top -n2 -d2
1153 1077 0 S 1028m205.7 0 0.0 {CFA} /usr/bin/ISS.exe
1147 1077 0 S 1028m205.7 0 0.0 {TMR#} /usr/bin/ISS.exe
1160 1077 0 S 1028m205.7 0 0.0 {AstT} /usr/bin/ISS.exe
1163 1077 0 S 1028m205.7 0 0.0 {PoeT} /usr/bin/ISS.exe
1197 1077 0 S 1028m205.7 0 0.0 {AA} /usr/bin/ISS.exe
1198 1077 0 S 1028m205.7 0 0.0 {WSIF} /usr/bin/ISS.exe
1196 1077 0 S 1028m205.7 0 0.0 {LLDP} /usr/bin/ISS.exe
1171 1077 0 S 1028m205.7 0 0.0 {LNXT} /usr/bin/ISS.exe
1180 1077 0 S 1028m205.7 0 0.0 {RAD} /usr/bin/ISS.exe
1166 1077 0 S 1028m205.7 0 0.0 {SnpT} /usr/bin/ISS.exe
1145 1077 0 S 1028m205.7 0 0.0 {ev} /usr/bin/ISS.exe
1203 1077 0 S 1028m205.7 0 0.0 {EVLS} /usr/bin/ISS.exe
1179 1077 0 S 1028m205.7 0 0.0 {OSPF} /usr/bin/ISS.exe
1202 1077 0 S 1028m205.7 0 0.0 {DAL} /usr/bin/ISS.exe
1193 1077 0 S 1028m205.7 0 0.0 {RMN2} /usr/bin/ISS.exe
1156 1077 0 S 1028m205.7 0 0.0 {LNXS} /usr/bin/ISS.exe
1164 1077 0 S 1028m205.7 0 0.0 {VLAN} /usr/bin/ISS.exe
1146 1077 0 S 1028m205.7 0 0.0 {intTask.73} /usr/bin/ISS.exe
1158 1077 0 S 1028m205.7 0 0.0 {L2DS} /usr/bin/ISS.exe
1175 1077 0 S 1028m205.7 0 0.0 {ARP} /usr/bin/ISS.exe
1200 1077 0 S 1028m205.7 0 0.0 {SNT} /usr/bin/ISS.exe
1144 1077 0 S 1028m205.7 0 0.0 {intTask.72} /usr/bin/ISS.exe
1333 1077 0 S 1028m205.7 0 0.0 {MSR} /usr/bin/ISS.exe
1174 1077 0 S 1028m205.7 0 0.0 {RTM} /usr/bin/ISS.exe
1168 1077 0 S 1028m205.7 0 0.0 {RT6} /usr/bin/ISS.exe
1191 1077 0 S 1028m205.7 0 0.0 {HRT3} /usr/bin/ISS.exe
2029 1077 0 S 1028m205.7 0 0.0 {C301} /usr/bin/ISS.exe
1150 1077 0 S 1028m205.7 0 0.0 {CLIC} /usr/bin/ISS.exe
1173 1077 0 S 1028m205.7 0 0.0 {LXMC} /usr/bin/ISS.exe
1178 1077 0 S 1028m205.7 0 0.0 {SAT} /usr/bin/ISS.exe
1167 1077 0 S 1028m205.7 0 0.0 {QOS} /usr/bin/ISS.exe
1162 1077 0 S 1028m205.7 0 0.0 {LaTT} /usr/bin/ISS.exe
1161 1077 0 S 1028m205.7 0 0.0 {PIf} /usr/bin/ISS.exe
1186 1077 0 S 1028m205.7 0 0.0 {D6CL} /usr/bin/ISS.exe
1188 1077 0 S 1028m205.7 0 0.0 {HST} /usr/bin/ISS.exe
1165 1077 0 S 1028m205.7 0 0.0 {FLU} /usr/bin/ISS.exe
1190 1077 0 S 1028m205.7 0 0.0 {HRT2} /usr/bin/ISS.exe
1184 1077 0 S 1028m205.7 0 0.0 {DHC} /usr/bin/ISS.exe
1187 1077 0 S 1028m205.7 0 0.0 {IGMP} /usr/bin/ISS.exe
1157 1077 0 S 1028m205.7 0 0.0 {IPDB} /usr/bin/ISS.exe
1169 1077 0 S 1028m205.7 0 0.0 {LI6T} /usr/bin/ISS.exe
1177 1077 0 S 1028m205.7 0 0.0 {RIP} /usr/bin/ISS.exe
1189 1077 0 S 1028m205.7 0 0.0 {HRT1} /usr/bin/ISS.exe
1192 1077 0 S 1028m205.7 0 0.0 {RMON} /usr/bin/ISS.exe
1149 1077 0 S 1028m205.7 0 0.0 {SMT} /usr/bin/ISS.exe
1152 1077 0 S 1028m205.7 0 0.0 {VcmT} /usr/bin/ISS.exe
1182 1077 0 S 1028m205.7 0 0.0 {DHS} /usr/bin/ISS.exe
1183 1077 0 S 1028m205.7 0 0.0 {DHRL} /usr/bin/ISS.exe
1201 1077 0 S 1028m205.7 0 0.0 {DNSR} /usr/bin/ISS.exe
1151 1077 0 S 1028m205.7 0 0.0 {PKTT} /usr/bin/ISS.exe
1159 1077 0 S 1028m205.7 0 0.0 {CFAT} /usr/bin/ISS.exe
1170 1077 0 S 1028m205.7 0 0.0 {PNG6} /usr/bin/ISS.exe
1176 1077 0 S 1028m205.7 0 0.0 {PNG} /usr/bin/ISS.exe
1181 1077 0 S 1028m205.7 0 0.0 {TACT} /usr/bin/ISS.exe
1185 1077 0 S 1028m205.7 0 0.0 {DCS} /usr/bin/ISS.exe
1194 1077 0 S 1028m205.7 0 0.0 {CTS} /usr/bin/ISS.exe
1195 1077 0 S 1028m205.7 0 0.0 {SSH} /usr/bin/ISS.exe
1199 1077 0 S 1028m205.7 0 0.0 {cnAg} /usr/bin/ISS.exe
1204 1077 0 S 1028m205.7 0 0.0 {ExtL} /usr/bin/ISS.exe
1205 1077 0 S 1028m205.7 0 0.0 {RLD} /usr/bin/ISS.exe
1707 1137 0 S 18484 3.6 0 0.0 /usr/bin/device-agent
1135 1077 0 S 2664 0.5 0 0.0 /usr/bin/udevd
1141 1135 0 S 2660 0.5 0 0.0 /usr/bin/udevd
1441 1135 0 S 2660 0.5 0 0.0 /usr/bin/udevd
2034 2030 0 S 2452 0.4 0 0.0 head -c-2
2030 1138 0 S 2448 0.4 0 0.0 sh -c export LINES=100;export COLU
MNS=100;echo -n ‘H’|
2033 2030 0 S 2448 0.4 0 0.0 tail -n+2
1 0 0 S 2364 0.4 0 0.0 init
1077 1 0 S 2364 0.4 0 0.0 {init.sh} /bin/sh /etc/init.sh
1127 1077 0 S 2364 0.4 0 0.0 sh /etc/pushbutton.sh
1137 1077 0 S 2364 0.4 0 0.0 sh /etc/cda.sh
1126 1077 0 S < 2168 0.4 0 0.0 /usr/bin/swWdog
1131 1127 0 S 2148 0.4 0 0.0 /usr/bin/pushbutton
7 2 0 SW 0 0.0 0 0.0 [rcu_sched]
221 2 0 SW 0 0.0 0 0.0 [kworker/0:1]
3 2 0 SW 0 0.0 0 0.0 [ksoftirqd/0]
994 2 0 SW 0 0.0 0 0.0 [irq/31-f10d0000]
431 2 0 SW 0 0.0 0 0.0 [kworker/u2:3]
16 2 0 SW 0 0.0 0 0.0 [kworker/u2:2]
10 2 0 SW 0 0.0 0 0.0 [kdevtmpfs]
2 0 0 SW 0 0.0 0 0.0 [kthreadd]
5 2 0 SW< 0 0.0 0 0.0 [kworker/0:0H]
8 2 0 SW 0 0.0 0 0.0 [rcu_bh]
9 2 0 SW 0 0.0 0 0.0 [migration/0]
11 2 0 SW< 0 0.0 0 0.0 [perf]
14 2 0 SW< 0 0.0 0 0.0 [writeback]
219 2 0 SW< 0 0.0 0 0.0 [crypto]
220 2 0 SW< 0 0.0 0 0.0 [bioset]
223 2 0 SW< 0 0.0 0 0.0 [kblockd]
231 2 0 SW< 0 0.0 0 0.0 [ata_sff]
338 2 0 SW< 0 0.0 0 0.0 [rpciod]
351 2 0 SW 0 0.0 0 0.0 [kswapd0]
352 2 0 SW< 0 0.0 0 0.0 [vmstat]
353 2 0 SW 0 0.0 0 0.0 [fsnotify_mark]
354 2 0 SW< 0 0.0 0 0.0 [nfsiod]

CPU Total Receive Counters
Unicast Packets : 98736
Multicast Packets : 593738
Broadcast Packets : 108045
ARP Packets : 92883
IGMP Packets : 190037
IP Multicast Packets : 55149
STP Packets : 238895
LLDP Packets : 109657
DHCP Packets : 101
Other Packets : 113797

PFR · June 27, 2023, 3:50pm

And same thing again.
100% CPU, barely restarted it yesterday.
Same thing today for several minutes
For now, I’m getting on this switch, but what will happen in a moment.
Everything except access via SSH and WEB and ping works
All devices are up and running, only the administration interface is just not responding and the pings are either 7000 or 4000 and a loss of 99%.

ex2052-2p# show os cpu top
Mem: 301616K used, 210268K free, 80800K shrd, 0K buff, 81064K cached
CPU: 24.1% usr 75.8% sys 0.0% nic 0.0% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 1.40 1.35 1.33 2/108 4301
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
1190 1076 0 R 1028m205.7 0 68.7 {HRT3} /usr/bin/ISS.exe
1125 1076 0 S < 2168 0.4 0 27.1 /usr/bin/swWdog
4299 4297 0 R 2452 0.4 0 0.9 /usr/bin/top -n2 -d2
1152 1076 0 S 1028m205.7 0 0.4 {CFA} /usr/bin/ISS.exe
1146 1076 0 S 1028m205.7 0 0.4 {TMR#} /usr/bin/ISS.exe
1159 1076 0 S 1028m205.7 0 0.4 {AstT} /usr/bin/ISS.exe
1155 1076 0 S 1028m205.7 0 0.4 {LNXS} /usr/bin/ISS.exe
7 2 0 SW 0 0.0 0 0.4 [rcu_sched]
3 2 0 SW 0 0.0 0 0.4 [ksoftirqd/0]
1162 1076 0 S 1028m205.7 0 0.0 {PoeT} /usr/bin/ISS.exe
1197 1076 0 S 1028m205.7 0 0.0 {WSIF} /usr/bin/ISS.exe
1196 1076 0 S 1028m205.7 0 0.0 {AA} /usr/bin/ISS.exe
1195 1076 0 S 1028m205.7 0 0.0 {LLDP} /usr/bin/ISS.exe
1165 1076 0 S 1028m205.7 0 0.0 {SnpT} /usr/bin/ISS.exe
1192 1076 0 S 1028m205.7 0 0.0 {RMN2} /usr/bin/ISS.exe
1179 1076 0 S 1028m205.7 0 0.0 {RAD} /usr/bin/ISS.exe
1170 1076 0 S 1028m205.7 0 0.0 {LNXT} /usr/bin/ISS.exe
1144 1076 0 S 1028m205.7 0 0.0 {ev} /usr/bin/ISS.exe
1202 1076 0 S 1028m205.7 0 0.0 {EVLS} /usr/bin/ISS.exe
1178 1076 0 S 1028m205.7 0 0.0 {OSPF} /usr/bin/ISS.exe
1201 1076 0 S 1028m205.7 0 0.0 {DAL} /usr/bin/ISS.exe
1163 1076 0 S 1028m205.7 0 0.0 {VLAN} /usr/bin/ISS.exe
1145 1076 0 S 1028m205.7 0 0.0 {intTask.73} /usr/bin/ISS.exe
1137 1076 0 S 1028m205.7 0 0.0 /usr/bin/ISS.exe
1157 1076 0 S 1028m205.7 0 0.0 {L2DS} /usr/bin/ISS.exe
1174 1076 0 S 1028m205.7 0 0.0 {ARP} /usr/bin/ISS.exe
1143 1076 0 S 1028m205.7 0 0.0 {intTask.72} /usr/bin/ISS.exe
1332 1076 0 S 1028m205.7 0 0.0 {MSR} /usr/bin/ISS.exe
1199 1076 0 S 1028m205.7 0 0.0 {SNT} /usr/bin/ISS.exe
1173 1076 0 S 1028m205.7 0 0.0 {RTM} /usr/bin/ISS.exe
1167 1076 0 S 1028m205.7 0 0.0 {RT6} /usr/bin/ISS.exe
1189 1076 0 S 1028m205.7 0 0.0 {HRT2} /usr/bin/ISS.exe
4290 1076 0 S 1028m205.7 0 0.0 {C301} /usr/bin/ISS.exe
1149 1076 0 S 1028m205.7 0 0.0 {CLIC} /usr/bin/ISS.exe
1177 1076 0 S 1028m205.7 0 0.0 {SAT} /usr/bin/ISS.exe
1161 1076 0 S 1028m205.7 0 0.0 {LaTT} /usr/bin/ISS.exe
1166 1076 0 S 1028m205.7 0 0.0 {QOS} /usr/bin/ISS.exe
1188 1076 0 S 1028m205.7 0 0.0 {HRT1} /usr/bin/ISS.exe
1160 1076 0 S 1028m205.7 0 0.0 {PIf} /usr/bin/ISS.exe
1187 1076 0 S 1028m205.7 0 0.0 {HST} /usr/bin/ISS.exe
1164 1076 0 S 1028m205.7 0 0.0 {FLU} /usr/bin/ISS.exe
1185 1076 0 S 1028m205.7 0 0.0 {D6CL} /usr/bin/ISS.exe
1186 1076 0 S 1028m205.7 0 0.0 {IGMP} /usr/bin/ISS.exe
1156 1076 0 S 1028m205.7 0 0.0 {IPDB} /usr/bin/ISS.exe
1168 1076 0 S 1028m205.7 0 0.0 {LI6T} /usr/bin/ISS.exe
1176 1076 0 S 1028m205.7 0 0.0 {RIP} /usr/bin/ISS.exe
1183 1076 0 S 1028m205.7 0 0.0 {DHC} /usr/bin/ISS.exe
1191 1076 0 S 1028m205.7 0 0.0 {RMON} /usr/bin/ISS.exe
1148 1076 0 S 1028m205.7 0 0.0 {SMT} /usr/bin/ISS.exe
1151 1076 0 S 1028m205.7 0 0.0 {VcmT} /usr/bin/ISS.exe
1181 1076 0 S 1028m205.7 0 0.0 {DHS} /usr/bin/ISS.exe
1182 1076 0 S 1028m205.7 0 0.0 {DHRL} /usr/bin/ISS.exe
1184 1076 0 S 1028m205.7 0 0.0 {DCS} /usr/bin/ISS.exe
1200 1076 0 S 1028m205.7 0 0.0 {DNSR} /usr/bin/ISS.exe
1150 1076 0 S 1028m205.7 0 0.0 {PKTT} /usr/bin/ISS.exe
1158 1076 0 S 1028m205.7 0 0.0 {CFAT} /usr/bin/ISS.exe
1169 1076 0 S 1028m205.7 0 0.0 {PNG6} /usr/bin/ISS.exe
1172 1076 0 S 1028m205.7 0 0.0 {LXMC} /usr/bin/ISS.exe
1175 1076 0 S 1028m205.7 0 0.0 {PNG} /usr/bin/ISS.exe
1180 1076 0 S 1028m205.7 0 0.0 {TACT} /usr/bin/ISS.exe
1193 1076 0 S 1028m205.7 0 0.0 {CTS} /usr/bin/ISS.exe
1194 1076 0 S 1028m205.7 0 0.0 {SSH} /usr/bin/ISS.exe
1198 1076 0 S 1028m205.7 0 0.0 {cnAg} /usr/bin/ISS.exe
1203 1076 0 S 1028m205.7 0 0.0 {ExtL} /usr/bin/ISS.exe
1204 1076 0 S 1028m205.7 0 0.0 {RLD} /usr/bin/ISS.exe
1700 1136 0 S 18484 3.6 0 0.0 /usr/bin/device-agent
1134 1076 0 S 2664 0.5 0 0.0 /usr/bin/udevd
1140 1134 0 S 2660 0.5 0 0.0 /usr/bin/udevd
1440 1134 0 S 2660 0.5 0 0.0 /usr/bin/udevd
4301 4297 0 S 2452 0.4 0 0.0 head -c-2
4297 1137 0 S 2448 0.4 0 0.0 sh -c export LINES=100;export COLU
MNS=100;echo -n ‘H’|
4300 4297 0 S 2448 0.4 0 0.0 tail -n+2
1 0 0 S 2364 0.4 0 0.0 init
1076 1 0 S 2364 0.4 0 0.0 {init.sh} /bin/sh /etc/init.sh
1126 1076 0 S 2364 0.4 0 0.0 sh /etc/pushbutton.sh
1136 1076 0 S 2364 0.4 0 0.0 sh /etc/cda.sh
1133 1126 0 S 2148 0.4 0 0.0 /usr/bin/pushbutton
221 2 0 SW 0 0.0 0 0.0 [kworker/0:1]
993 2 0 SW 0 0.0 0 0.0 [irq/31-f10d0000]
6 2 0 SW 0 0.0 0 0.0 [kworker/u2:0]
16 2 0 SW 0 0.0 0 0.0 [kworker/u2:2]
10 2 0 SW 0 0.0 0 0.0 [kdevtmpfs]
2 0 0 SW 0 0.0 0 0.0 [kthreadd]
5 2 0 SW< 0 0.0 0 0.0 [kworker/0:0H]
8 2 0 SW 0 0.0 0 0.0 [rcu_bh]
9 2 0 SW 0 0.0 0 0.0 [migration/0]
11 2 0 SW< 0 0.0 0 0.0 [perf]
14 2 0 SW< 0 0.0 0 0.0 [writeback]
219 2 0 SW< 0 0.0 0 0.0 [crypto]
220 2 0 SW< 0 0.0 0 0.0 [bioset]
223 2 0 SW< 0 0.0 0 0.0 [kblockd]
231 2 0 SW< 0 0.0 0 0.0 [ata_sff]
338 2 0 SW< 0 0.0 0 0.0 [rpciod]
351 2 0 SW 0 0.0 0 0.0 [kswapd0]
352 2 0 SW< 0 0.0 0 0.0 [vmstat]
353 2 0 SW 0 0.0 0 0.0 [fsnotify_mark]

ex2052-2p# show cpu-counter

CPU Total Receive Counters
Unicast Packets : 58889
Multicast Packets : 254313
Broadcast Packets : 36004
ARP Packets : 35667
IGMP Packets : 39561
IP Multicast Packets : 80586
STP Packets : 91461
LLDP Packets : 42705
DHCP Packets : 101
Other Packets : 59125

ex2052-2p# show os stacktrace
0xb694e720 in nanosleep () from /lib/libc.so.6

Thread 61 (LWP 4290):
#0 0xb694e200 in waitpid () from /lib/libc.so.6
#1 0xb68ef36c in ?? () from /lib/libc.so.6
#2 0xb68ef928 in system () from /lib/libc.so.6
#3 0x01658c4c in OsixGetCmdOutput ()
#4 0x01852c58 in cli_process_os_cmd ()
#5 0x007f6580 in mmi_action_procedure ()
#6 0x0084fa00 in CliExecuteCommand ()
#7 0x0082824c in CliProcessCommand ()
#8 0x00823c1c in CliMainProcess ()
#9 0x00864e28 in CliSshTaskStart ()
#10 0x020bbdb4 in ?? ()
#11 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#12 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 60 (LWP 1332):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x019ec420 in MibSaveTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 59 (LWP 1204):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0165e820 in RELOAD_taskEntry ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 58 (LWP 1203):
#0 0xb6a12c1c in mq_timedreceive () from /lib/librt.so.1
#1 0xb6a12ad8 in mq_receive () from /lib/librt.so.1
#2 0x01636560 in sysLogExtProc ()
#3 0x020bbdb4 in ?? ()
#4 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#5 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 57 (LWP 1202):
#0 0xb697b114 in select () from /lib/libc.so.6
#1 0xb697b108 in select () from /lib/libc.so.6
#2 0x04b67bd0 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 56 (LWP 1201):
#0 0xb697b114 in select () from /lib/libc.so.6
#1 0xb697b108 in select () from /lib/libc.so.6
#2 0x04964f88 in CDAIF_traceFlag ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 55 (LWP 1200):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00c2bb78 in DnsResolverMainTask ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 54 (LWP 1199):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x015fa670 in SntpMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 53 (LWP 1198):
#0 0xb6eab4a8 in recvfrom () from /lib/libpthread.so.0
#1 0xb6eab49c in recvfrom () from /lib/libpthread.so.0
#2 0x00000152 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 52 (LWP 1197):
#0 0xb697b114 in select () from /lib/libc.so.6
#1 0xb697b108 in select () from /lib/libc.so.6
#2 0x04a7796c in gLastSyncSuccessful ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 51 (LWP 1196):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0056fa30 in AA_Main ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 50 (LWP 1195):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x004b5768 in LldpMainTask ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 49 (LWP 1194):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00864758 in CliSshTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 48 (LWP 1193):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0085764c in CliTelnetTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 47 (LWP 1192):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0165a1a4 in OsixReceiveEvent ()
#4 0x0103a58c in Rmon2MainTask ()
#5 0x020bbdb4 in ?? ()
#6 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#7 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 46 (LWP 1191):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00ffc870 in RmonMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 45 (LWP 1190):
#0 0xb6a12b60 in mq_timedsend () from /lib/librt.so.1
#1 0xb6e0c638 in petWatchdogVersion () from /usr/lib/libswd.so
#2 0xb6e0c8d0 in petWatchdog () from /usr/lib/libswd.so
#3 0x016bb468 in SelectWrapper ()
#4 0x00c1364c in HttpGetChar ()
#5 0x00c13b88 in HttpProcessReqLine ()
#6 0x00c1211c in ProcessHttpReq ()
#7 0x00c10ce8 in HttpStartProcessing ()
#8 0x00c10a28 in HttpProcHandleWakeUpEvent ()
#9 0x00c10888 in HttpProcessMain ()
#10 0x020bbdb4 in ?? ()
#11 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#12 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 44 (LWP 1189):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00c1084c in HttpProcessMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 43 (LWP 1188):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00c1084c in HttpProcessMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 42 (LWP 1187):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00c105b0 in HttpMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 41 (LWP 1186):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00ea9b10 in IgmpTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 40 (LWP 1185):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x008e8a8c in D6ClMainTask ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 39 (LWP 1184):
#0 0xb697b114 in select () from /lib/libc.so.6
#1 0xb697b108 in select () from /lib/libc.so.6
#2 0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 38 (LWP 1183):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0088e5ac in DhcpCTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 37 (LWP 1182):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x008ceb68 in DhcpRelayMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 36 (LWP 1181):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x008ad478 in DhcpTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 35 (LWP 1180):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00f85428 in TacacsClientMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 34 (LWP 1179):
#0 0xb697b114 in select () from /lib/libc.so.6
#1 0xb697b108 in select () from /lib/libc.so.6
#2 0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 33 (LWP 1178):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0165a1a4 in OsixReceiveEvent ()
#4 0x00dbd974 in OSPFTaskMain ()
#5 0x020bbdb4 in ?? ()
#6 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#7 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 32 (LWP 1177):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0165a1a4 in OsixReceiveEvent ()
#4 0x011c8984 in SnmpMain ()
#5 0x020bbdb4 in ?? ()
#6 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#7 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 31 (LWP 1176):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0165a1a4 in OsixReceiveEvent ()
#4 0x00fb9308 in RIPTaskMain ()
#5 0x020bbdb4 in ?? ()
#6 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#7 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 30 (LWP 1175):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x009abb5c in PingTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 29 (LWP 1174):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0087c19c in ArpTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 28 (LWP 1173):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0098ae64 in RtmTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 27 (LWP 1172):
#0 0xb6eab550 in recvmsg () from /lib/libpthread.so.0
#1 0xb6eab544 in recvmsg () from /lib/libpthread.so.0
#2 0x00000010 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 26 (LWP 1170):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00971a5c in LnxIpMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 25 (LWP 1169):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00a80540 in Ping6TaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 24 (LWP 1168):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x009dfc84 in Lip6MainInit ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 23 (LWP 1167):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0165a1a4 in OsixReceiveEvent ()
#4 0x009f3f80 in Rtm6TaskMain ()
#5 0x020bbdb4 in ?? ()
#6 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#7 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 22 (LWP 1166):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x01acbda4 in QoSInit ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 21 (LWP 1165):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x01214090 in SnoopMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 20 (LWP 1164):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0165a1a4 in OsixReceiveEvent ()
#4 0x00d2245c in VlanFlushTaskMain ()
#5 0x020bbdb4 in ?? ()
#6 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#7 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 19 (LWP 1163):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x013ea8f8 in VlanMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 18 (LWP 1162):
#0 0xb6eaba30 in nanosleep () from /lib/libpthread.so.0
#1 0xb6eaba24 in nanosleep () from /lib/libpthread.so.0
#2 0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 17 (LWP 1161):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x01371470 in LaMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 16 (LWP 1160):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0133ab00 in PnacInterfaceTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 15 (LWP 1159):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0108dc30 in AstTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 14 (LWP 1158):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x00696ad4 in CfaPacketTxTask ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 13 (LWP 1157):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0165a1a4 in OsixReceiveEvent ()
#4 0x006f5778 in L2DSMain ()
#5 0x020bbdb4 in ?? ()
#6 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#7 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 12 (LWP 1156):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0165a1a4 in OsixReceiveEvent ()
#4 0x006ddef4 in IPDBMain ()
#5 0x020bbdb4 in ?? ()
#6 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#7 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 11 (LWP 1155):
#0 0xb697b114 in select () from /lib/libc.so.6
#1 0xb697b108 in select () from /lib/libc.so.6
#2 0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 10 (LWP 1152):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0066ff14 in CfaMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 9 (LWP 1151):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x005e3e5c in VcmTaskMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 8 (LWP 1150):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0069664c in CfaPeriodicPktTxTsk ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (LWP 1149):
#0 0xb6eaafe0 in read () from /lib/libpthread.so.0
#1 0xb6eaafd4 in read () from /lib/libpthread.so.0
#2 0x03aaab44 in gCliSessions ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 6 (LWP 1148):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x01656c10 in OsixEvtRecv ()
#3 0x0162be70 in SysLogSmtpMain ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 5 (LWP 1146):
#0 0xb6eaba30 in nanosleep () from /lib/libpthread.so.0
#1 0xb6eaba24 in nanosleep () from /lib/libpthread.so.0
#2 0x00002710 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 4 (LWP 1145):
#0 0xb6eaafe4 in read () from /lib/libpthread.so.0
#1 0x020ab0f0 in ?? ()
#2 0x020bbdb4 in ?? ()
#3 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#4 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 3 (LWP 1144):
#0 0xb6ea7aa8 in pthread_cond_wait () from /lib/libpthread.so.0
#1 0x020bb938 in osSemWait ()
#2 0x020562fc in cpssEventSelect ()
#3 0x00cc63d4 in IssCpssEventHandler ()
#4 0x020bbdb4 in ?? ()
#5 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#6 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 2 (LWP 1143):
#0 0xb6eaafe4 in read () from /lib/libpthread.so.0
#1 0x020ab0f0 in ?? ()
#2 0x020bbdb4 in ?? ()
#3 0xb6ea2e8c in ?? () from /lib/libpthread.so.0
#4 0xb6981f08 in ?? () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (LWP 1137):
#0 0xb694e720 in nanosleep () from /lib/libc.so.6
#1 0xb694e714 in nanosleep () from /lib/libc.so.6
#2 0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

PFR · June 27, 2023, 4:16pm

There is a probable cause!
It starts to happen when there is no internet connection for a long time. So at least 20 minutes up.

TamN · June 27, 2023, 5:28pm

Thank you. We will analyze the traces.
Can you also post the ‘show running-config’?

djdrastic · June 28, 2023, 8:25am

Have you had any snmp sweeps or security compliance scanners ?
I have some customers where their flavor of NMS / Compliance Software will sometimes trigger this 100% bug at times.

Makoto · June 28, 2023, 7:31pm

Hi, reading this thread I took a look on my switch EX2010P and it he was on also 100% CPU dont know how many days I didnt notice any packet loss or something … reboot fixed it ! Fw 4.3.r3 !

PFR · June 29, 2023, 4:46am

This is a hotel. I have a total of 10 matrixes installed in it. This is the only thing that happens to this one.
They are all in one group with the same configuration and the same hardware.
I have SNMP turned off completely, there is no NMS in the network.
As I wrote, it starts to happen after there is no internet for a while?
the swWdog process seems to be firing something that kills the system.

num_ber · July 3, 2023, 8:49am

we are running on 4.5-r3, yes we also faced same type of CPU load increasing in previous version. couple of months ago, almost all switches CPU load was high- at that time we need to reboot all switches and CPU load went back to normal condition.

Jon_McNamara · November 10, 2023, 12:22pm

We’ve got similar issues with our switch where high CPU is related to “/usr/bin/ISS.exe”. Other than rebooting the switch, was there a permanent fix found?

This is a production switch so it’s not something I’m able to reboot easily during the day.

PFR · November 10, 2023, 3:57pm

I did’t observe this issue on 5.0.1

Eric_Ozrelic · November 11, 2023, 1:45am

They’ve resolved the 100% CPU issues in R5.0.1-R4:

TamN · November 11, 2023, 8:48pm

5.0.1-r4 included a fix for 100% cpu condition as described in the RN. It was reproducible in our engineering lab. 5.0.1-r4 also included improvements for cpu performance. There may be other cases that cause high cpu usage. We will continue to improve cpu utilization in the future releases.

StephenP · January 4, 2024, 9:23pm

We have 50 to 60 cnMatrix switches on our network, and we are still experiencing the 100% CPU utilization issue. I will be opening another support case on this tomorrow when I’m back in the office. I’m unable to capture any tech support logs since the switch keeps ending my remote session. We’ve had 2 to 3 switches running version 5.0.1-r4 experience this issue in the last two months. Is anyone else still experiencing this issue besides us?

Springs · January 22, 2024, 1:39pm

Any word on this yet?

StephenP · January 22, 2024, 1:56pm

We have a ticket open with technical support and are actively troubleshooting the issue. They have acknowledged the problem, and it is currently being investigated by the cnMatrix engineering team. I would urge you to open a ticket as well, so they are aware that others are having the issue. I will let you know if I make any progress on my end though.