View details of system processes, memory used, interface parameters for advanced troubleshooting

The cnPilot AP CLI include commands that wrap around a number of common Linux system commands (the base operating system on the APs is Linux). These commands can be useful for advanced troubleshooing for debugging. These are mainly for engineering level debug and the commands are all under a 'service show' prefix as they are not needed during normal operation of the device, just for advanced debugging:

Service show memory : to view the state of allocated and available memory on the AP. For example:

cnWest-E400(config)# service show memory
             total         used         free       shared      buffers
Mem:        126208        71176        55032            0            0
-/+ buffers:              71176        55032
Swap:            0            0            0
cnWest-E400(config)#

Service show top : to view the state of every running process and a snapshot of the amount of CPU and memory it is using. If the device dashboard indicates 100% or 99% CPU utilization, this command will be useful to see which process is taking up all the CPU.

CPU:   9% usr   0% sys   0% nic  90% idle   0% io   0% irq   0% sirq
Load average: 0.00 0.01 0.05 1/39 5550
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 5550  5517 root     R     1504   1%   9% top -n1
  984   962 root     S     5580   4%   0% /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
  985   962 root     S     5444   4%   0% /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
  433     1 root     S     4604   4%   0% /usr/sbin/snmpd -f -c /var/run/etc/snmp/snmpd.conf -p /tmp/snmpd.pid
  432     1 root     S     4588   4%   0% /usr/sbin/wifid -d
29396     1 root     S     4456   4%   0% /usr/sbin/device-agent -d2 -shttps://cloud.cambiumnetworks.com
  430     1 root     S     4404   3%   0% /usr/sbin/scmd -d
  962     1 root     S     3904   3%   0% /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
 5517   441 root     S     3876   3%   0% -clish
 1140     1 root     S     3464   3%   0% /usr/bin/stunnel /var/run/etc/stunnel/stunnel.conf
  431     1 root     S     1620   1%   0% /usr/sbin/infrad -d
  441     1 root     S     1508   1%   0% /usr/sbin/telnetd -F
    1     0 root     S     1504   1%   0% init
  515     1 root     S     1504   1%   0% /usr/sbin/ntpd -n -p pool.ntp.org
  436     1 root     S     1504   1%   0% /sbin/getty -L ttyS0 115200 vt100
  320     1 root     S     1500   1%   0% /sbin/klogd -n

service show ps : to view the PIDs of every running process on the AP

cnWest-E400(config)# service show ps
PID   USER     COMMAND
    1 root     init
    2 root     [kthreadd]
    3 root     [ksoftirqd/0]
    5 root     [kworker/0:0H]
    7 root     [watchdog/0]
    8 root     [khelper]
    9 root     [kdevtmpfs]
   10 root     [kworker/u2:1]
   93 root     [writeback]
   96 root     [bioset]
   98 root     [kblockd]
  134 root     [kswapd0]
  179 root     [fsnotify_mark]
  190 root     [ath79-spi]
  286 root     [kpsmoused]
  301 root     [deferwq]
  309 root     [yaffs-bg-1]
  320 root     /sbin/klogd -n
  321 root     /sbin/syslogd -n
  355 root     /usr/sbin/rngd -r /dev/urandom
  429 root     /usr/sbin/sysmond -d
  430 root     /usr/sbin/scmd -d
  431 root     /usr/sbin/infrad -d

service show route : to view the route table on the AP. This will include routes learnt over DHCP (Eg: a default route learnt from DHCP options) as well as routes configured by the user.

cnWest-E400(config)# service show route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.140.134.254  0.0.0.0         UG    0      0        0 br0.1
10.140.134.0    0.0.0.0         255.255.255.0   U     0      0        0 br0.1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 br0.1
cnWest-E400(config)#

service show netstat : information on every socket open on the AP. This includes connections out to other servers from the AP, as well as all the ports on which the AP itself is either listening for connections or has a client now connected.

cnWest-E400(config)# service show netstat
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.1.1:8888          0.0.0.0:*               LISTEN 

service show ifconfig : information about all the networking interfaces on the AP, their state, their MAC address, the number of bytes tx and rx from each

cnWest-E400(config)# service show ifconfig
br0       Link encap:Ethernet  HWaddr 00:04:56:F8:33:90 
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5799458 errors:0 dropped:44 overruns:0 frame:0
          TX packets:474383 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:447833423 (427.0 MiB)  TX bytes:88194328 (84.1 MiB)
 
br0.1     Link encap:Ethernet  HWaddr 00:04:56:F8:33:90 
          inet addr:10.140.134.154  Bcast:10.140.134.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5799414 errors:0 dropped:3688 overruns:0 frame:0
          TX packets:474377 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:425410367 (405.7 MiB)  TX bytes:88193860 (84.1 MiB)

The br interfaces in ifconfig are bridged interfaces. Br0 is the default bridge and for every L3 vlan interface X a new interface called br0.X is created. In the example above br0.1 is the L3 vlan SVI for VLAN1 and the IP address learnt over that VLAN was assigned to it here.

The eth interfaces refer to the wired etherent ports of the AP. Depending on the AP model there might be one or more of these.

The wlan interface refer to the BSSIDs of the AP. Wlan0-wlan15 refer to all the BSSIDs on radio1. Wlan16-wlan31 refers to all the BSSIDs on radio2.

Service show iwconfig: shows the status of the radio interfaces (detailed information on each BSSID of the AP):

cnWest-E400(config)# service show iwconfig
wlan25    IEEE 802.11ac  ESSID:"" 
          Mode:Master  Frequency:5.745 GHz  Access Point: Not-Associated  
          Bit Rate:0 kb/s   Tx-Power=23 dBm  
          RTS thr:off   Fragment thr:off
          Encryption key:off
          Power Management:off
          Link Quality=0/94  Signal level=-95 dBm  Noise level=-95 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:0   Missed beacon:0

Service show dmesg : shows all the system startup messages, as well as radio-driver kernel logs. The radio driver logs are very verbose, and also the driver is common to a number of chipsets so a warning or even an error reported in the log might just be informational, for the radio-logs it is recommended to run things past Cambium support or Cambium engineering.

cnWest-E400(config)# service show dmesg|more
[4045418.180000] ar9300_handle_radar_bb_panic: BB status=0x04008009 rifs=0 - disable
[4045418.190000] ar9300_abort_tx_dma[938]: ar9300_stop_dma_receive failed
[4045418.190000] ar9300_reset[5836]: ar9300_stop_dma_receive failed
[4049876.910000] wmi_unified_event_rx : no registered event handler : event id 0x901b
[4049877.490000] [radio1] FWLOG: [120452248] WAL_DBGID_SECURITY_UCAST_KEY_SET ( 0x0 )
[4049877.500000] [radio1] FWLOG: [120452266] RATE: ChainMask 1, phymode 10, ni_flags 0x06013006, vht_mcs_set 0xfffe, ht_mcs_set 0x00ff, legacy_rate_set 0x72df4bc
[4049877.510000] [radio1] FWLOG: [120452284] WAL_DBGID_SECURITY_UCAST_KEY_SET ( 0x0 )
[4049877.520000] [radio1] FWLOG: [120452284] WAL_DBGID_SECURITY_ENCR_EN (  )
[4049877.530000] [radio1] FWLOG: [120452284] WAL_DBGID_SECURITY_ALLOW_DATA ( 0x436ee8 )
[4049877.530000] [radio1] FWLOG: [120452349] WAL_DBGID_TX_BA_SETUP ( 0x436ee8, 0x6, 0x2, 0x20, 0x1 )
[4049877.540000] [radio1] FWLOG: [120452350] RATE: ChainMask 1, phymode 10, ni_flags 0x06013006, vht_mcs_set 0xfffe, ht_mcs_set 0x00ff, legacy_rate_set 0x72df609

Service system-trace : this command can be used to invoke the strace program which displays all system calls made by the process. This is useful to help troubleshooting a process that might be spinning (100% CPU utilization) or appears stuck.

cnWest-E400(config)# service system-trace 432

Process 432 attached

clock_gettime(CLOCK_MONOTONIC, {4522705, 765175967}) = 0

gettimeofday({1464680180, 268531}, NULL) = 0

clock_gettime(CLOCK_MONOTONIC, {4522705, 766201541}) = 0

epoll_wait(4, {}, 130, 118)             = 0

clock_gettime(CLOCK_MONOTONIC, {4522705, 885457006}) = 0

clock_gettime(CLOCK_MONOTONIC, {4522705, 885954506}) = 0