Hi, we have 4/5 cpes that crashes every 10/15 minutes with the following debug crashlog
Time: 1465476057.473358
Modules: tdd_umac@c0cdc000+11d693 tdd_link_test@c0b3a000+eeb tdd_dev@c0ae8000+3d4b6 tdd_spectral@c0a7c000+9d13 tdd_dfs@c09c0000+a1f74 tdd_hal@c08ff000+9ba11 tdd_acm@c0877000+1e06 tdd_unblockSA@c0868000+2b5 tdd_pps@c085a000+2a0f gpslic@c084a000+46f tdd_cdf@c083e000+ff4 tdd_adf@c082f000+26d5 tdd_asf@c081f000+1b93 athrs_gmac@c07d7000+b7ee nf_nat_sip@c076b000+1470 nf_conntrack_sip@c075a000+313d nf_nat_snmp_basic@c0742000+1b10 nf_nat_pptp@c0733000+540 nf_conntrack_pptp@c0727000+c90 nf_nat_h323@c0719000+11c0 nf_conntrack_h323@c0703000+849f nf_nat_proto_gre@c06ee000+3c0 nf_conntrack_proto_gre@c06e2000+a55 nf_nat_amanda@c06d5000+2e0 nf_conntrack_amanda@c06c9000+6b0 tdd_netlink_socket@c0444000+20b macvlan@c0437000+1140 nf_conntrack_netlink@c0427000+2820 ebt_ulog@c0416000+1080 ebt_ip@c0409000+520 ebt_arp@c03fd000+650 ebt_vlan@c03f1000+650 ebt_pkttype@c03e4000+260 ebt_limit@c03d8000+400 ebt_802_3@c03cc000+2f0 ebtable_filter@c03ba000+3b0 ebtable_broute@c03a8000+300 ebtables@c0399000+3ac5 nfnetlink@c0388000+7df nf_nat_tftp@c037b000+210 nf_conntrack_tftp@c036f000+9b0 xt_HL@c0355000+560 xt_hl@c0349000+3e0 ipt_ECN@c033d000+580 xt_CLASSIFY@c0331000+240 xt_tcpmss@c0325000+430 xt_statistic@c0319000+380 xt_DSCP@c030d000+5d0 xt_dscp@c0301000+440 xt_quota@c02f5000+340 xt_pkttype@c02e9000+2a0 xt_physdev@c02dd000+590 xt_owner@c02d1000+300 ipt_REDIRECT@c02c5000+2f0 ipt_NETMAP@c02b9000+2f0 ipt_MASQUERADE@c02ad000+440 iptable_nat@c02a1000+9b8 nf_nat@c0292000+2db0 xt_CONNMARK@c0282000+360 xt_recent@c0275000+1600 xt_helper@c0266000+390 xt_conntrack@c025a000+830 xt_connmark@c024d000+2e0 xt_connbytes@c0241000+530 xt_NOTRACK@c0235000+270 iptable_raw@c0229000+2e0 xt_state@c021d000+350 nf_conntrack_ipv4@c0210000+1fb2 nf_defrag_ipv4@c0200000+306 nf_conntrack@c01e9000+a219 pppoe@c01cc000+20c0 pppox@c01bd000+59a ipt_REJECT@c01b0000+710 xt_TCPMSS@c01a3000+7a0 ipt_LOG@c0195000+1110 xt_multiport@c0187000+750 xt_mac@c017b000+2a0 xt_limit@c016f000+440 iptable_mangle@c0163000+430 iptable_filter@c0156000+350 ip_tables@c0147000+2255xt_tcpudp@c0138000+730 x_tables@c012a000+279e ppp_async@c0119000+1960 ppp_generic@c0106000+4b55 slhc@c00f1000+112b ts_fsm@c00e4000+a80 ts_bm@c00d7000+600 ts_kmp@c00cb000+590 crc_ccitt@c00bb000+42b cambium_iprst@c00a8000+bf0 leds_gpio@c0097000+5c0 button_hotplug@c008a000+a80 gpio_buttons@c007d000+880 input_polldev@c0070000+643 input_core@c005e000+469e
<5>Linux version 2.6.32.27 (builder@builder) (gcc version 4.3.3 (GCC) ) #2 Mon May 16 23:28:31 EEST 2016
<4>flash_size passed from bootloader = 8
<4>arg 1: console=ttyS0,115200
<4>arg 2: root=31:02
<4>arg 3: noinitrd
<4>arg 4: mtdparts=ath-nor0:256k(u-boot),64k(u-boot-env),384K(config),7360K(uImage),64K(uImageI),64k(ART)
<4>arg 5: mem=64M
<6>CPU revision is: 0001974c (MIPS 74Kc)
<4>ath_sys_frequency: cpu srif ddr srif cpu 560 ddr 400 ahb 200
<6>Determined physical RAM map:
<6> memory: 02000000 @ 00000000 (usable)
<6>User-defined physical RAM map:
<6> memory: 04000000 @ 00000000 (usable)
<6>Initrd not found or empty - disabling initrd
<4>Zone PFN ranges:
<4> Normal 0x00000000 -> 0x00004000
<4>Movable zone start PFN for each node
<4>early_node_map[1] active PFN ranges
<4> 0: 0x00000000 -> 0x00004000
<7>On node 0 totalpages: 16384
<7>free_area_init_node: node 0, pgdat 802c5580, node_mem_map 819f9000
<7> Normal zone: 128 pages used for memmap
<7> Normal zone: 0 pages reserved
<7> Normal zone: 16256 pages, LIFO batch:3
<4>Built 1 zonelists in Zone order, mobility grouping on. Total pages: 16256
<5>Kernel command line: console=ttyS0,115200 root=31:02 noinitrd mtdparts=ath-nor0:256k(u-boot),64k(u-boot-env),384K(config),7360K(uImage),64K(uImageI),64k(ART) mem=64M
<6>PID hash table entries: 256 (order: -2, 1024 bytes)
<6>Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
<6>Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
<4>Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
<4>Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
<6>Writing ErrCtl register=00000000
<6>Readback ErrCtl register=00000000
<6>Memory: 38292k/65536k available (1970k kernel code, 27176k reserved, 486k data, 23656k init, 0k highmem)
<6>SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
<6>Hierarchical RCU implementation.
<6>NR_IRQS:128
<4>plat_time_init: plat time init done
<6>Calibrating delay loop... 279.34 BogoMIPS (lpj=1396736)
<4>Mount-cache hash table entries: 512
<6>NET: Registered protocol family 16
<4>bio: create slab <bio-0> at 0
<6>Switching to clocksource MIPS
<6>NET: Registered protocol family 2
<6>IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
<6>TCP established hash table entries: 2048 (order: 2, 16384 bytes)
<6>TCP bind hash table entries: 2048 (order: 1, 8192 bytes)
<6>TCP: Hash tables configured (established 2048 bind 2048)
<6>TCP reno registered
<6>NET: Registered protocol family 1
<4>ATH GPIOC major 0
<6>squashfs: version 4.0 (2009/01/31) Phillip Lougher
<4>Registering mini_fo version $Id$
<6>JFFS2 version 2.2 (NAND) (SUMMARY) (ZLIB) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
<4>yaffs May 16 2016 23:28:27 Installing.
<6>msgmni has been set to 74
<6>io scheduler noop registered
<6>io scheduler deadline registered (default)
<6>Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
<6>serial8250.0: ttyS0 at MMIO 0xb8020000 (irq = 19) is a 16550A
<6>console [ttyS0] enabled
<4>Flash Manuf Id 0xef, DeviceId0 0x40, DeviceId1 0x17
<4>8MB Flash Detected
<5>6 cmdlinepart partitions found on MTD device ath-nor0
<5>Creating 6 MTD partitions on "ath-nor0":
<5>0x000000000000-0x000000040000 : "u-boot"
<5>0x000000040000-0x000000050000 : "u-boot-env"
<5>0x000000050000-0x0000000b0000 : "config"
<5>0x0000000b0000-0x0000007e0000 : "uImage"
<5>0x0000007e0000-0x0000007f0000 : "uImageI"
<5>0x0000007f0000-0x000000800000 : "ART"
<6>TCP westwood registered
<6>NET: Registered protocol family 17
<5>Bridge firewalling registered
<6>802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
<6>All bugs added by David S. Miller <davem@redhat.com>
<4>athwdt_init: Registering WDT success
<4>ath_otp_init: Registering OTP success
<4>ath_clksw_init: Registering Clock Switch Interface success
<4>apPx_spi_init: initialize Cambium New SPI Driver....
<4>Board SKU .... 0xb
<4>Entering ath_934x_config_init()
<4>Initialize GPIO PINs
<4>GPIO 14 and 15 set for XLNA0 and XLNA1
<4>apPx_rst_button_init: platform_device_regsiter: avenger_keys_device status = 0
<6>Freeing unused kernel memory: 23656k freed
<6>Button Hotplug driver version 0.3.1
<4>
<4>gpio_keys_probe: Initialize RST_BUTTON Driver....GPIO(16)
<4>gpio_keys_probe:gpio_keys_isr registered to IRQ = 48
<6>input: gpio-keys as /devices/platform/gpio-keys/input/input0
<6>PPP generic driver version 2.4.2
<6>ip_tables: (C) 2000-2006 Netfilter Core Team
<6>NET: Registered protocol family 24
<4>nf_conntrack version 0.5.0 (969 buckets, 3876 max)
<4>CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
<4>nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
<4>sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
<4>nf_nat_ftp: Unknown symbol nf_nat_ftp_hook
<4>Netfilter messages via NETLINK v0.30.
<6>Ebtables v2.0 registered
<4>ctnetlink v0.93: registering with nfnetlink.
<4>tdd_netlink_socket: module license 'Proprietary' taints kernel.
<4>Disabling lock debugging due to kernel taint
<4>ATHR_GMAC: Length per segment 1722
<4>ATHR_GMAC: fifo cfg 3 01f00140
<4>ATHR_GMAC: RX TASKLET - Pkts per Intr:16
<4>ATHR_GMAC: Mac address for unit 0:bfff0000
<4>ATHR_GMAC: 00:04:56:fb:13:aa
<4>mac:0 Registering AR8035...
<4>ATHR_GMAC: Max segments per packet : 1
<4>ATHR_GMAC: Max tx descriptor count : 128
<4>ATHR_GMAC: Max rx descriptor count : 128
<4>ATHR_GMAC: Mac capability flags : 8002381
<4>athr_gmac_ring_alloc Allocated 2048 at 0x814a8000
<4>athr_gmac_ring_alloc Allocated 2048 at 0x814a8800
<4>WASP ----> AR8035 PHY
<4>Setting Drop CRC Errors, Pause Frames and Length Error frames
<4>Setting PHY...0
<4>Wait for Autoneg to complete
<4>AR8035_PHY: Port 0, Neg Success
<4>AR8035_PHY: unit 0 phy addr 0 checking for the az feature of 8035...
<4>Disabling 802.3az feature...
<4>Restart auto-negotiation
<4>TXFCTL enabled in Mac:0
<6>tdd_hal: 0.9.17.1 (AR9380, DEBUG, WRITE_EEPROM, 11D)
<6>tdd_dfs: Version 2.0.0
<6>Copyright (c) 2005-2006 Atheros Communications, Inc. All Rights Reserved
<6>tdd_spectral: Version 2.0.0
<6>Copyright (c) 2005-2009 Atheros Communications, Inc. All Rights Reserved
<6>SPECTRAL module built on May 16 2016 23:19:52
<6>tdd_dev: Copyright (c) 2001-2007 Atheros Communications, Inc, All Rights Reserved
<6>ath_ahb: 9.2.0_U10.5.13 (Atheros/multi-bss)
<4>AH_CAL_IN_FLASH_AHB defined
<4>__ath_attach: Set global_scn[0]
<4>TxBuf flow control is disabled
<4>hal_conf_parm.calInFlash 1
<6>Bootstrap clock 40MHz
<6>Wasp Enterprise mode: 0x00000000
<6>ar9300RadioAttach: Need analog access recipe!!
<6>Restoring Cal data from Flash
<6>Restoring Cal data from Second Radio in Flash
<4>Allow 5.9 channels: cal peer[7]=5950
<4>ATH_MAC_TIMER: enet unit:0 is up...
<4>Mii 100Mbps full duplex
<4>ATH_MAC_TIMER: done cfg2 0x7115 ifctl 0x10000 miictrl
<7>dfs_attach: use DFS enhancements
<7>dfs_init_radar_filters: Unknown dfs domain 0
<4>ath_get_caps[5889] rx chainmask mismatch actual 3 sc_chainmak 0
<4>ath_get_caps[5864] tx chainmask mismatch actual 3 sc_chainmak 0
<4>tdd_classifier_init: ic=816402c0 classifier=81657488
<4>ath_descdma_setup: tx DMA: 1024 buffers 1 desc/buf 128 desc_len
<4>ath_descdma_setup: tx DMA map: 81580000 (135168) -> 1580000 (135168)
<4>ath_descdma_setup: success, name = tx, nbuf = 1024
<4>ath_descdma_setup: beacon DMA: 8 buffers 1 desc/buf 128 desc_len
<4>ath_descdma_setup: beacon DMA map: a1681000 (4096) -> 1681000 (4096)
<4>ath_descdma_setup: success, name = beacon, nbuf = 8
<6>wifi0: Atheros 9340: mem=0xb8100000, irq=2
<4>TXFCTL enabled in Mac:0
<4>camb_debug_print_setup: type=63, counter=3
<4>wlan_vap_create : enter. devhandle=0x816402c0, opmode=IEEE80211_M_STA, flags=0x1
<4>latency control is ON, latency value is 150000000
<4>wlan_vap_create : exit. devhandle=0x816402c0, opmode=IEEE80211_M_STA, flags=0x1.
<4>SWB: CAC time = 67000 MILLIseconds
<4>SWB: NOP time = 1800000 MILLIseconds
<4>50, 30, 20
<4>ieee80211_ioctl_getparam : parameter drv:tdd 0x284 not supported
<4>Wait for Autoneg to complete
<4>AR8035_PHY: Port 0, Neg Success
<4>AR8035_PHY: unit 0 phy addr 0 ATH_MAC_TIMER: unit 0: phy 4 not up carrier 1
<4>ATH_MAC_TIMER: enet unit:0 is up...
<4>Mii 100Mbps full duplex
<4>ATH_MAC_TIMER: done cfg2 0x7115 ifctl 0x10000 miictrl
<7>dfs_init_radar_filters: Unknown dfs domain 0
<4>TPC set initial Tx-Power to 30dbm
<4>Flags 0, Scheduler Mode 0
<4>connection is UP
<4>SM associated with AP[00:04:56:ce:89:2c]
<4>tdd_ratectrl_stat_add: suspicious ARQ statistic, ARQ fn = 34398, RA last FN = 0
<1>CPU 0 Unable to handle kernel paging request at virtual address 00004919, epc == c0af07bc, ra == c0af1804
<4>Oops[#1]:
<4>Cpu 0
<4>$ 0 : 00000000 00000001 00000000 00000004
<4>$ 4 : 830d0000 00000005 00000000 0000003b
<4>$ 8 : 00000029 e48a1b10 00000080 00000000
<4>$12 : 00000000 00000000 00000000 00000000
<4>$16 : 830d0000 830d1814 00000000 c0b20000
<4>$20 : b4b5aaca 000000b4 816402c0 830d0000
<4>$24 : 00000000 8007e3f0
<4>$28 : 802b2000 802b3850 830d0000 c0af1804
<4>Hi : 0000003b
<4>Lo : 00000001
<4>epc : c0af07bc ah_write_ccfreq_proc+0x50/0x238 [tdd_dev]
<4> Tainted: P
<4>ra : c0af1804 ath_tdd_sta_ul_stop_timer_handler+0x134/0x47c [tdd_dev]
<4>Status: 1100ff03 KERNEL EXL IE
<4>Cause : 00800008
<4>BadVA : 00004919
<4>PrId : 0001974c (MIPS 74Kc)
<4>Modules linked in: tdd_umac tdd_link_test(P) tdd_dev(P) tdd_spectral(P) tdd_dfs(P) tdd_hal(P) tdd_acm(P) tdd_unblockSA(P) tdd_pps(P) gpslic tdd_cdf tdd_adf tdd_asf(P) athrs_gmac nf_nat_sip nf_conntrack_sip nf_nat_snmp_basic nf_nat_pptp nf_conntrack_pptp nf_nat_h323 nf_conntrack_h323 nf_nat_proto_gre nf_conntrack_proto_gre nf_nat_amanda nf_conntrack_amanda tdd_netlink_socket(P) macvlan nf_conntrack_netlink ebt_ulog ebt_ip ebt_arp ebt_vlan ebt_pkttype ebt_limit ebt_802_3 ebtable_filter ebtable_broute ebtables nfnetlink nf_nat_tftp nf_conntrack_tftp xt_HL xt_hl ipt_ECN xt_CLASSIFY xt_tcpmss xt_statistic xt_DSCP xt_dscp xt_quota xt_pkttype xt_physdev xt_owner ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE iptable_nat nf_nat xt_CONNMARK xt_recent xt_helper xt_conntrack xt_connmark xt_connbytes xt_NOTRACK iptable_raw xt_state nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack pppoe pppox ipt_REJECT xt_TCPMSS ipt_LOG xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables ppp_async ppp_generic slhc ts_fsm ts_bm ts_kmp crc_ccitt cambium_iprst leds_gpio button_hotplug gpio_buttons input_polldev input_core
<4>Process swapper (pid: 0, threadinfo=802b2000, task=802b4b40, tls=00000000)
<4>Stack : 816402c0 830d0000 830d0000 816502c0 830d0000 816502c0 00000000 00000000
<4> 000956a0 816402c0 838c8000 c0cff658 00000006 00000005 00010000 830e0000
<4> 00000000 00000000 00000308 802b3e28 00000000 80063794 00000003 802b3e28
<4> 000c3099 1d57a810 81000000 007d964c 00000000 816402c0 8142bcc0 00000000
<4> 8142bcc0 816536ea 838c80c4 816536fc 81653820 0000007b 830d0000 813a3070
<4> ...
<4>Call Trace:
<4>[<c0af07bc>] ah_write_ccfreq_proc+0x50/0x238 [tdd_dev]
<4>
<4>
<4>Code: 27bd0018 27bdffe8 afb00010 <afbf0014> 9082492f 14400005 00808021 8fbf0014 8fb00010
<0>Kernel panic - not syncing: Fatal exception in interrupt