ePTP 40MHz crash

I have a 13.7 mile link on 2 foot dishes running in the 5.1 band. It was on R2.5 and 20MHz channel width for months with zero issues. We were recently hitting the capacity limit of the link (about 80Mbps to the slave) so I decided to switch the link to 40MHz. Enabled 40MHz bandwidth scanning on the slave, saved config, moved master to 40MHz on the same channel (5195) and saved config. Everything looked good.

Over the next few days, the slave was rebooting at least once per day. A couple times in the mornings, once around noon and a few more times in the evening. So I can't correlate it with traffic load or temperature. So I loaded 2.6 on both radios yesterday and hoped that would resolve it. It did not, the slave crashed and rebooted about 7AM today.

These are regular connectorized 5GHz radios with 00:04:56:C8 ESNs.

Here's the 2.6 crashlog from this morning. A kernel panic doesn't sound good...

Time: 1452862883.127804
Modules: umac@c0b68000+ad306 tdd_link_test@c0a6f000+eeb ath_dev@c0a27000+35962 ath_spectral@c09c4000+9d03 ath_dfs@c0907000+a2284 ath_rate_atheros@c08de000+a571 ath_hal@c081f000+a05ea tdd_unblockSA@c0797000+2b5 tdd_cdf@c078b000+ff4 adf@c077c000+2651 asf@c076c000+1b93 athrs_gmac@c0724000+b7eetdd_netlink_socket@c0444000+20b macvlan@c0437000+1140 nf_conntrack_netlink@c0427000+2820 ebt_ulog@c0416000+1080 ebt_ip@c0409000+520 ebt_arp@c03fd000+650 ebt_vlan@c03f1000+650 ebt_pkttype@c03e4000+260 ebt_limit@c03d8000+400 ebt_802_3@c03cc000+2f0 ebtable_filter@c03ba000+3b0 ebtable_broute@c03a8000+300 ebtables@c0399000+3ac5 nfnetlink@c0388000+7df nf_conntrack_tftp@c036f000+9b0 xt_HL@c0355000+560 xt_hl@c0349000+3e0 ipt_ECN@c033d000+580 xt_CLASSIFY@c0331000+240 xt_tcpmss@c0325000+430 xt_statistic@c0319000+380 xt_DSCP@c030d000+5d0 xt_dscp@c0301000+440 xt_quota@c02f5000+340 xt_pkttype@c02e9000+2a0xt_physdev@c02dd000+590 xt_owner@c02d1000+300 ipt_REDIRECT@c02c5000+2f0 ipt_NETMAP@c02b9000+2f0 ipt_MASQUERADE@c02ad000+440 iptable_nat@c02a1000+9b8 nf_nat@c0292000+2db0 xt_CONNMARK@c0282000+360 xt_recent@c0275000+1600 xt_helper@c0266000+390 xt_conntrack@c025a000+830 xt_connmark@c024d000+2e0 xt_connbytes@c0241000+530 xt_NOTRACK@c0235000+270 iptable_raw@c0229000+2e0 xt_state@c021d000+350 nf_conntrack_ipv4@c0210000+1fb2 nf_defrag_ipv4@c0200000+306 nf_conntrack@c01e9000+a219 pppoe@c01cc000+20c0 pppox@c01bd000+59a ipt_REJECT@c01b0000+710 xt_TCPMSS@c01a3000+7a0 ipt_LOG@c0195000+1110 xt_multiport@c0187000+750 xt_mac@c017b000+2a0 xt_limit@c016f000+440 iptable_mangle@c0163000+430 iptable_filter@c0156000+350 ip_tables@c0147000+2255 xt_tcpudp@c0138000+730 x_tables@c012a000+279e ppp_async@c0119000+1960 ppp_generic@c0106000+4b55 slhc@c00f1000+112b ts_fsm@c00e4000+a80 ts_bm@c00d7000+600 ts_kmp@c00cb000+590 crc_ccitt@c00bb000+42b cambium_iprst@c00a8000+bf0 leds_gpio@c0097000+5c0 button_hotplug@c008a000+a70 gpio_buttons@c007d000+880 input_polldev@c0070000+643 input_core@c005e000+469e
<5>Linux version 2.6.32.27 (builder@builder) (gcc version 4.3.3 (GCC) ) #2 Tue Dec 29 21:23:08 EET 2015
<4>flash_size passed from bootloader = 8
<4>arg 1: console=ttyS0,115200
<4>arg 2: root=31:02
<4>arg 3: noinitrd
<4>arg 4: mtdparts=ath-nor0:256k(u-boot),64k(u-boot-env),384K(config),7360K(uImage),64K(uImageI),64k(ART)
<4>arg 5: mem=64M
<6>CPU revision is: 0001974c (MIPS 74Kc)
<4>ath_sys_frequency: cpu srif ddr srif cpu 560 ddr 445 ahb 222
<6>Determined physical RAM map:
<6> memory: 02000000 @ 00000000 (usable)
<6>User-defined physical RAM map:
<6> memory: 04000000 @ 00000000 (usable)
<6>Initrd not found or empty - disabling initrd
<4>Zone PFN ranges:
<4> Normal 0x00000000 -> 0x00004000
<4>Movable zone start PFN for each node
<4>early_node_map[1] active PFN ranges
<4> 0: 0x00000000 -> 0x00004000
<7>On node 0 totalpages: 16384
<7>free_area_init_node: node 0, pgdat 802c5580, node_mem_map 819cb000
<7> Normal zone: 128 pages used for memmap
<7> Normal zone: 0 pages reserved
<7> Normal zone: 16256 pages, LIFO batch:3
<4>Built 1 zonelists in Zone order, mobility grouping on. Total pages: 16256
<5>Kernel command line: console=ttyS0,115200 root=31:02 noinitrd mtdparts=ath-nor0:256k(u-boot),64k(u-boot-env),384K(config),7360K(uImage),64K(uImageI),64k(ART) mem=64M
<6>PID hash table entries: 256 (order: -2, 1024 bytes)
<6>Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
<6>Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
<4>Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
<4>Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
<6>Writing ErrCtl register=00000000
<6>Readback ErrCtl register=00000000
<6>Memory: 38476k/65536k available (1970k kernel code, 26992k reserved, 486k data, 23472k init, 0k highmem)
<6>SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
<6>Hierarchical RCU implementation.
<6>NR_IRQS:128
<4>plat_time_init: plat time init done
<6>Calibrating delay loop... 279.34 BogoMIPS (lpj=1396736)
<4>Mount-cache hash table entries: 512
<6>NET: Registered protocol family 16
<4>bio: create slab <bio-0> at 0
<6>Switching to clocksource MIPS
<6>NET: Registered protocol family 2
<6>IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
<6>TCP established hash table entries: 2048 (order: 2, 16384 bytes)
<6>TCP bind hash table entries: 2048 (order: 1, 8192 bytes)
<6>TCP: Hash tables configured (established 2048 bind 2048)
<6>TCP reno registered
<6>NET: Registered protocol family 1
<4>ATH GPIOC major 0
<6>squashfs: version 4.0 (2009/01/31) Phillip Lougher
<4>Registering mini_fo version $Id$
<6>JFFS2 version 2.2 (NAND) (SUMMARY) (ZLIB) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
<4>yaffs Dec 29 2015 21:23:04 Installing.
<6>msgmni has been set to 75
<6>io scheduler noop registered
<6>io scheduler deadline registered (default)
<6>Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
<6>serial8250.0: ttyS0 at MMIO 0xb8020000 (irq = 19) is a 16550A
<6>console [ttyS0] enabled
<4>Flash Manuf Id 0xc2, DeviceId0 0x20, DeviceId1 0x17
<4>8MB Flash Detected
<5>6 cmdlinepart partitions found on MTD device ath-nor0
<5>Creating 6 MTD partitions on "ath-nor0":
<5>0x000000000000-0x000000040000 : "u-boot"
<5>0x000000040000-0x000000050000 : "u-boot-env"
<5>0x000000050000-0x0000000b0000 : "config"
<5>0x0000000b0000-0x0000007e0000 : "uImage"
<5>0x0000007e0000-0x0000007f0000 : "uImageI"
<5>0x0000007f0000-0x000000800000 : "ART"
<6>TCP westwood registered
<6>NET: Registered protocol family 17
<5>Bridge firewalling registered
<6>802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
<6>All bugs added by David S. Miller <davem@redhat.com>
<4>athwdt_init: Registering WDT success
<4>ath_otp_init: Registering OTP success
<4>ath_clksw_init: Registering Clock Switch Interface success
<4>apPx_spi_init: initialize Cambium New SPI Driver....
<4>Board SKU .... 0x1
<4>Entering ath_934x_config_init()
<4>Initialize GPIO PINs
<4>GPIO 18 and 21 set for XLNA0 and XLNA1
<4>apPx_rst_button_init: platform_device_regsiter: avenger_keys_device status = 0
<6>Freeing unused kernel memory: 23472k freed
<6>Button Hotplug driver version 0.3.1
<4>
<4>gpio_keys_probe: Initialize RST_BUTTON Driver....GPIO(16)
<4>gpio_keys_probe:gpio_keys_isr registered to IRQ = 48
<6>input: gpio-keys as /devices/platform/gpio-keys/input/input0
<6>PPP generic driver version 2.4.2
<6>ip_tables: (C) 2000-2006 Netfilter Core Team
<6>NET: Registered protocol family 24
<4>nf_conntrack version 0.5.0 (969 buckets, 3876 max)
<4>CONFIG_NF_CT_ACCT is deprecated and will be removed soon. Please use
<4>nf_conntrack.acct=1 kernel parameter, acct=1 nf_conntrack module option or
<4>sysctl net.netfilter.nf_conntrack_acct=1 to enable it.
<4>nf_nat_ftp: Unknown symbol nf_nat_ftp_hook
<4>Netfilter messages via NETLINK v0.30.
<6>Ebtables v2.0 registered
<4>ctnetlink v0.93: registering with nfnetlink.
<4>tdd_netlink_socket: module license 'Proprietary' taints kernel.
<4>Disabling lock debugging due to kernel taint
<4>ATHR_GMAC: Length per segment 1722
<4>ATHR_GMAC: fifo cfg 3 01f00140
<4>ATHR_GMAC: RX TASKLET - Pkts per Intr:16
<4>ATHR_GMAC: Mac address for unit 1:bfff0000
<4>ATHR_GMAC: 00:04:56:c8:ab:35
<4>mac:1 Registering S27...
<4>ATHR_GMAC: Max segments per packet : 1
<4>ATHR_GMAC: Max tx descriptor count : 128
<4>ATHR_GMAC: Max rx descriptor count : 128
<4>ATHR_GMAC: Mac capability flags : 2D88
<4>athr_gmac_ring_alloc Allocated 2048 at 0x8143c000
<4>athr_gmac_ring_alloc Allocated 2048 at 0x8143c800
<4>WASP ----> S27 PHY MDIO
<4>Setting Drop CRC Errors, Pause Frames and Length Error frames
<4>ATHRS27: resetting s27
<4>ATHRS27: s27 reset done
<4>Setting PHY...1
<4>TXFCTL enabled in Mac:1
<4>enet1 port4 up 100Mbps Full duplex
<4>ATHR_GMAC: Enet Unit:1 PHY:4 is UP RGMii 1000Mbps full duplex
<4>ATHR_GMAC: done cfg2 0x7215 ifctl 0x0 miictrl
<4>WASP ----> S27 PHY MDIO
<4>Setting Drop CRC Errors, Pause Frames and Length Error frames
<6>ath_hal: 0.9.17.1 (AR9380, DEBUG, WRITE_EEPROM, 11D)
<6>ath_rate_atheros: Copyright (c) 2001-2005 Atheros Communications, Inc, All Rights Reserved
<6>ath_dfs: Version 2.0.0
<6>Copyright (c) 2005-2006 Atheros Communications, Inc. All Rights Reserved
<6>ath_spectral: Version 2.0.0
<6>Copyright (c) 2005-2009 Atheros Communications, Inc. All Rights Reserved
<6>SPECTRAL module built on Dec 29 2015 21:20:22
<6>device eth0 entered promiscuous mode
<6>ath_dev: Copyright (c) 2001-2007 Atheros Communications, Inc, All Rights Reserved
<6>br-lan: port 1(eth0) entering forwarding state
<6>ath_ahb: 9.2.0_U10.5.13 (Atheros/multi-bss)
<4>AH_CAL_IN_FLASH_AHB defined
<4>__ath_attach: Set global_scn[0]
<4>ACBKMinfree = 48
<4>ACBEMinfree = 32
<4>ACVIMinfree = 16
<4>ACVOMinfree = 0
<4>CABMinfree = 48
<4>UAPSDMinfree = 0
<4>hal_conf_parm.calInFlash 1
<6>Bootstrap clock 40MHz
<6>Wasp Enterprise mode: 0x00000000
<6>ar9300RadioAttach: Need analog access recipe!!
<6>Restoring Cal data from Flash
<6>Restoring Cal data from Second Radio in Flash
<4>Don't allow 5.9 channels: cal peer[7]=5825
<7>dfs_attach: use DFS enhancements
<4>Use default DFS regisers
<7>DFS min filter rssiThresh = 17
<7>DFS max pulse dur = 151 ticks
<4>ath_get_caps[5786] rx chainmask mismatch actual 3 sc_chainmak 0
<4>ath_get_caps[5761] tx chainmask mismatch actual 3 sc_chainmak 0
<6>wifi0: Atheros 9340: mem=0xb8100000, irq=2
<4>TXFCTL enabled in Mac:1
<4>wlan_vap_create : enter. devhandle=0x83a802c0, opmode=IEEE80211_M_STA, flags=0x1
<4>wlan_vap_create : exit. devhandle=0x83a802c0, opmode=IEEE80211_M_STA, flags=0x1.
<4>Enhanced PTP slave mode - enabled
<6>device ath0 entered promiscuous mode
<6>br-lan: port 2(ath0) entering forwarding state
<4>ieee80211_ioctl_getparam : parameter drv:80211 0x0284 not supported
<4>connection is UP
<4>Adding WDS entry for 00:04:56:c8:ab:36, through ni=00:04:56:c8:a1:8e
<4>SM associated with AP[00:04:56:c8:a1:8e]
<4>Adding WDS entry for 00:0c:42:50:e7:f5, through ni=00:04:56:c8:a1:8e
<3>Assertion failed! !((tid->tx_buf_bitmap[cindex >> 5] & (1 << (cindex & (32-1)))) != 0):ath_tx_addto_baw /home/builder/jenkins/workspace/Avenger-Release/backfire_lsdk/package/at-aquila/drivers/wlan/os/linux/../../lmac/ath_dev/ath_xmit_ht.c:416
<4>Call Trace:
<4>[<802495e4>] dump_stack+0x8/0x34
<4>[<c0a439a4>] ath_tx_node_free+0x1e8/0x1f4 [ath_dev]
<4>[<c0bca318>] ieee80211_complete_wbuf+0x1d4/0x10a0 [umac]
<4>
<0>Kernel panic - not syncing: Take care of the assert first

George, 

This crash looks similar to the one we already know about. We are fixing it in 2.6.1 for which we will have a beta release on Monday along with the DDR timing fix. 

Please load the beta release on this link once its available. 

Thanks,

Sriram

1 Like

I forgot about this until the link was acting stupid and crashed again for about four hours this morning. Just loaded 2.6.1 official on it. Hopefully this goes away.

1 Like