450M AP not getting any mu-mimo gain.

castarritt · March 30, 2018, 2:55pm

Hello,

One of our deployed 450M APs (running 15.1.3), which has been working well for months with 90+ SMs, suddenly stopped doing any mu-mimo.

A multi-vc link test used to give a 200+ Mbps result, but now it is ~50 Mbps, and 100% of the VCs have a slot group size of 1 (ungrouped). When I look at the sounding statistics, almost all of the VCs have a channel distortion of around -8,-15, and are sounding state 4 (UNTRUSTED)" whereas our working 450M APs will show sounding state 3 (TRACKING), and with lower distortion values.

I ran spectrum analysis from the AP as well as a few of its SMs, and the channel we are on looks perfectly clean.

Anyone have any ideas?

DL: VC   18, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.61,-15.45}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   19, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.23,-12.40}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   20, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -7.64,-14.57}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   21, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -6.47,-10.58}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   22, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.63,-14.72}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   23, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.97,-16.23}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   24, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.55,-15.08}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   25, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.64,-15.10}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   26, reference SF:  377 (VALID), soundingState  2 (ASSESSING), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.79,-18.97}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   27, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.01,-15.22}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   28, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.83,-18.45}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   29, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.80,-18.16}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   30, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  6, channelDistortion { -9.52,-13.10}, nullingSNR -32.78,  cnResponseCountSM  39386, cnResponseCountAP    907, missedTagCount      0 
DL: VC   31, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.73,-15.41}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   32, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.42,-14.97}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   33, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.55,-14.88}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   34, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault 13 (VC_CEST+SM_ERROR+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.23,-16.11}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   35, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -7.24,-16.11}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   36, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.33,-14.74}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   37, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.43,-15.28}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   38, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.75,-16.68}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   40, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.41,-15.00}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   41, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.00,-12.05}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   42, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.62,-15.03}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   43, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.00,-14.90}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   44, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.30,-16.76}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   45, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.74,-14.33}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   46, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.72,-16.60}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   47, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.68,-13.85}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   48, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.55,-14.85}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   49, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.30,-13.07}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   50, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  8 (CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.54,-16.48}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   51, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.37,-18.60}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   52, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.00,-17.27}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   53, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.92,-17.11}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   54, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.23,-16.68}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   55, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.42,-18.01}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   56, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  1, channelDistortion { -8.56,-14.26}, nullingSNR -12.42,  cnResponseCountSM  22938, cnResponseCountAP     40, missedTagCount      0 
DL: VC   57, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.25,-16.17}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   58, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.65,-15.37}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   59, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.69,-16.53}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   60, reference SF:  881 (VALID), soundingState  3 (TRACKING), soundingFault  8 (CHANNEL_DISTORTION), mumimoVetoCount  9, channelDistortion {-10.33,-16.10}, nullingSNR -27.83,  cnResponseCountSM  15332, cnResponseCountAP  16088, missedTagCount      0 
DL: VC   61, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.76,-17.71}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   62, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.10,-16.87}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   63, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.34,-14.44}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   64, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault 13 (VC_CEST+SM_ERROR+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.97,-15.26}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   65, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount 36, channelDistortion { -9.93,-18.72}, nullingSNR -26.11,  cnResponseCountSM  65453, cnResponseCountAP  17421, missedTagCount      0 
DL: VC   66, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.18,-15.79}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   67, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.86,-18.11}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   68, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  1, channelDistortion { -9.20,-17.17}, nullingSNR -19.01,  cnResponseCountSM  10434, cnResponseCountAP     35, missedTagCount      0 
DL: VC   69, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.76,-18.05}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   70, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.62,-15.23}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   71, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.81,-15.87}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   72, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.35,-14.97}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   73, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.36,-16.55}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   74, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.59,-16.40}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   75, reference SF:  269 (VALID), soundingState  3 (TRACKING), soundingFault  8 (CHANNEL_DISTORTION), mumimoVetoCount 120, channelDistortion {-10.68,-15.57}, nullingSNR -20.28,  cnResponseCountSM  58548, cnResponseCountAP  19114, missedTagCount      0 
DL: VC   76, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.77,-16.15}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   77, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.64,-16.67}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   78, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.19,-13.24}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   79, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.97,-15.41}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   80, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.33,-16.60}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   81, reference SF:  908 (VALID), soundingState  3 (TRACKING), soundingFault  8 (CHANNEL_DISTORTION), mumimoVetoCount 32, channelDistortion {-10.36,-17.56}, nullingSNR -26.84,  cnResponseCountSM  14915, cnResponseCountAP  20501, missedTagCount      0 
DL: VC   82, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.90,-15.41}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   83, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.68,-15.96}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   84, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.61,-14.77}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   85, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.51,-17.49}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   86, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.91,-16.05}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   87, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount 46, channelDistortion { -9.84,-15.65}, nullingSNR -22.74,  cnResponseCountSM  37036, cnResponseCountAP  20948, missedTagCount      0 
DL: VC   88, reference SF:  895 (VALID), soundingState  3 (TRACKING), soundingFault 14 (NULLING_SNR+SM_ERROR+CHANNEL_DISTORTION), mumimoVetoCount 104, channelDistortion {-10.70,-16.30}, nullingSNR -21.27,  cnResponseCountSM  15493, cnResponseCountAP  24668, missedTagCount      0 
DL: VC   89, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault 25 (VC_CEST+CHANNEL_DISTORTION+UNSTABLE_SF), mumimoVetoCount  0, channelDistortion { -8.71,-16.48}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   90, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.97,-16.14}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   91, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.55,-15.29}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   92, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.89,-16.67}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   93, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.24,-15.65}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   94, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.86,-15.92}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   95, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.49,-14.69}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   96, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.75,-15.85}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   97, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -7.17,-14.62}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   98, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.69,-16.46}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC   99, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.99,-16.96}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  100, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -7.80,-12.75}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  101, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.10,-17.03}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  102, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.49,-14.93}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  103, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.30,-15.91}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  104, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.71,-15.55}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  105, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.80,-15.04}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  106, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.60,-17.55}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  107, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.71,-17.32}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  108, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.22,-15.91}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  109, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -8.11,-17.29}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0 
DL: VC  110, reference SF: 2048 (INVALID), soundingState  4 (UNTRUSTED), soundingFault  9 (VC_CEST+CHANNEL_DISTORTION), mumimoVetoCount  0, channelDistortion { -9.72,-16.24}, nullingSNR   0.00,  cnResponseCountSM      0, cnResponseCountAP      0, missedTagCount      0

Cambium_AndyR · March 30, 2018, 9:47pm

Hi castarritt,

That looks very odd.

Could collect some diagnostics from the unit (IPADDRESS/field_diags.cgi) and send them to me (andrew.rimmer@cambiumnetworks.com). I'll get them looked at (although this may take a little while as people are away for Easter holidays).

Also:

1) Might be worth a visiting the site personally to check antenna isn't misaligned or being obstructed.

2) raise a support ticket in case there is a hardware fault that needs a return.

best regards,

Andy.

Cambium_AndyR · April 18, 2018, 6:39pm

For the record we have tracked this down to a hardware fault.

castarritt collected some field_diags (up to half the SMs were showing MU-MIMO by this point) and after some new analysis on these we have deduced one of the RF chains is suspect, causing the channel distortion measurements to be affected and the MU-MIMO operation to be borderline at best.

The unit is being RMA'd to establish the root cause of any hardware fault as we haven't seen a failure of this type before.

Thanks,

Andy

Ryan_Ray · April 18, 2018, 6:54pm

Thanks for the update Andy. Great to see this kind of response quickly on hardware issues.

CambiumMatt · April 18, 2018, 7:42pm

I agree. Vigilance among 450m units helps us keep on top of what's going on, so the better we address these, the better off ALL of our customers will be.

Thanks Andy and team!