I have a 4500AP with 5 SMs in a pretty good MU-MIMO favorable distribution.
I ran a 2 SM 20s link test to two SMs that are very far apart on the azimuth. Both are near MCS10/11.
On a 75/25 split and 40mhz channel the AP showed near 90+% MU-MIMO traffic but each SM only got ~160Mbps. I then did a 3 SM test and each got around ~160Mbps. That was lower than I remember.
I then connected to the router behind one client (mikrotik) and ran a bandwidth test back to our core and with just that SM doing traffic it got a little under 300Mbps, which seemed low as it is getting MCS11/10 easy. While it was running I ran a 2 SM link test to two other SMs and watched the traffic in realtime on the Mikrotik router drop to under 30Mbps while the link tests to the other SMs again was around ~160Mbps.
I then rebooted the AP and after all SMs reconnected I did the same tests and this time with a 2 SM link test I got around 600-700 total on the DL with each SM getting around 300+.
I then did the same test where I had the one SM’s mikrotik router doing a DL bandwidth test and alone it would get ~345Mbps and during a 2 SM link test (two other SMs) this one dropped only to around ~240Mbps instead of 20Mbps and the AP showed that during that 2 SM link test with a 3rd SM doing the Mikrotik bandwidth test the AP was near 800-900Mbps where before the reboot it was only pushing around 500-600.
UL/DL rates on the SMs did not change after the reboot and the AP during this time was under very low load <10mbps average.
I did take a support file from the AP before reboot but forgot to take screenshots of the link tests before the reboot.
Here is the link test charts after the AP reboot. Before the reboot the same tests were around 1/3 to 1/2 less. The ~100-300Mbps was me running the mikrotik bandwidth test on one SM.
One thing I did just notice is that the SM UL rates were higher and more stable after the AP reboot. Not sure if that was affecting the DL bandwidth of the SMs.
Chart shows TX/RX rates from AP perspective. So TX is SM DL rates and RX is SM UL rates. Yellow vertical line is after AP reboot and SMs reconnecting.
We’ve noticed some strange issues with having MU-MIMO turned on the 4500 8x8 integrated. First off, when MU-MIMO is being used, beamforming gains are nullified, meaning that the 6dB (2-3 increase in modulation levels on downlink) is not available, this means that all the SM’s participating in MU-MIMO are possibly using a lower modulation level. Another consideration is that due to chipset limitations, only 3 groupings are available, despite it being an 8x8 radio. We ended up disabling MU-MIMO and going back to beamforming only and it seems to perform better over time. I think that the ePMP dev team still has room to optimize MU-MIMO and we’ll try again when there’s another major update.
@Eric_Ozrelic I do not believe this has anything to do with the 6db SM gain difference when doing beamforming vs MU-MIMO. Even if the SMs went from MCS11 to MCS8/9 I would still see around 200-250Mbps per SM when having 3 SMs push max DL bandwidth but that is not what I saw. Also the 6db gain loss would still be present after the AP reboot so the fact I saw double to triple aggregate AP bandwidth after the reboot doesn’t seem to line up with the beamforming vs MU-MIMO gain loss.
My guess is that the scheduler was having issues (especially UL) and the unstable UL rates which did coincide with each SM having higher than normal UL retrans rates was messing up the DL bandwidth and MU-MIMO.
Look at the charts for 3 SMs below. Each one started seeing high UL retans % at the same time and it went away at the same time, when I rebooted the AP (red line). After the reboot and the UL rates and retrans % improved back to normal and the DL rates and MU-MIMO were working normally again.
I do not know the exact correlation and impact the UL retans % can have on AP DL bandwidth and MU-MIMO and its exact cause. UL retransmissions are usually caused by interference at the AP side. As rebooting the AP fixed it I do not believe it was sudden interference from another AP or wifi device that also went away the exact time of the AP reboot, but instead self-interference from something getting out of wack on the TDD scheduler causing the SMs to interfere with each other at the AP side.
I am still seeing this issue. 4500 AP with 5 SMs. MU-MIMO is enabled. What I still observe is that after some amount of time it seems MU-MIMO is not able to push as much bandwidth as I believe it should.
As you can see earlier in the thread SM distribution is pretty good for MU-MIMO. I am testing using AP Multi-SM link tests.
Before AP reboot I did a link test to one SM. Link performed as expected with around 350+ DL on a 40mhz channel. I then did a 2 SM link test and while it showed 100% MU-MIMO traffic the AP bandwidth only increased around 50% to a max or around 425Mbps. See photo below.
I then rebooted the AP (grabbed support file first) and waited 10 minutes for the AP to reboot and SMs to reconnect and start normal traffic again. This time the same 2 SMs on the same link test yielded a 200Mbps increase over the same test before the reboot. See photo below.
Also forgot to mention on a 4 SM link test before the reboot it maxed around 612Mbps with around 80-90% of the traffic MU-MIMO. After the reboot the same 4 SM test maxed at around 712Mbps with the same MU-MIMO traffic % for a gain of 100Mbps.
Also like before I noticed that before the AP reboot that all SMs had higher UL retransmission % than after the AP reboot. This lower UL retrans % has continued for 4 hours since the AP reboot.
A simple AP reboot should not affect UL retrans % unless the retransmissions are due to an AP scheduling issue and not actual interference or signal issues.
SM UL retrans % are reported by cnMaestro was around 3-5% per SM. After reboot and still now ~12 hours later the % is between .5-1.5% so a drop of around 3-4% per SM.
I did not try Asymmetrical UL as the AP reboot fixes the issue. I have used Asym UL before on 4600 APs but that was due to high interference (>20% UL retrans rates) that was tanking the UL MCS rates and causing some SMs to disconnect.
This 4500AP does not exhibit those symptoms. Individual SM performance tests fine. Just the MU-MIMO is lower than normal before the AP reboot and the SMs have a lower UL retran % after the reboot along with MU-MIMO performing as expected.
Spectrum chart looks clear but it does show another WISP with a Cambium AP covering part of my channel. However, it might be synced so I am not sure if that would be interference or not.
Red box is my 40mhz AP. Yellow column is the other WISP AP.
I will also mention that so far I have not seen this situation on my 4600 AP that is running MU-MIMO w/25 SMs. Just the 4500 AP. Maybe a difference in the 8x8 vs 4x4?
It has been around 22 days and this same 4500 AP is showing lower than expected MU-MIMO performance in synthetic tests (Link test) while the UL Retrans % on all SMs went from around 1-2% to 3-5%. As you would also expect the UL MCS rates fluctuated more as well.
As before a reboot of just the AP fixes the UL Retrans % and the MU-MIMO performance via Link tests. Also the UL MCS rates were higher and more stable.
Before AP only reboot MU-MIMO tests (4 SMs) seemed to cap out at around 500-600Mbps. After AP reboot same 4 SMs tested around 750Mbps. Before/After single SM tests yielded around 380Mbps, but 2 SM link test was only around 500. After AP reboot single SM tests were still around 380 but 2 SM link test was up to 705 with each SM getting almost full single SM performance. A 3 SM test after AP reboot gave a very impressive 900Mbps down with each SM getting between 270-300Mbps.
Not sure what is causing this. UL Retrans % increase would seem to indicate AP side interference, but that should not go away with an AP reboot. I am also assuming the higher UL retrans % is cutting into the DL performance.
Here is one of the SMs Retransmission chart for the last 24hrs with the red line showing the AP reboot. Also included is the last 7 days chart showing the recent increase the last few days.
