Unfortunately we still have a few 430 SMs out there (on 450i APs). I have discovered several that keep ending up at 3X/4X MIMO-B according to the downlink status at the AP. When this happens the SM reports low SNR and the link works badly. Customers report problems like rebuffering.
Shouldn't this be impossible? 430 SM should only be 1X, 2X or 3X MIMO-A.
Is this a known issue with some AP or SM firmware version or something? We still have the 430 SMs on 13.4.1.
Are the SM's really close to the AP with generally really good RSSI? On the AP side, does it show a big chain SnR mismatch when it's operating at MIMO-B? I don't have any 430 SM's, but I'm guessing that the AP is hearing the one 430 stream on both of it's chains and that's what is causing this to happen. I remember this happening with Ubiquiti gear when using 1x1 clients with a 2x2 Rocket AP.
Not close to AP, these are 5-6 miles from the tower.
The AP will report 4X MIMO-B downlink, but the SM doesn't have a concept of MIMO-B or VPOL and HPOL. It just reports one SNR number and it will be something like 12. If we kill the session, then it comes up at 3X/3X MIMO-A and 35 dB SNR. Until it does the screwy thing again.
And of course these radios don't actually measure SNR or MSE, the infer it from the error rate. The actual link performance seems worse than you would even expect with a 12 dB SNR. The first thing to tip me off when one of these links gets screwed up again is the "latency" reported in Preseem goes sky high. I'm not sure Preseem latency exactly corresponds to what we think of as latency, but when it's up at 200 ms (the max it will report), something's wrong. I think they try to report TCP latency by comparing the packets and when they get ACK'd.
Weird, among a bunch of MIMO-A and MIMO-B I found one in the listing at the AP that said SISO. Reset the session and it went to MIMO-A.
This can't be right.
The only change we've made, which was last year, we cleared out all the 430 SMs on 2 of the sectors and replaced the APs with 450m. I can't think why that should matter, the weird stuff is on the 450i sectors that still have some 430 SMs.
So I opened a ticket with Cambium support and of course they want me to bring the APs and SMs to latest FW, or at least same FW. To be honest, I thought 13.4.1 was the end of the line for 430 SMs. I'm trying to upgrade one now to 15.2 and getting an invalid image error.
Still it seems pretty basic, the 430 SM doesn't support MIMO-B or 4X, the AP should know that. And I can't find anything in any of the release notes about this being a bug that was fixed.
The last sofware for the PMP 100/400/430 platforms was indeed 13.4.1. However, when you're operating in "compatibility mode", (i.e. 430 SM connected to 450 platform AP), then you can update software beyond that.
All of this said, the absolute best thing for your network is to ditch the old hardware. I know that's probably not what you want to hear, but your network performance will improve once you're on the more capable hardware. If you can replace those SMs, I would recommend going with 450b (rather than the original 450 SM).
Regarding the particular issue you're talking about, the 450 system does indeed know that the 430 is SISO, but need to make sure the correct software is running.
We absolutely intend to replace the remaining 430 SMs with 450. We are down to maybe 40 of them that we didn't get done before last winter. We've completely cleared out the 430 SMs on sectors that we ugraded last year from 450i to 450m APs.
It's easier with the discontinued C054045C003B that we can just pop into the existing dish. I think I've got one box of 20 left. Bare SMs and SMs with CLIPs we'll use a 450b mid-gain. We have to be judicious about where we use the 450b high gains, as the physical mounting is different from a reflector dish, also they don't always perform as expected when mounted on a J-leg on a roof. Cambium actually had an app note about this basically saying put the SM higher. Easier said than done, J-leg on roof is the most common mounting method, and higher means a tripod. Because of this we've eaten into our stock of the old SMs doing new installs because we're gun shy about the 450b high gain acting like there are reflections. Waiting to see if a 450b retro version materializes.
Anyway, per suggestion from Cambium support, we found that we could upgrade the FW on the 430 SMs as long as we went to 4.1.2 first. We were able to go 13.4.1 --> 14.1.2 --> 15.2.
Now we just need to wait a day or two and see if any of them go back to MIMO-B downlink. Not sure why we occasionally see SISO instead of MIMO-A, but MIMO-B should be invalid when transmitting to a SISO SM.
I'm hoping this was a firmware glitch and that having the AP and SM on the same FW version will alleviate the problems for a couple months. Once winter is over I can send techs out to replace the last of the 430 SMs. Yippy, more e-waste.
Ken, Correct, the PMP 430 SMs need to go to 14.1.2 then can be upgraded from there.
Note, PMP 430 support continues to 16.0.1.1. I recommend that release over 15.2.
If you have your whole sector on 16.0.1.1 and are still seeing this issue, please let me know.
I guess I will have to try that. 15.2 AP and SM did not fix the problem. Also I tried an experiment on a sector with just a few SMs changing the 430 Interop setting from MIMO-A to SISO. I expected to see all the 430 SMs to say SISO in the Sessions tab, but actually they still said MIMO-A or MIMO-B. And now several SMs on this sector, which had never exhibited the problem before, had the MIMO-B problem. It's as if 15.2 FW on the AP doesn't understand that you can't transmit to a SISO SM using MIMO-B.
Also the performance problem typically manifests itself in a downstream linktest of ~10 Mbps, also the SM reporting SNR of 12 dB. It seems like maybe the fragments are being retransmitted at successively lower modulation until finally the AP tries 1X MIMO-A and the fragment gets through.
Most of the SMs on these sectors are 450 and the changes so far have not disrupted their service. Taking the AP and all the 450 SMs to 16.0.0.1 will be a lot more disruptive, and of course there is no time anymore to do firmware updates without pissing off customers. Maybe also I'm more willing to mess with the 430 SMs because they are all on the lowest speed plans or they would have already been upgraded to a 450 SM. Which is to say these are typically people who are sleeping at night, not gaming or streaming.
Took an AP and all its SMs to 16.0.1.1 and a day later had a 430 SM go to 3X/6X MIMO-B.
I am going to assume this is a firmware bug. AP transmitting downstream to a 430 SM using MIMO-B should be invalid. And if it is a 430 specific firmware bug, it's not going to be fixed, since 16.0.1.1 was the end of the line for 430 backward compatibility.
So I'm going to have to manually monitor for these sessions and reset them until we can change out the 35 or so remaining 430 SMs for 450 and 450b. We had prioritized customers on higher speed plans and on 450m APs and had postponed the remaining upgrades until spring. Obviously any truck roll to a 430 customer e.g. for a repair we had also replaced those SMs.
Ken,
Thanks for loading 16.0.1.1. Would you mind sending me the engineering.cgi from the AP and a few SMs both 430 and 450 to charlie.galik@cambiumnetworks.com?
I've confirmed with Ken on email there's an issue there.
The work around is to monitor the downlink rate adapt and if it goes into MIMO-B, manually drop the session.
If anyone is interested in a test load for an AP that will detect this and drop the session, please email me.