Has anyone seen this? Trying to get some reassurance before replacing the AP.
I assumed it was the usual CBRS interference from T-Mobile but it doesn’t follow the usual pattern.
When it happens, it affects all SMs on the sector and both upstream and downstream, usually it’s mostly upstream or just a few SMs. Also no interference on SA, changing freq doesn’t help, and 5ms frame with LTE timing doesn’t help. Sometimes all SMs will go to 1X MIMO-A or even deregister. Oddly, a 5 second power cycle via the Rackinjector will clear up the problem for a few hours.
Sounds almost like it may be similar to what we have seen with a few APs recently. Are there any temperature or humidity changes that correlate with the rate drops? Does the reported RSSI imbalance go up for SMs at the same time?
Maybe. There are some similarities and some differences. Most of the time the SMs are working, also this is not a new AP, I think we moved it from a tower where we lost our lease in 2020.
I can’t find a correlation to any temperature or humidity changes. I believe yes I have noticed the V/H ratio gets worse when the problem happens, but not a dramatic change, maybe from around 1 dB to 4 or 5 dB. And I think it might have only been in one direction.
The only thing I can think of other than an AP going bad is that the POE cable is pretty close to 328 feet, and we are powering it from a 48V power supply not 56V. But we have a 5 GHz 450m at the site powered the same way and it is happy.
OK, this is weird. I just had another AP start doing pretty much the same thing. Different tower, around 40 miles away, not even the same P/N (this one is a connectorized AP). All SMs download stuck at 1X.
SA doesn’t show any interference. After running SA from the AP the problem was still there. Played with AP xmt pwr and SM target RSSI with no success. Finally tried changing AP freq from 3620x20 to 3610x20 and now all 10 SMs are at 7X or 8X. My suspicion is it won’t last.
Jacob, what FW are you using? We’re on 22.1.2 for all our CBRS APs. I think this was the one that added the extended TX expiry time. Is it possible there’s some kind of bug where the AP gets screwed up if it doesn’t talk to the SAS every 5 minutes, and forcing a grant renewal somehow fixes it? That seems like grasping at straws, but we’re kind of at that point with this problem.
I was assuming I had an AP that was going bad after 5 years. Maybe something like the chip that switches the antenna between TX and RX was going bad.
But if it’s happening across multiple APs, and mine are old and yours are new, maybe it’s some kind of firmware bug? And is the SNR actually bad, or does the modulation rate algorithm just think it is? What happens is kind of reminiscent of going from 16.x to 20.x where the links would be stuck at 1X if AP was upgraded and SM was still on old FW. (That wasn’t a bug, they told us that would happen.)
Unfortunately Cambium support was of no help at all. They went through their scripts and told me there must be noise or some other environmental factor, which there wasn’t and isn’t.
The only environmental condition I can think of is we are having more than the usual multipath issues over the flat cropland this year. In general, not specifically on these APs. So theoretically, ducting might be causing interference. I don’t really like that explanation since all SMs are affected, and both DL and UL.
Trying to give Cambium support the benefit of the doubt, what if a cell carrier is using all 150 MHz of CBRS but only to send beacons, no significant amount of customer data. Could that cause a tiny burst of errors that messes up the Cambium modulation algorithm, but still the SA is clean?
Usually when we get cellular interference, it affects mostly UL, and we can mitigate it by going to a 5 ms frame and trying to get the LTE timing parameters right. I don’t have a good way other than trial and error to match cell carrier timing, we could have a new one such that our previous numbers aren’t working. Still, why are all the SMs affected, both UL and DL?
I remember several years ago, maybe around 15.x FW, Cambium had the rate adaption algorithm set too slow and SMs would rereg if, for example, some tree branches were blowing in the wind despite a really strong RSSI. And they sped it up. I wonder if they sped it up too much?
We can set max modulation, and MIMO-A/B or MIMO-B only, but not a minimum modulation. I guess that makes sense, but it would be an interesting experiment in this situation.
Oh, and 450 radios don’t actually measure SNR from vector error, they extrapolate from BER. So anything that causes bit errors could lead to a modulation change.
I run all of my 365 with the 5ms, LTE Frame 2, S Frame 7, 80% Download and usually have little issues except when a cell tower is in the path or right at my customer. I have also seen rogue operators causing me interference that shows up on both sides of the link though they are usually only in the upper band (3650 - 3700). I have to go out with my spectrum analyzer and a parabolic to find those operators. Usually a call has them turning off.
Your 5 second reboot sounds like maybe there is some GPS timing drift? Have you discovered the cause yet. I personally have never seen a radio fail in the way you are describing (I operate on all versions of 22.x and 24).