A number of our sectors have been seeing interference from various operators moving channels during the great PAL shuffle. When this occurs we begin seeing session drops at a seemingly regular cadence (30m-1h) . The fix so far has been to move the sector to a new cleaner channel within the available spectrum. However having worked with other radio systems quite extensively it seems strange that we don’t see consistent demodulation due to interference rather just complete session drops. We’re looking to determine if this is a normal behavior of the PMP450 radio system or if we are missing some configuration parameter that would improve our equipment’s tolerance to these interference events.
Error seen on SM at time of session drop:
04/22/2021 : 10:13:20 UTC : 2090295: SetPiggyMode: 1 SM: using all of uplink and bypassing bandwidth request delay. Users:1
04/22/2021 : 10:13:23 UTC : 2090546: VcErrorInfoSM: VC Error VC:20 status:5(Frame Builder, uplink couldn’t get scheduled) (Unknown reason: 0)
04/22/2021 : 10:13:23 UTC : 2090546: VcErrorInfoSM: Session Uptime:00:57:16 Rx Power:-60.5 dBm ( -64.0 dBm V / -63.0 dBm H )
04/22/2021 : 10:13:23 UTC : 2090546: VcErrorInfoSM: Beacons:100% Maps:100%
04/22/2021 : 10:13:23 UTC : 2090546: VcErrorInfoSM: Setting scantype as normal due to VCError
We are currently running the latest firmware v20.2.1 (BETA-1) on all of our equipment currently in the network. However we’ve seen this behavior going back to 16.2.3, the issue has likely been made worse by increased congestion on remaining GAA channels after the PAL(s) went live.
@sdotson Thanks for posting. Several things:
First of all, I’d prefer not posting the Engineering Debug Log entries as they aren’t always helpful or exactly accurate and sometimes raise more questions than give answers. We won’t spend time explaining these logs.
Second of all, “VC Error” (Virtual Circuit Error) is a generic error that means the link could not get a packet through a Data Channel after multiple repeated attempts. This can be due to any number of reasons such as: changing RF conditions like interference as you mentioned, low Rx power, low Tx power, wind moving the antenna or blowing trees in the way, etc. Or it could be simply because the other side reset unexpectedly, or other side dropped the session first. Or it could be due to a bug in the firmware. Unfortunately, this is impossible to tell from a simple log which of the many possibilities it is.
What we do know is that there is a known issue with CBRS and 20.2 resulting in SMs unable to register or maintain session. Switching to Other/Other or downgrading to an earlier release addresses this specific error. We are investigating that specific error.
However, having said that, @Eric_Ozrelic has reported issues with dropped sessions on certain problematic SMs on the entire 20.X release line. These “problematic” SMs have their receive power fluctuating as their nLOS shots go through pine trees but they also often start with a good modulation, like 6X. We have investigated and discovered that we are not dropping the modulation fast enough to make it to lower modulations. With the introduction of Intermediate Modulation Modes (7X/5X/3X MIMO-B) one side could end up dropping the session as it could not pass data before the modulation rate adapted all the way down to 1X MIMO-A. We have made a change to the firmware and it will be included in 20.2.1 BETA-2 which should be available soon.
However, if you were experiencing session drops on earlier release line than 20.X (such as 16.2.3 as you mentioned), I do not think this is your issue. As in those releases the radio is able to modulate all the way down to 1X MIMO-A before session drop. Also if changing to Other/Other or downgrading to 20.1 release fixes the session drop it also is not this issue.
The only configuration item you have is locking the max modulation mode or locking the radio to MIMO-A only. Both will use more RF resources increasing your Frame Utilization and decreasing sector capacity, so I do not recommend changing them.
Charlie, what you described with the 20.x release and being able to adequately step down modulation does make a lot of sense for our situation. We previously did not see this behavior on 16.2.3 but moved to 20.x in order to support multigrant functionality on all of our AP(s) as we PAL(s) were assigned. We’ll stay posted for the 20.2.1 BETA-2 release as we’re currently in a stable configuration through shrinking channel sizes to allow for more diversified channel assignment.
Seeing the same here. Upgraded my home sector from 16.2.3.1 to 20.2, relinquished wide grant to get multi-grant. That all went fine. I’ve got a tree in the way and my signal is -55 in the winter, but now it’s anywhere from -67 to -78. Wind blows and it’s all over the place. It was hanging onto the session fine until the upgrade to 20.2. Will definitely give beta2 a try as soon as it’s available.
We have 2 CBRS SMs that randomly drop sessions, mostly on windy days. Both are <1 mile links through a tree or two. We normally don’t shoot through trees, but these are really short links with excellent signal and SNR numbers most of the time. It seems like the problem is when the branches move in the wind, and I have been suspecting it had something to do with the radio not being able to track the signal fast enough as it changed. Other SMs farther out on the same sector (but with no tree in the path) don’t have any problems.
This happened back on 16.2.2 and 16.2.3.1, we didn’t see any change when we moved to 20.2. Yes we see the intermediate modulations which is nice, but it didn’t seem to have any impact on this issue.
It isn’t occurring at regular intervals like sdotson describes though.
I loaded up 20.2.1 beta-2 on my sector about 45 minutes ago and my SM has held session so far. I was losing session right before that while on 20.2 official. Luckily maintained long enough to burn the new build. Beta2 seems good so far. Just a regular old 450 AP and SMs. A few on the sector are 450b’s though and they seem fine.
I had reported this issue to support during the beta of 20.x. I’m glad to see it’s getting resolved. I have held off migrating to 20.x until I know it’s stable and the CPE’s keep their session up.
We’ve been testing the beta on some 450i AP’s that have lots of nNLOS clients that were experiencing massive swings in modulation and thus dropping session because IMM couldn’t react fast enough. The beta has resolved these issues completely.
@Ken_Hohhof yes, this is exactly what 20.2.1 BETA-2 issue addresses: Starting and high modulation and not modulating down quick enough. If you’d load that and try it out and let me know how it goes I think it’s resolved and want to know ASAP if it’s not.
@ThomasNM, I encourage you to try 20.2.1 BETA-2 on at least sector as it addresses this issue specifically and it would give us time to fix any other issues found.
@Eric_Ozrelic, I appreciate your help in reporting, debugging and verifying the fix.
OK, I’m updating one of the sectors to BETA-2 now, we’ll see if it helps. But note this problem existed when we were on 16.x. It’s hard to say if it got worse with the change to 20.2 because it is somewhat random, definitely correlated to trees and windy days though.
We were having a lot of dropped registrations on 2 SMs on 2 different sectors so I changed both sectors to BETA2 yesterday evening.
So far so good, no drops on either SM, but it’s less than 24 hours and winds tend to be a daytime thing. Tomorrow is supposed to be windy. We’ll see.
It’s strange though because this problem isn’t new for us with 20.x FW, these 2 customers have had problems on windy days for a long time and back on 16.x FW. Maybe it got worse with the intermediate modulations, but that’s not when it started.
On a side note, dropped registrations can be amplified with CBRS if the SM misses a heartbeat and loses its grant, plus there are so many frequencies to scan now. Once a CBRS SM drops, the outage can easily stretch to around 5 minutes.
Windy today and the 2 problem SMs have stayed registered, so it looks like BETA2 solved our problem.
Which brings up other questions, like was there a tradeoff or downside to this change? Anything we should be looking out for? And any timeline for an official release?
And I’m thinking of this as specific to 3 GHz and CBRS/multigrant specific. But wouldn’t this apply to 5 GHz as well? Could we be experiencing this problem there also and just don’t know it? Maybe not due solely to trees but also things like wind turbine blades passing through the path? I don’t want to put beta FW on a bunch of production sectors for no good reason, but once there is a production release incorporating this change, should we be rolling it out everywhere on an expedited basis?
The problem has to do with the radios not being able to shift modulation steps or make large leaps in modulation reacting to large swings in RSSI or SnR. This might have been a problem prior to R20, but not as readily apparent because there were less modulation steps (less of a difficultly making larger jumps) and there are now more people that are using PMP450’s on 3GHz for nNLOS shots (like me). The problem was most likely amplified when the intermedia modulation steps (IMM) was introduced in R20. These addition steps (7x,5x,3x) again, amplified this problem. I don’t think most operators are trying to install their 450 5GHz radios in nNLOS, so no on using this band reported issues that I’m aware of.
This fix covers all platforms and all bands.
I worked on this issue with @Charlie perhaps he can provide some additional detail.
Exactly correct, the problem is not introduced in 20.X, but made worse with more modulation levels.
The tradeoff could possibly be running in lower modulation more often then before the change in quickly challenging modulation environments (wind+trees, interference, etc.) This change down shifts faster.
We are working out the issue with CBRS Multigrant and then will do an official release when those are settled out. Sorry, I’m not sure when that will be yet, as it depends on how long those issues take. We are trying to resolve ASAP.
Yes, this change applies all frequency bands. I don’t expect wind turbines to affect this the same way, as I’d expect these to be more constant? Just guessing there. If you’re having dropped sessions with SMs that generally have high modulations, this firmware would be good to load on those 450/450i/MicroPoP sectors.
We agreed this is a good change to make on 450m as well and is included in 20.2.1 BETA-3.
Thank you @Charlie this really helped us a lot. We upgraded to v20.3.1.1, from v 20.2.2.1, not only did it solved SMs that were dropping with good Modulation. but it also solved the uplink issue on our 900 SMs.
Before, some of our 900 APs were seeing 1x uplink on all SMs but had good downlink. We audited our frequencies and sent our riggers to check and repoint antenna but no luck. After updating it to v20.3.1.1 the uplink issue was resolved.
However, with v20.3.1.1 we can’t push our golden config via cnMaestro. it showed internal error in cnMaestro. but downgrading the SM to later version (v16.x.x or v 20.2) would work.
Please start a new thread for config file changes. For debugging, I’d suggest running the config file on a radio via Configuration → Unit Settings and checking the output there as it should give feedback as to what failed.
