Uplink Frame Utilization cap since 16.1.1 now using 16.1.1.1

Hello guys,

I am looking for ideas or anwers for what "appears" to have happened over night when I pushed 16.1.1 across the network, I did see where 4 co-located 450i ap's in 5 and 3 ghz and others on other sites all started to plateau on the upload utilization in cnmaestro (Version 2.2.1-r32),  from what I have seen in the ap's under frame utilization they are hitting 90%+ for long periods of time so I know cnmaestro is reporting mostly correctly. I have applied 16.1.1.1 and the problem has not changed, this seems to have affected at least 10 ap's that previously did not have a problem and for the most part have 64qam modulation rates up and down. as of right now example ap is 100% uplink with 4~5 mbps traffic. 

I appreciate any input.

Andrew

If you actually login to the AP's when this is happening and look at the Statistics -> Frame Utilization tab, is it showing high upload utilization that matches cnMaestro? If you perform a link test on a client during periods of high uplink frame utilization, are you seeing really low upload numbers? On the AP's link status page, are your SM's upload modulation healthy? Lastly, are clients reporting upload issues?

Yes under AP stats uplink utilization is high 98% as of now, and stays that way almost anytime I look. So I feel what I see on cnmaestro is accurate.

Some complaints from customer's of sluggish connections

Link test to sm 1 now is 22.94 Mbps uplink at 24% 64qam V and H

Link test to sm 2 now is 18.84 Mbps uplink at 34% 64qam V and H

Link test to sm 3 now is 23.15 Mbps uplink at 33% 64qam V and H

AP uplink is at 100% with 1 minute interval with 3~4 Mbps

The worst of link status.

Hmmm... this is really strange. Everything seems to indicate that your AP should be able to sustain much higher upload rates before hitting max frame utilization. I'm still running 16.1 and have not noticed this issue... so I can't compare it to the two versions you've tried. I'm out of ideas on what more to troubleshoot. You should probably open up a support ticket while you're waiting for someone to from Cambium to respond here.

Thanks for your input Eric.

I decided to downgrade to 16.1 since there was no problem there and if you look at the 8:30-9 time on uplink utilization you can see the drop.

There is some issue from 16.1.1 to 16.1.1.1 that persists in our network.

Side note I should have mentioned is the problem found in 16.1.1 that was fixed in 16.1.1.1

CPY-16458 : SM does not honor QoS Configuration in NAT mode

We do not normally NAT the sm. and we seen a big impact in uplink utilization but not downlink. but not on all AP's.

We are seeing the exact same issue with 16.1.1.1.

Uplink maxed at 100% on all APs, when they never used to max out.

We also enabled auto contention slots and thought maybe  that had something to do with it.

Disabled it on a few APs and the uplink was still pinned at 100%.

Seems to be a firmware issue.

Will be putting in a support ticket.

Should also add that downgrading to 16.1 fixes the issue.

We're seeing this issue also.

I'm going to pull engineering.cgi and downgrade to 16.0.1.1 tonight since we haven't deployed auto-contention yet.

Thanks Guys, Happy it's not just us.

I moved back to 16.1 we did'nt have any problems there.

All,

We've confirmed the issue. It is purely a reporting issue that affects 450 and 450i AP's but not does not affect 450m. There are several uplink Frame Utilization statistics that can report two to four times the correct value. It affects 16.1.1, 16.1.1.1 and 16.2 releases.

We have targeting this fix to 16.2.1 and we are working to get a 16.2.1 beta release out with this fix included. Our internal tracker number CPY-16525.

At this time, even with this bug, I would still recommend 16.1.1.1 over 16.1.

We are seeing this affect a 450m AP aswell. It is saying that it is only able to push 100mb at 100% utilzation. But we can watch it in the switch that it is plugged in to and see that it is easly pushing 200mb +. I have an open ticket with support about it as well. For the 450m that we are seeing the issue on.

Do you have an ETA on release on 16.2.1 that fixes this?

I have an open ticket also. The APs 450, 450i and 450m perform much better running 16.1. Down side is I need 16.2 or higher for CBRS.  I would live to send the phone calls to Cambium for the customers that are suffering.

As stated in this thread, this high utilization rate is purely a reporting issue.

If there are actual issues as a result of this, it's most likely due to something else.

We are working toward a 16.2.1 release shortly that has this corrected, but it should not be affecting actual operation, only the reporting.

We use the graphing of frame utilization to determine what APs need upgraded, frequency, changed etc. etc.  Not having frame utilization functioning correctly throws a wrench in this.

Does this bug only affect upstream?

Yes, it is only upstream stats.

Totally understand that operators rely on these statistics. This will be fixed in an upcoming release.


@hci wrote:

Do you have an ETA on release on 16.2.1 that fixes this?


This fix will be included in 16.2.1 BETA-2. I hope we will have that load ready for open beta within the next week. We do not have an ETA for the 16.2.1 official release yet.

Will 16.2.1 BETA2 be available on cnMaestro Cloud?


@hci wrote:

Will 16.2.1 BETA2 be available on cnMaestro Cloud?


Yes, we make our all open beta builds available on cnMaestro cloud. Sometimes it does takes an extra day to get there.