Sudden network Slowdown - 9.4.2 firmware? Something else?

On one of my neworks i suddely have almost network wide speed complaints. We have our own local on site speed test so we know it is not our uplink. I get 75meg speeds across our backbone from tower sites so it’s ONLY Canopy Customers having this issue. Speeds have suddenly dropped down to 25% of what they once were. RF tests at 100% up and down, no noise issues. No lost packets anywhere. Latency is normal. All customers are on either 5.2 or 5.7 and all APs are Advantage. The only recent change was the 9.4.2 firmware but it’s hard to tell when these things start as complaints come in at later dates etc.

I have all APs set to: 12 miles, 80% down, 3 control slots

I have all APs QOS set to: sustained up = 8000, burst 500,000 up, sustained down 32000, burst 500,000 and broadcast downlink cir to 200

Customers are almost all set to 2500 up and 4500 down with no burst and no high priorty data.

Some APS have as few as 50 and as many as 135 customers on them. I track everything using PRTG and a QOS box that shows me protocols, packets, usage in every way one can imagine and these aps are stopping at around 5 to 6 meg thrughput where they used to hit 10 meg and stll perform better. Its like they all became non advantage APs overnight. I get speed complaints from customers when the ap only has 4 meg traffic when i would get none at 8 meg or even 10 meg. When i put just one customer who gets slow speeds on the same ap by changing color codes they get over 10 meg down and 2 meg up without issue on same ap.

Anyone else having these same issues or ever had them?

Hi,
just a simple question: are your customers connecting in 2X mode?
you can see that on the status page:
e.g. REGISTERED VC 54 Rate 2X/1X

BigTrumpet wrote:
Hi,
just a simple question: are your customers connecting in 2X mode?
you can see that on the status page:
e.g. REGISTERED VC 54 Rate 2X/1X



Some are and some are not. Every AP is different. It depends on signal levels, line of site, miles from tower ec. Probably only about 25% on average are 2x.

What kind of RF outdiscards are you seeing on the AP’s? IMO 100+ SM’s on a single Adv AP is awfully crowded when they all have a QOS of “2500 up and 4500 down with no burst”. Are you filtering P2P? You may want to consider adding an additional sector(s) to balance the load.

jwelch wrote:
What kind of RF outdiscards are you seeing on the AP's? IMO 100+ SM's on a single Adv AP is awfully crowded when they all have a QOS of "2500 up and 4500 down with no burst". Are you filtering P2P? You may want to consider adding an additional sector(s) to balance the load.


Radio has been up 3.5 hours and it has 27 total discards on hte RF outdiscards.

note that almost every ap on the network has started to have the same problem at about the same time. While i would like to claim that each AP got 50 new customers this is not the case. This is a sudden downgrade of performance for ?? reason. That's why I was looking at the firmware as a possibility.

And note that these APs will provide VERY low speeds even when they are only passing 2 meg of data. if i kick off all the customers except one having speed issues it will jump to 10 meg plus down and over 2 meg up.

so with 60 to 115 customers on an AP even if there is only 2 meg total traffic = 1 meg down = total of 3 meg ...sometimes total of as high as 6.x
with 1 customer = 10 plus meg down and 2 meg up.

If you have a hunch that 9.4.2 is the source of the problem then use CNUT to downgrade one AP to a previous release.

ex. 9.4.2->9.4->9.3->9.0->8.2.7

Are all SM’s running 9.4 or above?

jwelch wrote:
If you have a hunch that 9.4.2 is the source of the problem then use CNUT to downgrade one AP to a previous release.

ex. 9.4.2->9.4->9.3->9.0->8.2.7

Are all SM's running 9.4 or above?




Yes all SMs are 9.4 or above.

how did you push out your update? did you follow the upgrade path or do the direct jump to the new firmware thats supposed to work now?

thatoneguysteve wrote:
how did you push out your update? did you follow the upgrade path or do the direct jump to the new firmware thats supposed to work now?



i had some towers running old firmware and even tried going down to 8.27 and it seemed to make no difference.

And when has moto EVER had firmware that works? I have always just gone for the lesser evil version of firmware!

i have been thinking about upgrading to 9.4.2 but i am not so sure now.

anyone else having issues with the new update?

Not sure if I read this properly, but have you downgraded back to 9.0 or something?

the_NAT wrote:
i have been thinking about upgrading to 9.4.2 but i am not so sure now.


Was hanging back until Moto fixed the whole lack of RF Rate reporting in Prizm, now it seems there are more reasons :open_mouth:

what are the AP’s connected to?

justfortimd wrote:
[quote="BigTrumpet":azax9kvv]Hi,
just a simple question: are your customers connecting in 2X mode?
you can see that on the status page:
e.g. REGISTERED VC 54 Rate 2X/1X



Some are and some are not. Every AP is different. It depends on signal levels, line of site, miles from tower ec. Probably only about 25% on average are 2x.[/quote:azax9kvv]

I think it's not so strange that you don't achieve 10 Mbps down if just 25% is 2X.
In order to achieve 14 Mbps aggregate you should have 100% running 2X mode.
I think maximum down speed in your case cannot be roughly > 8.5 Mbps (7 mbps * 75% + 14 mbps * 25%) * 80% depending on usage balancing between 1X and 2X.