ePMP 1000 4.5 100% CPU/Frame Utilization

Recently we have noticed a high level of frame utilization coupled with high levels of CPU usage near peak hours. There are several flat tops on the graphs indicating sustained periods of redlined performance. We have about 30 customers on this sector all utilizing a 15x5 QOS.

Can somebody help me understand what may be happening here, how to solve it, and whether or not ePMP 2000 may yield better performance?

The quality of the customer/SM/client connections will greatly influence cpu and frame utilization.

For example , a customer with a 15x5 and a great connection to the AP will only use only a fraction of AP resources compared to a customer with 15x5 and marginal or poor connection.

If we assume you are running out of frame time when the AP is only delivering 35 or so Mbps due to less than ideal customer connections the only way I see the 2000 helping is that, at least in FCC land it may not be the case wherever you are, the 2000 is allowed to use more tx power in some frequencies than the 1000 is, so that could improve the MCS rates on some/many/all of your subscribers and thus improve the amount of data your AP can deliver to them.

Otherwise I believe only real difference between the 2000 and 1000 is (1) the 2000 has a more powerful processor ? This might help a little but not much I don’t think. (2) The 2000 can use the beam forming/smart antenna. This only helps with uploading from the customer to the AP so could help a little if your customers upload MCS rates / speeds need a little boost. (3) The 2000 has edge filters that allow it deal with non-synced sources of noise at the edges of the channel they are using. Again, only useful for customer upload MCS rates / speeds but could still help you a little.

Otherwise, and again assuming customer radio connections are the issue, the only thing that will help a lot would be improving the customer connections with more powerful antennas (upgrade a F180 to a F200 for more gain at the customer end) , re-aiming or eliminating interference issues (RF and physical obstructions) at customer end, removing any customers from the AP that have really bad/low connections/MCS levels that can not be improved.

Even though ePMP will not allow a single poor customer connection to drag down an entire AP customers with poor connections still affect the overall capacity of the AP.

Something else that may help if you have customers who’s radios jump around between MCS rates a lot is forcing those customer radios to lower MCS rates. It can help with the overall performance if there aren’t a bunch of SM’s constantly re-negotiating modulation.

As far as evaluating why you are running out of frame time when the AP is only delivering about 35Mbps :

If you look at Monitor > Performance > Downlink Packets Per MCS the more data you have being sent at lower MCS rates the more CPU and Frame time it is taking to send/receive that data. If you look above that data on the same page at the Subscriber Module Statistics and the up/down “Packet Drops” you can see which SMs are having to have data resent. That will allow you to see which client radios are using the most resources and driving up the CPU/Frame Utilizatoin. The “Capacity Packet Drops” will tell you how many customers are not getting the bandwidth they are supposed to be getting because the AP is maxed out.

10 customers with 15Mbps rates and perfect connections running 100% at MCS 15 and all 10 trying download all at the same time would pull around 150Mbps combined from the AP and the AP would pull 150Mbps from its Internet connection. However if those those same 10 customers had not so great connections and most of their transfers were at , say , MCS 7 or 6 or so and bouncing around between MCS rates then the CPU / Frames will still be maxed out and the AP will use all it’s 150Mbps / frame time delivering 35Mbps of usable data from the Internet to the customer radios.

So in this scenario the perfect connections is 150Mbps of Internet data goes in the AP and 150Mbps data is delivered to the Customers. The not so good connections is 35Mbps of Internet data goes into the AP and it burns 150Mbps of air time delivering that 35Mbps to the customers so you see 35Mbps on your throughput and 100% utilization of your Frame Time.

2 Likes

50% of the time, if I factory reset an SM the CPU reverts back to being normal around 3-5%. Otherwise I have to replace the SM with a new one, which I did the other day and this particular SM was behaving for over a week. Then we had a power issue in the server room and all the SMs on our network lost connection, when we got them back online, lots of SMs were maxing out the SMs CPU. It was evident from the ‘week’ time period of performance monitoring that something had changed to trigger the SMs to misbehave, which would suggest that signal strength had nothing to do with what was happening. Also, the SMs I replace usually have the CPU max out, and they disconnect. Generally we use the Force 300-25 or 300-16 running 4.6.1. Also rebooting them remotely has done nothing.