ePMP PtMP performance - speedtest woes

We have a fair number of 5GHz ePMP deployments, but we've seen an alarming trend for those on the higher ends of our speed plans - specifically, they often have crappy speedtest performance.

The first time I directly tested this was with a customer on a 10 Mbps plan. The AP showed he was on the correct plan as per the AAA server, and the Wireless Link Test showed the radi was fully capable of handling the speed. The RF quality was very good, at MCS 14 or better.On that trip, I had no issues with Speedtest websites or iPerf if I connected directly to the power injector, but I did see a big drop if I connected through the router. After a bit of brainstorming, I turned off the Auto MTU setting on his router and set it to 1480. After that, going through the router caused no performance hit to Speedtest or iPerf.

For the next year or so, I recommended that solution to subscribers whose Speedtest results over ePMP were sub-expectation. Yesterday I went to a site with a 30 Mbps down plan, because simply changing the MTU on the router to 1480 wasn't doing the trick. The setting I had to revise to fix the problem was not in the router this time, but it was the MTU setting in the STA - changing it to 1564 (and leaving the router's setting on auto). After that change, I was able to connect through the customer's router and get the expected levels of performance.

Am I alone in this? Have others using ePMP also found that the default settings for MTU induce packet fragmentation which kills Speedtest scores? I know that real-world performance, over a variety of packet sizes, would probably not exhibit these specific symptoms, but let's be honest: Internet Subscribers love running Speedtests, and all but the most savvy of them believe their results are gospel.

1 Like

I have forwarded your post to the product team, and they should have information for you soon.

I would like to see this information also as we are seeing this at varying degrees on a lot of  customer installs.

I don't know what to tell the customers when they ask me why their speed is so crappy through their router and want to know how to fix it.  I can't tell them "well buy a new/better router" because people with new/better routers are having the exact same problem.

Hi,

Path MTU issue is well known issue for IP networks.

Ethernet default MTU value is 1500 bytes.

The same value has ePMP devices in Ethernet driver. It can be increased up to 1700 bytes.

ePMP Wireless driver has fixed MTU size in 1708 bytes.

To get maximum trhroughput between two IP servers/hosts each network device should have same MTU size configured along the path.

Fragmentation occurs when a router forwards IP packet onto an interface with a smaller MTU than the source interface. Fragmentation of IP packets as intermediate routers can affect throughput.

That decreases the utilization efficiency of the path to the destination node. IP fragmentation also increases the processing burden on the intermadiate router that performs the fragmentation and on the destination node that reassembles the fragments. This can lead to degradation of performance be increasing CPU and memory consuption.

There is specific method which provide ability to calculate optimal MTU size for your network.

It is called Path MTU discovery.

All public shared information regarding Speedtest measurement you can find here:

https://support.speedtest.net/hc/en-us/articles/203845400-How-does-the-test-itself-work-How-is-the-result-calculated-

You can see that it is not clear which host is selected, how many hops exist between speedtest host and tested one, packet sizes, etc. After that it is not clear how to start troubleshooting with this.

And you can see there is any information about Ookla recomendations regarding MTU size for hosts.

Now I am trying to contact Ookla support to discuss this ascpect.

Yes, speedtest is very popular application for customers, but we cannot recommend to use it for throughput measurement on ePMP links.

We recommend to use iperf, where you can see and change tests parameters and environment.

I will keep you updated after Ookla support will provide more information regarding their measurement methodology.

Thank you.

1 Like

Not really seeing this problem with our FTTH , Ubiquiti or 450i customers, just ePMP customers. Also it's all speed tests, not just Ookla.

"Yes, speedtest is very popular application for customers, but we cannot recommend to use it for throughput measurement on ePMP links"

You can not recomend it all you want but the customers will continue to use them and they will continue to call us to complain and they will continue find another ISP if we can't make their speed tests look better. When my competitor  plugs the customer's wifi router into a Mimosa radio and speedtest.net says their getting 90Mbps and speedtest.net was only saying 12Mbps through their router connected to my ePMP then no one but you cares what iPerf would have shown.

1 Like

Fedor, I've seen these same issues with iPerf, as well. Again, I think this is more a speed testing over ePMP issue than one of actual performance problems with real-world traffic, but the fact is that my customers seriously overuse speedtest.net, and I don't know of a way to get them to stop.

They use speedtest.net as some sort of game, always trying to get a high score, and as soon as a video stream reports buffering (even though other members of the family are uploading their photo library to the cloud and downloading an 15 GB Xbox One game), they hit "Begin Test" on speedtest.net, and immediately fire off a nasty letter to our support desk (and maybe copy Yelp and the BBB) when it doesn't match the results they got when no one was using the connection.

Has Cambium used this sort of model during the in-house performance testing? Have your tests shown any issues when computers and tablets are connected through typical, consumer-grade routers?

Here are two things we do.  They aren't really a solution to lower than realworld SpeedTest.net exactly, BUT they do help.

1) One of the things we did was to have our own speedtest written which saves the results into a database locally. We encourage people to use not only SpeedTest.net, but also to use Fast.com and SpeakEasy and TestMy.net and BandwidthPlacecom as well as our SpeedTest program.  We tell them that only our local SpeedTest script will be the only one which accurately tests their connection and our network, and which doesn't put ''the internet'' between us and some remote testing server 1/2 way around the world into the mix.

Then, when someone reports ''even your speedtest only showed 3.6 Mbit last night!!!'' we can go back and see by their IP how many times they've tested and what results were shown on their screen for each test.  Sometimes that's 1 test per week for some cleints, but sometimes that's 20 tests per day.  And since they don't know we store the results in a database, if that 20 times per day tester chooses to report his one 3.6 Mbit result, and fails to mention the 10 Mbit results that he got 60 seconds before and after... then we know he's a hypochondriac.  It's nice to have the data to say...  "OK, I see you got a 3.6 Mbit results at 8:06 PM...  I also see you got 10.0 Mbit at 8:05 PM and 8:07 PM and 8:11 PM and 8:14 PM....".   

2) The other thing we use is our ePMP Chrome Extension - which shows real time Throughput on the AP for them and for all connected clients.  That way, we can see what they are using right while they run a test ( and see what else in their house is TX/RXing data at the same time) and we can also see what else the AP and the rest of the clients are doing.  Of course, ePMP does a really really good job of sharing airtime fairly, but at the end of the day, there is only so much to go around.  So, maybe he really IS having a performance issue caused by packet retry or caused by other users doing more than expeted at that same time too.

http://community.cambiumnetworks.com/t5/ePMP-2000-and-1000/Real-Time-Throughput-amp-Packet-info-for-ePMP/m-p/53178#U53178

So - neither of those two things really solves a situation where SpeedTest.net's results are inaccurately lower than they really should be.  BUT, it's something to tell the the clients to show that you do know what's going on on your network.

3 Likes

back to MTU and the default size of your MTU packet.    some services out there don't discover the full path MTU.  i've learned this the hardware and adding that extra work to your routers, and their router can cause you headaches. 

if you've got vlans in use, you need to expand your MTU by 28 bytes.   if you've got other services involved that may effect your headers, you'll want to furter increase your MTU.   

the routers on both end of the link is where the MTU matters, the devices in the middle, only need to be large enough.    ping your clients devices, or from your clients devices up with your desired MTU size for the client (for us its 1500 regular ethernet) 

and use the dont fragment flag.    if you can't get that ping through, start adjusting your MTU.    I had a problem with UBNT ERLs cutting our bandwidth in half to some sites, it ended up being related to the MTU interactions.     changing cambiums MTU, and the routers MTU from 1500 to 1528 cleared everything up for us.   

packet fragmentation can give you tons of heads.      also, if you can become a speedtest.net host.   its pretty handy ;)  best marketing peice i've done in a long time. 

3 Likes

We are already a speedtest host, and for our PMP450 subscribers, it works beautifully. We really just see this issue with ePMP subscribers, even when they run on our speedtest server (confirmed by watching in Torch on the Mikrotik or the Speedtest logs). As I mentioned, even iPerf displays these symptoms

My "MTU-embiggening" strategy does work, so it's not an insurmountable issue, and most subscribers never see (well, never report) any problem.


@dshea wrote:

We are already a speedtest host, and for our PMP450 subscribers, it works beautifully. We really just see this issue with ePMP subscribers, even when they run on our speedtest server (confirmed by watching in Torch on the Mikrotik or the Speedtest logs). As I mentioned, even iPerf displays these symptoms

My "MTU-embiggening" strategy does work, so it's not an insurmountable issue, and most subscribers never see (well, never report) any problem.


Hello,

We are certainly taking this issue of perceived performance seriously as I can imagine the end users comparing and complainig. We are running some detailed speedtest related tests and will make a solid recommendation shortly. 

Thanks

Sakid

2 Likes

I'm not sure if this info helps or not - but with my 2.4 Ghz ePMP from home, it's QOS is set to 10 Mbit down and 1 Mbit up. When I test to SpeedTest.net or to our local SpeedTest server, I get the same results - bang on 10/1 all the time.

FEB_01_17_TODDC_CAMBIUM.jpg

1 Like

@Sakid Ahmed wrote:
We are certainly taking this issue of perceived performance seriously as I can imagine the end users comparing and complainig.

HEY - you know what would help...   a 'burst mode' or a 'bit bucket mode' or a 'quota / fallback mode' in the QOS settings. ;)

1 Like

The public speed test system has been a problem for every ISP out there since its inception, Those of us ISPs whom actually give our clients the promised speed are over-shadowed by the few ISPs whom have hid behind the marketing terms "upto". Depending on where you are, this can have a different meaning. In Canada this basically means (federaly) that if 10% of your clients can actually get the advertised speeds, then your not lying.

This gets compounded by the client whom is accessing a test server outside your network control. Yes they are paying for a package on your network, but if you peer with a network at a lower rate and the "best path" is through that peered network, then the test flubbs and they rant and rave about a speed test score thats horrible. This is why most of the big ISP's have added an internal, at the edge official speed test server. "If you get your speeds to the edge then its not us!"  and this is enforced by the EUA used to bind the client in agreement. We actually have a clause in our user agreement that specifies that our speedtest server is the only speedtest server that will be used to confirm speeds for any package offered. This has had some clients ruffled but simply explaining that if the town clears the snow from the local streets but the highways haent been plowed, do you yell at the town for not plowing the highways which are not in there control? the message hits home quickly. (use whatever apt analogy that the client can relate to, we have even harkend yelling at the landscaper for somthing the roofer across the street did)

Make sure your clients know that you are only one part of the wide internet, not the whole internet.

That said:

To answer your question, yes I have had MTU based fragmentation issues. This was an issue when we first deployed the epmp hardware and we worked around it by ensuring we could handle the max MTU possible for the equipment and pushing a lower MTU to the client STA. TSS clamping would have been nice but its not required. For the record the APs are set to 1700MTU and the STAs are set by radius/snmp to 1508. The 8bytes extra are for the wireless side headers that are usually appended without fragmentation to all smaller than MTU minus 8bytes packets. We came to this size by adjusting the MTU untill a full 1500byte stream would pass unfragmented. This works but has made the AP slower to interact with directly since its expecting 1700byte packets. 

1 Like