We have a network of ePMP 3000 AP and is using Force 300-25 on our clients. We have issue on throughput particularly when at 100 Mbps and above. We were getting 300+ Mbps Downlink on the Wireless Link Test but barely 100 Mbps on the Ookla Speed Test. I was told that Wireless Link Test is based on UDP traffic ang Ookla is TCP based but the difference should be this much. Can anybody help me figure out the problem here.
Transport protocols is not the only difference here. Build in speed test in Cambium radios tests the link only. But when you run Ookla it involves whole your network and other networks between the radio link and SpeedTest servers. There can be bottlenecks.
To get more precise result you can connect laptops to AP and SM to try iperf in TCP or UDP mode.
Edit: not saying Andrii isn’t right. If you want to know if the speed problem is in fact the radios (SM and AP) then you will need to test from a device directly connected to the customer radio at one end and another one directly connected to the AP at the other end. Now if this was Ubiquiti it would be as simple as running the speed test built into the customer radio to any Ubiquiti AP (or any other Ubiquiti radio anywhere on your network really) on the other side of the customer’s AP because Ubiquiti radios allow you to specify the IP of whatever you want to run the test to… ePMP however can only do a link test between the CPE and the AP it is connected to. Perhaps they will make this available some day, call it “advanced link testing tools” and charge license fee for it…
One very common culprit when test can’t break 100Mbps is somewhere (the connection between the customer radio and the router ? The connection between the router and the device you are testing from ? You aren’t trying to do this via WiFi are you? ) you have a 100Mb Ethernet connection in the mix.
Could also be ePMP firmware. In the past we have had real problems with ePMP being really bad at speed tests (and Cambiums clueless as always answer was “well you need to have your customers run iperf because speed tests are not accurate”). Of course the speed tests were fine after the customer dumped us for the cable, DSL or other wireless provider all whom delivered the expected speed test results that customers these days use as the last word in whether or not you are providing the the speed they are paying for.
That said, they are technically correct, all Internet speed tests are, at best, unreliable. I almost always run 3 different tests to 3 different sites to get an idea (and it’s still not 100% accurate) of what the link can do. Ookla is only as good as the ISP it connects you to, IMHO it’s the least reliable of all the speed tests.
Of course what works where I am may not be the best where you are. I’m connected to my LAN/WAN with 1Gb ethernet and have a 10Gb Internet connection that is currently only moving about 2.1Gbps down 170Mbps up of its 10Gb/10Gb capacity. I ran each of the below speed tests 3 times each:
Hi Andrii. I don’t think there is bottleneck in the network. We did a speed test (Ookla) on the network port where the AP is connected and we were able to get 350+ Mbps. Afterwards we connected the AP and then we did the same speed test (connected to the same test server) at the ethernet port of the SM. We barely hit 100 Mbps. I’m okay with iperf but Ookla is the widely used tool by clients here in my country in checking their internet subscription. It works on the wired network like FTTH, DOCSIS, etc.
To give you further info on this link, the AP is newly installed and is just trying to serve its first client whose subscription is 200 Mbps. The client is very near the AP, around 900 meters only. Channel Bandwidth used is 80 MHz. In the Downlink direction, the modulation is steady at DS MCS 9. Link Capacity and Link Quality in the downlink are both 100%. SNR is at 50 dB. In the Uplink, MCS varies from DS 6 to DS 9.
To me, this is almost a perfect link so we were expecting to hit max capacity of 500-600 Mbps (aggregate throughput) as advertised in the Cambium ePMP Force 300-25 data sheet. We got that number in the Wireless Link Test but not in the real internet traffic.
Just my observation, we have no problem with the Cambium ePMP AP and SM if the required throughput is less than 100 Mbps. We were never successful in putting up 100+ Mbps in a point to multipoint set-up.
We have one client before subscribing for 100 Mbps internet bandwidth. We tried to serve the client via our ePMP 3000 AP. We got 100+ Mbps on the Wireless Link Test but barely 50 Mbps on the Ookla Speed Test. Client won’t accept it so we resorted to Point-to-Point Link using the Cambium Force 300-25.
Please investigate on this.
Hi brubble1. We did the speed test directly at the ethernet port of the Cambiume Force 300-25.
We tried different versions of the firmware, i.e. v4.5.6, v4.6.0 and v4.6.1. We even tried having the SM on different version with the AP. Same results. Not tried the v4.6.2 which has just come out.
So, when you are plugged directly into the customer radio with this laptop, or whatever you are using, is your ethernet connection 1000Mb/Gig ? If you look at the AP does it show 1000Mb Ethernet link ? I would lean towards it having to be a problem on the wired side of the radios if it was 20Mhz or 40Mhz channels because I’ve used those a lot and one the one thing (hell the only thing) Cambium ePMP is good at is delivering the advertised speeds (again, at 20 and 40Mhz in my experience) but I’ve really never done 80Mhz channels on multipoint so I can’t claim that I know for a fact it works as advertised.
Definitely sure that the Ethernet Port Status of the Access Point (AP) and Subscriber Module (SM) is 1,000 Mbps. We have been deploying Cambium Radios for a couple of years now and we are very particular on the Ethernet Port Status. The Ethernet Status is on the top section of our Field Acceptance Check List. The laptop computers that we are using are all with Gigabit Ethernet port.
The issue is not on the 80 MHz channel size. It does work at 80 MHz since we are getting the expected throughput for an 80 MHz channel size in the Wireless Link Test. The issue is the huge difference with the throughput based on Ookla speed test.
I know that throughput based on Ookla is lower compared to that in the Wireless Link Test because of the overheads but it should be not that far.
What I was saying was there may well be an issue with 80Mhz channels and TCP vs UDP or something that causes the radio to puke on itself when something about the conditions of running a speed test happen.
Is the customer radio in bridge or router mode ? Using PPPoE ?
^^THIS^^ Among the many, many things that Cambium gear doesn’t do because Cambium doesn’t think it’s needed, is this. THIS feature (available in nearly every other brand of radio we’ve ever used, other than Cambium) is super useful to take only about “3 minutes to diagnose where in the 100 km chain of 8 towers and 24 radios the broken cable is, or the wonky switch which needs a power cycle is”.
It’s “under consideration”…
Subscriber Module is on Bridge Mode.
Well now I’m curious… if I can I’m going to bump the 3000 I’m connected to at home to 80Mhz (in the very early hours when almost no one is using it) and see what I get speed test vs link test.
Edit: Would also be curious to know what the frame time % looks like on the AP when you are running speed tests. Might be telling, might not.
Please try to limit UL MCS to 6 on the SM. I think I have an idea why does it happen.
It can be due to retransmissions of TCP ACK packets. SM switches to lower MCS when there are some lost packets. If one TCP ACK is lost AP keeps all next until it get the lost one retransmitted. To maintain in-order delivery. Then it will push all ACKs to Ethernet and it will make TCP server to issue a lot of data at one moment. It will look like a burst. And this burst can die along the way. So TCP server will decrease the transmission speed.
Specifically write in your TOS that service rates are not guaranteed to testing sites off your network.
The problem is that if the wireless link shows throughput as good but the ethernet links through the wireless does not then you have more going on that you think and need to systematically test everything including possibly swapping radios.
Testing to an off network test server exposes every little problem between you and said server as your problem when it really isnt, hence the iperf/jperf suggestions. A simple and cheap RPie on each tower allows you to test between sites to figure out what is going on. Keep in mind that you should be testing without the clients network connected to the SM or you will never get a good test.
My issue is the Downlink throughput. I am getting 100% Link Quality and 100% Link Capacity in the downlink. Packet retransmission should not be an issue since Link Quality is 100%.
Just because you have 100% link quality and capacity does not mean that retransmissions are also good! There are many reasons for high retransmission rates and Signal quality is just one. You could be on a very good reflection and not know it.
I would still get a jPerf/iPerf server (can be a rasberry Pie or an old PC with a gigabit NIC in it) and test to the tower with it.
I’ve just showed the scenario where uplink transmission can affect downlink TCP traffic.
We say our radios are transparent bridge and simply forward what they get. But in deed it is not so simple. There are plenty complicated traffic handling algorithms to maintain good throughput, latency, jitter and user experience.
Is there any update on your issue?
If you use Ethernet Tester, both Spirent Test Center (premium test equipment) and portable VeEx Ethernet Tester, both yields to abnormal throughput and latency test results either via manual throughput single stream tests only or standard RFC2544 tests. Applicable, based on actual tests with our customers for PTP550 and Force200.