Force 110 (2.3.3) TCP performance issues

I have deployed two pairs of ePMP Force 110 backhauls and both are experiencing the same issues, hopefully we can find some insight to this.  It appears to be an issue similar to that which was caused by the Ubiquiti ToughSwitch where buffer issues were occurring when you have dual speed (100/1000) connections on the same switch and there wasn't enough of a buffer to hold onto the packets to switch them down to 100Mbps.

What I am seeing is as follows:

Mikrotik -> ePMP Force 110 ~~~> ePMP Force 110 -> Mikrotik

When I run the wireless link capacity test within the radios I get 93/73Mbps for downlink/uplink.

When I run bandwidth test within the Mikrotik routers I get approximately 91-95Mbps/70-75Mbps each way when testing with UDP.

When I run bandwidth test with the same parameters, just TCP instead of UDP, I can never exceed 31Mbps, but average about 25-27Mbps each way.

I am wondering if anybody else has experienced this issue with TCP traffic.

 Look at Mikrotik CPU when  TCP bandwidh test is running. Does it have   100% load?

Yes, however this is the case on any of the single core routerboards.  Using Ubiquiti backhauls between the two sites we were seeing 50-60Mbps in the same tests.  I really feel this is isolated down to the Force 110 rather than the routers because of this one simple fact.


@Jordan Young wrote:

Yes, however this is the case on any of the single core routerboards.  Using Ubiquiti backhauls between the two sites we were seeing 50-60Mbps in the same tests.  I really feel this is isolated down to the Force 110 rather than the routers because of this one simple fact.


What is DL and UL RSSI level? Set STA target receive level -50 dBm and achive stable 100/100Mbps UDP simplex or 50/50 Mbps UDP duplex. Upgrate device to v.2.3.4. Previous 2.3.3 has bug -losses of retransmited fragmented packets. Packets losses may reduce TCP throughtput. ePMP ussually has equal TCP and UDP test results due to high link quality and low Packet Error Rate (PER). In your tests you shoud also get equal TCP and UDP test results .

Is that bug also in 2.3.1?

Yes.

Working with OP. Updated to 2.3.4  , issue remains exactly the same.

Wireless link test shows 93/73.

Mikrotik UDP btest shows 93/73

Mikrotik TCP test shows 24/24

Mikrotik CPU expectedly hits 100% on the TCP test, trying to investigate if thats the bottleneck here.

We will post a writeup on this shortly that has to do with TCP window size on MT software and relationship to latency.

In the meantime, could you please tell us what version of MT router OS you are running and which routerboard?

The Force 110 link is setting between two RB493AH's running 6.11

We're not opposed to upgrading them to a more recent version if you have known issues with certain versions.

A document has been posted in our knowledgebase section about TCP tput and Mikrotik bandwidth test tool.

http://community.cambiumnetworks.com/t5/ePMP-Networking/bd-p/ePMP_Networking


@Sakid Ahmed wrote:

A document has been posted in our knowledgebase section about TCP tput and Mikrotik bandwidth test tool.

http://community.cambiumnetworks.com/t5/ePMP-Networking/bd-p/ePMP_Networking


That post gives a lot of good information.  Can you give some more details about the test.  What was the link distance, how was the wireshark capture done.  Did you have to manually make the changes on the mikrotik routers or were they done by routeros automatically.  What were the full duplex test results, if I understand how your sync mode works, shouldn't the full duplex test result equal close to the sum of the single direction download and upload tests?

Was the cpu maxed out on the routerboards?

Hello Chris,

I will attempt to answer your questions.

- This was a lab test ran with ePMP connectorized devices using RF cabling with signal level around -60dBm to allow highest modulation.

- This specific wireshark capture was done using Riverbed AirPcap Nx product, using proprietary dissectors.

- The test was performed without any changes to the RouterOS.

- With TCP, the full duplex results were around 24-22Mbps each way. Remember that the TCP sliding window mechanism relies on packets to be acknowleged by the receiving end, so the window can advance and more data be sent. With full duplex, you not only have acknowledgement packets but also data packet contending for bandwidth each direction, so this has an impact on thoughput.

- MikroTik CPU utilization was not monitored during the tests.

Regards,

Luis

Thanks for the quick response.  I suppose you see a real nice increase when you increase the connection count. 

I'm curious, have you run similar tests with the pmp450 line?

Hello Chris,

Unfortunately I don't believe this specific test has been tried in our labs with a PMP450 link. We normally use other tools during system test, which are specifically design for that purpose (e.g. Ixia IxExplorer/IxLoad/IxAutomate suite of tools).

Regards,

Luis

Upgrading to 6.24 seems to have made a difference.  We are now up from ~25Mbps to ~70Mbps and its possible we are now genuinely maxing out the Mikrotik's CPU.  Will continue to monitor.

what happened to this? I currently have 2 Eforce 180 on version 2.5.2, 4km distance (2.5 miles). On the wileress link test I get in flexible mode 150Mbps DL and 30Mbps UL. when I do a test via speedtest.net I only pass 30mbps dl and 30mbps UL. Please note my carrier gives me 200mbps DL and 200mbos UL so I am trying to pass as much as I can.

I have no idea why the reality is different, can someone help? thank you


@Charles_4 wrote:

Upgrading to 6.24 seems to have made a difference.  We are now up from ~25Mbps to ~70Mbps and its possible we are now genuinely maxing out the Mikrotik's CPU.  Will continue to monitor.


what happened to this? I currently have 2 Eforce 180 on version 2.5.2, 4km distance (2.5 miles). On the wileress link test I get in flexible mode 150Mbps DL and 30Mbps UL. when I do a test via speedtest.net I only pass 30mbps dl and 30mbps UL. Please note my carrier gives me 200mbps DL and 200mbos UL so I am trying to pass as much as I can.

I have no idea why the reality is different, can someone help? thank you