Force 400C - QoS

I’ve had a single pair of Force 400C’s im using as the 5G failover on a Siklu 70G radio. This site has grown beyond the capacity of the 5G during heavy rain events and I was anticipating using the QoS DSCP traffic prioritization feature as found on the Force 300 but it appears DSCP hasnt made it to the Force 400 yet.

Is the radio DSCP aware at all? Is there any way to QoS over these radios? This really sucks. I may have to replace them.

not in the current software.

depending on the switches/routers you have on both ends, you can implement a policy and enforce shaping and queueing on the 5ghz link without needing the radios to do anything more than they currently do.

1 Like

Thanks for the confirmation.

Yea, but the 5Ghz (200M) is just a failover while the 70Ghz (1G) is the primary. So then the Mikrotik routers would somehow have to know that the link is in failover (scripting I imagine), then enable QoS policy for 5G.

We do have a fiber failover from a fellow WISP, but its an unofficial 200M. Im gonna have to learn some OSPF magic now. It was inevitable I guess

since you have tiks, use policy based routing and include the policy and queueing after the decision point. use an ip sla to monitor the connections and choose when to start moving data over the backup link.

You will need someone more knowledgeable with Mtiks to help get this setup but since I do this with cisco routers, it can be done.

1 Like

Depending on your network, there are a couple of ways to set this up. So I have a few preliminary questions:

  1. Are you already using dynamic routing (OSPF, BGP, or both) between these sites?
  2. Do you care if traffic is not flowing across this unit unless its a failover situation?

We have the failover set up on our network over a couple of larger links like the one you described. Simplifying it down, we have both links being run via OSPF, the 5Ghz is cost high (100) in our case. We then have a scheduled script that acts as our SLA and watches the neighbors. If the 70Ghz link in your case drops its neighbor it changes the cost of the other link. This does force our OSPF to re-build, but that is usually a quick drop since the link was already established at the higher cost. Once the main link is back online the same script reverts the cost back to 100 to tell traffic to go back to the 70Ghz radio.

After this setup, all you would need would be to set up your Mangle/QoS to handle the traffic that is only going over the 5Ghz link.

Here is the script we use:

# OSPF SLA Script for Mikrotiks

:local pri
# Primary link: set to the name of the interface in the router
:set pri "CHANGE ME"

:local back
# Backup link: set to the name of the interface in the router
:set back "CHANGE ME"

/routing ospf neighbor
:local found
:set found 0

:foreach n in=[find] do={
    :if ([get $n interface] = $pri) do={
        :if ([get $n state] = "Full") do={
            :set found 1
        } else={}
    } else={}
}

/routing ospf interface
:if ($found = 0) do={
    :if ([get [find interface=$back] cost] = 100) do={
        set [find interface=$back] cost=10
    } else={}
} else={
    :if ([get [find interface=$back] cost] = 10) do={
        set [find interface=$back] cost=100
    } else={}
}

We usually install this directly in a system scheduler instance with a reasonably low interval (5 seconds for us) based on your case. Just remember you are putting more load on the router checking in really short intervals, so I wouldn’t go much lower than 5. All you have to do is change the lines that say “CHANGE ME” with the name of the interfaces of your radios. The script checks for a neighbor using that interface and if it is not up, changes the cost of the other interface. Side note, these links are set up as Point-to-point for us. I cannot guarantee how well it will work with other OSPF interface types.

Here is a terminal-friendly version you can just paste in:

/system scheduler
add disabled=yes interval=5s name=SLA on-event="# OSPF SLA Script for Mikrotiks\r\
    \n\r\
    \n:local pri\r\
    \n# Primary link: set to the name of the interface in the router\r\
    \n:set pri \"CHANGE ME\"\r\
    \n\r\
    \n:local back\r\
    \n# Backup link: set to the name of the interface in the router\r\
    \n:set back \"CHANGE ME\"\r\
    \n\r\
    \n/routing ospf neighbor\r\
    \n:local found\r\
    \n:set found 0\r\
    \n\r\
    \n:foreach n in=[find] do={\r\
    \n    :if ([get \$n interface] = \$pri) do={\r\
    \n        :if ([get \$n state] = \"Full\") do={\r\
    \n            :set found 1\r\
    \n        } else={}\r\
    \n    } else={}\r\
    \n}\r\
    \n\r\
    \n/routing ospf interface\r\
    \n:if (\$found = 0) do={\r\
    \n    :if ([get [find interface=\$back] cost] = 100) do={\r\
    \n        set [find interface=\$back] cost=10\r\
    \n    } else={}\r\
    \n} else={\r\
    \n    :if ([get [find interface=\$back] cost] = 10) do={\r\
    \n        set [find interface=\$back] cost=100\r\
    \n    } else={}\r\
    \n}\r\
    \n" policy=read,write start-time=startup
3 Likes

Oh hell yea! This is amazing! Yes, we just transitioned to a OSPF/BGP based network. I have a firm grasp of it now, but im still pretty new to solving these types of problems.

I actually just started using ECMP to make traffic go over both paths (we had very heavy rain storm come through the other day) to keep traffic under 200M for now. And Ive started entertaining the idea of doing an OSPF Transit Fabric Traffic Engineering for unequal links (basically create multiple VLANs and ECMP all proportionally to each path). But one caveat to this method is that my failover traffic has to go over an EOIP tunnel back to the Core to maintain the iBGP peering (due it being over another ISP). This has the consequence of a slighly lower IP MTU of 1458. Im not exactly sure how much of an issue that poses but its something I feel I should avoid if possible. Not to mention, this failover was offered free of charge as a kind gesture from a fellow WISP, so I certainly dont want to abuse it.

Anyways, I say all of that to say, this seems like a much better option. Thank you!! Ill let you know how it goes.

After looking over the script more cloesly, I realized I failed to mention that the 5G failover is built into the Siklu antenna radio so the transition from 70G to 5G is internal on the Siklu (sub-milisecond failover) and is all going over the same OSPF interface. All that said, this is a great starting point and I believe there may be a way for the Siklu radio to signal to the Mikrotik that its on 5G. Ill keep yall posted.

You could use the QoE solution! A WISP with a similar scenario already used it during a failover and the QoE save the day by avoiding the whole traffic overload the PTP backup link.

Are you referring to the Cambium QoE? Or something like Preseem? I’m using Preseem but they cannot QoE on backhauls, only the AP’s themselves.

And Force400C does not have software available for QoS/QoE as far as I’m aware.

Our Cambium QoE solution gives you similar functionality as Pressem, but with the chance to accelerate TCP, do rate limiting and traffic shaping, this last one could be the clue to reduce traffic and don’t overload the PTP. This solution could be sit in a box/appliance at your NOC and that way you could control and enhance the backhaul as well as the last mile!
If somehow I can help, you could reach me at mauricio.padilla@cambiumnetworks.com

We actually have been running that setup for a couple of years now. There were some definite growing pains, but the BGP on OSPF routing has made it so much easier to manage traffic flows and the VLAN balancing is a lifesaver on dissimilar links.

As far as the EoIP failover link, the MTUs will affect you some but if you’re not using jumbo frames the effect will be minimal for most traffic. We actually use eBGP between sites and then just clean up the private ASNs in the core before advertising to our uplinks. It’s definitely a task to implement (it’s actually the latter half of the slide show you referenced) but can be worth it in these situations. If that link is only between those two routers, and those routers only talk over it and no other link router to router, then you could use eBGP routing policies to keep your traffic off of it until there is a failure and traffic will just drop to it. Otherwise, this SLA script has been a life saver in those situations. We even have a similar version set up on our Juniper core that does this if our Preseem server fails for whatever reason. As soon as Preseem is unable to send the OSPF control data across the “link” our traffic fails to a direct connection around the unit.

While looking for a reference for you, it looks like our friends at IP Architechs have created a new series that go over using communities to influence traffic flow on their blog. There are 3 parts as of now and the others are listed in the pingbacks at the bottom of the first writeup.

As for the Siklu, I have not played with it, but I know they support vLAN configurations per ethernet/wireless port, so you might be able to split the traffic up that way. I have only looked at it once and never spent much time working on that deployment.

2 Likes

Man, every bit of this post is gold, thank you!

As far as the EoIP failover link, the MTUs will affect you some but if you’re not using jumbo frames the effect will be minimal for most traffic

Of course, I just got finished implementing jumbo frames to my entire network…so this is a very good tidbit to know! So if I were to revert the frames for the EOIP, would it be enough to do it on that particular Tower router or would it have to be the entire path including my Core/Edge routers?

At this particular site, I do have all VLANs going over the SFPs of a CCR1036 so I could easily break out the EOIP path on the router to an ethernet port and switch ports with standard frames. But what about everything before and after it?

EDIT: And by problems, do you mean with Router speed performance? I have this EOIP tunnel between 2 CCR1036’s and I am able to saturate the +400M connection I do have. Im just concerned if the “problems” will be noticeable on the user end.

I’m not too informed on jumbo frames, so I can’t answer most of the questions you have regarding running standard frames on the EoIP link.

As far as the performance is concerned, sending larger/full-size frames through the link causes fragmentation across the EoIP. Basically, the packet is so large that the transmitting router has to break it apart and the receiving one tries to re-assemble it once it receives all the fragments. This slows the performance of the router some (honestly, I’m not sure how much at the traffic level you are describing). The biggest hit is if that traffic needs to be real-time (video conferencing, large VPN packets, etc.) it can cause issues that customers may notice. You can force the MikroTiks not to fragment the packets across the tunnel. According to my limited understanding (someone please correct me if I’m wrong, and RTFM and do your research as well), this is the same as the device sending speed notifications back to the subscriber. The sub’s equipment receives the notice and re-transmits a smaller packet out to the internet. This might cause some slowness or interrupts, but they tend to be all at once and then back to business as usual once the program/device knows what size to send over.

The best example I can give you of this is gaming with excessive latency. With fragmentation enabled, the link can cause jitter as the connection jumps from great to medium/poor and back again depending on the level of fragmentation for that one customer’s connection. With fragmentation disabled, it reduces jitter, after determining the latency necessary on the link. I can play reasonably well with a consistent latency of 200ms after the initial handshake, but if that latency is jumping around it’s more likely to cause issues.

Standard warnings apply: YMMV, RTFM, and I’m learning as I go as well.

2 Likes

Thanks for the reply.

I just had this epiphany regarding enabling jumbo frames (and please correct me if im wrong), is that if the sender is NOT sending jumbo frames (none of our CPE are set for jumbos, its just on our Towers and Core), then having jumbo frames enabled isnt going to force a normal frame to become a jumbo throughout the network, its just an option if a jumbo frame happens to appear. And even if I set all the CPE for jumbos, the customer would still have to have their own devices set for jumbos for it to create the jumbo packet (which I highly doubt).

So if thats the case, then I think im good. I say that under the assumption that we arent aware of any issues running over the EOIP. Just trying to make sense of it.