I’ve verified now that the 3 latest links we’ve deployed cause VPLS tunnels to flap every 3~5 minutes. Two of these links were currently AF5xHD and had no issues until the Force 425 radios were swapped in, running 5.3 at the time of swap. The third link, I upgraded from 5.1.3 to 5.3 over the weekend and pushed traffic over it from the link it was acting as a backup for.
The resolution on all three links today was to downgrade them to 5.1.3. That is the only change required for stable tunnel operation. Fortunately, none of these links carried critical traffic, other than the third link. But that also meant that it took until businesses opened today for me to find out about the issue.
I have not tweaked any of the performance knobs in 5.3 on these radios - the performance was set to balanced. Ethernet MTU on the radios set to 1700 and 1600 on the router interfaces, per our usual.
Time permitting, I’ll setup a lab to verify, but simply downgrading firmware on all three links today resolved all issues in and of itself.
Putting this out there in case others run into the same issue.
thank you for the report! I will try to replicate this case in the lab.
At the same time could you tell me if you noticed anything in performance of radios or any logs regarding VPLS in your routers?
Nothing in router or radio logs and no sudden swings in MCS rates or other performance indicators that are obvious. The tunnels will just drop approximately every 3 minutes with 5.3. LDP neighbors stay established just fine, it is just the VPLS tunnels.
Downgrading to 5.1.3 fixes the problem immediately without any other changes.
Are you using any QOS settings? VPLS doesnt like the f300-25 with any QOS enabled for some reason so maybe there is something similar?
I am not. There are no QOS options on the 400/425 at this time, but I wouldn’t use them on backhaul links like this anyway.
I put F425 link between two RB1100 and setup VPLS. It works smooth with 5.3.0 for a couple of days.
@Jacob_Turner, please PM me in case we can have a remote session. I want to see this issue.
@aka to really test you would need the tunnel to terminate on a router somewhere else on a network, rather than on a directly connected router. I’ll setup some remote access and PM you. I can really only play with one of these links as the others carry customer traffic.
Please forgive my child-like example of a topology drawing here. The red lines are the VPLS tunnels. You can see in the image the various paths between sites. The red lines are examples of where this problem has demonstrated itself with 5.3 but works fine with 5.1.3
We are also experiencing this problem with force400c radios, we have downgraded back to 5.1.3 in several locations
Could you please provide the configuration of VPLS tunnel?
We tried to replicate the issue in the lab and it works well for us.
Thank you in advance.
3 posts were split to a new topic: Problem with 400c RSSI
We are currently experiencing flapping issues with the F425 device in our OSPF network setup:
R1 <==> sw1 == F425 === F425 == sw2 <==> R2====WAN
To address this problem, we have decided to downgrade the firmware and observe if it resolves the issue.
We know about the issue in ePTP mode 160MHz channel. VPLS tunnels can flap. We are currently investigating. But it was not seen on 80 MHz.
Could you PM tech support files when the issue is in progress and confirm if it is present on 80MHz band?
@aka just a FYI - all of the VPLS tunnel flapping I’ve had has been when the radios are on 20Mhz or 40Mhz channels.
Is it on 5.4.1 also? Do you perform any actions to recover or it goes away by itself?
Can I have tech supports? We have already an engineering ticket. But in our lab we see it on 160 MHz only.
Sure will run another test and download the file.
we also have the same problem with 5.3 while with 5.1.3 , everything ok now let’s try 5.4.1
We have already published RC15 in beta thread!
This VPLS tunnel flapping issue still exists on 5.6 @aka
I have two links running 80Mhz channels that this behavior is being experienced on. I need to do a bit more digging, but I feel like the problem doesn’t present itself when running 20Mhz channels. I’ve played with MTU on the router interfaces and no difference is made. Currently running 1640 on the router interfaces and 1700 on the ePMP radios. The tunnels flap both when loaded with traffic and when idle.
I’ll gather tech support files from one of the links and PM them to you. Both of these links are to be replaced with something else once the weather is better. They both serve as backup links now, but backups that flap are almost as bad as no backup at all.
I’ve got you techsupports. Thank you!
How often does it happen?
Which equipment terminates the tunnels? Does it show anything in logs?
Is there a chance to show us when it is in progress?
I will analyze techsupports, discuss it internally and return to you with some update.
Every 3~5 minutes.
Mikrotik. Nothing is logged, the tunnel drops and comes right back up.
I could set something up in a portion of our network where the tunnels aren’t needed for customer traffic. This issue is very service-affecting for customers though, so I can’t push live traffic back over these force425 links.