Force 425 firmware 5.3 causing VPLS tunnel flapping

Products ePMP
1

I’ve verified now that the 3 latest links we’ve deployed cause VPLS tunnels to flap every 3~5 minutes. Two of these links were currently AF5xHD and had no issues until the Force 425 radios were swapped in, running 5.3 at the time of swap. The third link, I upgraded from 5.1.3 to 5.3 over the weekend and pushed traffic over it from the link it was acting as a backup for.

The resolution on all three links today was to downgrade them to 5.1.3. That is the only change required for stable tunnel operation. Fortunately, none of these links carried critical traffic, other than the third link. But that also meant that it took until businesses opened today for me to find out about the issue.

I have not tweaked any of the performance knobs in 5.3 on these radios - the performance was set to balanced. Ethernet MTU on the radios set to 1700 and 1600 on the router interfaces, per our usual.

Time permitting, I’ll setup a lab to verify, but simply downgrading firmware on all three links today resolved all issues in and of itself.

Putting this out there in case others run into the same issue.

5 Likes
2

Hi @Jacob_Turner!

thank you for the report! I will try to replicate this case in the lab.
At the same time could you tell me if you noticed anything in performance of radios or any logs regarding VPLS in your routers?

3

Nothing in router or radio logs and no sudden swings in MCS rates or other performance indicators that are obvious. The tunnels will just drop approximately every 3 minutes with 5.3. LDP neighbors stay established just fine, it is just the VPLS tunnels.

Downgrading to 5.1.3 fixes the problem immediately without any other changes.

4

Are you using any QOS settings? VPLS doesnt like the f300-25 with any QOS enabled for some reason so maybe there is something similar?

5

I am not. There are no QOS options on the 400/425 at this time, but I wouldn’t use them on backhaul links like this anyway.

6

I put F425 link between two RB1100 and setup VPLS. It works smooth with 5.3.0 for a couple of days.

@Jacob_Turner, please PM me in case we can have a remote session. I want to see this issue.

7

@aka to really test you would need the tunnel to terminate on a router somewhere else on a network, rather than on a directly connected router. I’ll setup some remote access and PM you. I can really only play with one of these links as the others carry customer traffic.

Please forgive my child-like example of a topology drawing here. The red lines are the VPLS tunnels. You can see in the image the various paths between sites. The red lines are examples of where this problem has demonstrated itself with 5.3 but works fine with 5.1.3

image
image1301×765 133 KB

2 Likes
8

We are also experiencing this problem with force400c radios, we have downgraded back to 5.1.3 in several locations

9

Could you please provide the configuration of VPLS tunnel?
We tried to replicate the issue in the lab and it works well for us.

Thank you in advance.

10

3 posts were split to a new topic: Problem with 400c RSSI

11

We are currently experiencing flapping issues with the F425 device in our OSPF network setup:

R1 <==> sw1 == F425 === F425 == sw2 <==> R2====WAN

To address this problem, we have decided to downgrade the firmware and observe if it resolves the issue.

Thank you

12

Hello @Alfred_George!

We know about the issue in ePTP mode 160MHz channel. VPLS tunnels can flap. We are currently investigating. But it was not seen on 80 MHz.

Could you PM tech support files when the issue is in progress and confirm if it is present on 80MHz band?

13

@aka just a FYI - all of the VPLS tunnel flapping I’ve had has been when the radios are on 20Mhz or 40Mhz channels.

14

Is it on 5.4.1 also? Do you perform any actions to recover or it goes away by itself?
Can I have tech supports? We have already an engineering ticket. But in our lab we see it on 160 MHz only.

1 Like