Multi PoP deployments

Products cnWave 60 GHz
1

Following aspects should be taken care in cnWave Multi PoP deployments

Layer 2 domain

All cnWave PoP nodes must be connected to the same Layer 2 broadcast domain. PoP nodes learn about other PoP nodes using IPv6 multicast packets, which do not cross broadcast domain.

This allows cnWave PoP nodes to forward traffic to other cnWave PoP nodes via a wired connection when the routing path of the other PoP node is closer to the traffic’s destination. This concept is known as tromboning, because the traffic enters a PoP node and then leaves toward another PoP node.

Open/R on the PoP interface port

PoP interface port should be configured to run Open/R protocol. This can be done by marking the port as Multi PoP/ Relay port.

image

MTU of upstream switch ports

PoP ports use an 2000 MTU size. So, all the switch ports should be atleast 2000. Even if the user traffic is limited to 1500 sized packets, switch ports should allow the higher MTU size. Following packets exchanged between PoPs can be of higher size - Open/R packets, L2GRE packets (in Layer 2 mode), Software download packets.

Prefix Allocation

Its is recommended to use Deterministic Prefix Allocation

image

2 Likes
2

Hi Kiran,

Do all wired ports in the physical path between PoP nodes need to support larger MTU size?

3

@Enzo_Tirone Thats correct.

4

Thanks for the response. I noticed that in the user guide there is a disclaimer which notes that “Multi-PoP deployment is not recommended if onboard E2E is enabled”.

Are you able to provide more info on why it is not recommended and what are the limitations/risks if Multi-Pop is deployed with onboard E2E?

5

Disadvantages of onboard controller

  • Topology is limited to 21 nodes. Typically a single POP would suffice for a small network
  • In Multi POP deployments, if the POP hosting E2E goes down, the network will no longer be manageable.
  • Upcoming Network Analyzer and Network Optimizer (NANO) features require external E2E.
1 Like
6

Aside form what you have mentioned, do you see any issues/challenges with having Multi-Pop in a 21 node network?

Thank You

7

Otherwise, it works.

8

Does the MTU on the E2E Controller need to be 2000 bytes or can it be 1500 bytes MTU on the E2E Server, is 2000 bytes MTU just required for DN’s to Communicate or also for E2E to communicate to the DN’s

9

Setting the MTU size to 2000 is only necessary for POPs to communicate with each other via their wireline interface (core switch interfaces / bridge that all the POPs use). It is not necessary to set an MTU of 2000 on the E2E controller interface.

1 Like
10

With the requirement for DN’s / POPs and E2E Controller to all be in the same Layer 2 Domain , is there any potential issues with using MPLS/VPLS / Tunnels between POPs and the E2E controller instead of a flat switched network ?

11

If I understand it right, at the end all of them will be in same broadcast domain. So Tunnels between POPs and controller is fine.

12

Hi Kiran, thanks for the info.
How does one onboard a POP node to an existing E2E controller via the wire?
I have them on the same broadcast domain, added the MAC of the 2nd V5000 to the E2E, however, the node just continues to show as offline.

13

@dlbNet - To add PoPs in cnMaestro we need to follow the below step. Hope the same was followed for onboarding First PoP.

  1. Add the Site in cnMaestro
  2. Add PoP node in the site with reachable wired IPv6 config(IPv6 address and Gateway)
  3. Once the Node addition is successful, Download the onboarding config from Second PoP in cnMaestro
  4. Open the PoP node GUI and import the configuration(json file downloaded in 3rd step)
  5. Verify the PoP minion logs to ensure that it is able to connect to External E2E Controller IPv6 address to onboard
14

In Multi POP deployments, what happens of the hosted E2E goes down? Presumably the same sort of failure? Is there any built-in redundancy for hosted E2E controllers?

15

When the E2E controller is down, network will continue to work. Customer traffic continues to be bridged/routed. But network will be not manageable, that is configuration cannot be viewed/modified and statistics not available.

At present, there is no redundancy of E2E controller.

16

We currently have a single PoP deployment but will soon be moving to a much larger multi PoP deployment. We are using an external E2E controller that is running as a VM at our data centre. We use BPG to route IPv6 between the PoP and its upstream router, as the E2E controller is not on the same LAN as the PoP. The PoP DN and its upstream router are using different private AS numbers for the BGP peering. When we move to a multi PoP setup should the other PoP DNs use the same private AS number or does this matter? Also, do the other PoP DNs need to have a BGP peering with each other or just with the upstream router? I am guessing that they do not need to peer with each other, as Open R will take care of routing between them, but I just wanted to be sure on this.

17

All PoPs should use same ASN. PoPs do not need peering with each other. PoP need to have BGP session just with upstream router.

18

We have a use case where we will have two POP nodes that are connected to different BGP routers on our network, with only Layer 3 routing between them i.e. the two POP nodes are not connected to same LAN. I know there are guidelines that POP nodes should have Layer 2 connectivity between them, but this may not always be possible or convenient. Should this kind of setup work ok and what (if any) are the possible issues with doing this? Thanks.

19

First step is to enable flags.enable_pop_prefix_broadcast as described in Unconnected PoPs. Otherwise second POP and the devices in its zones may not receive prefix allocation.

Additionally disable MultiPoP/Relay port as PoPs are not in same Layer 2 domain.

Before we get to possible issues, can you please tell me if you intend to enable Layer 2 in cnWave or use it in Layer 3 mode.

20

We will be enabling Layer 2 in cnWave as we use PPPoE for provisioning customer routers.