We’ve got about 2000 customers seperated in 6 different network segments. Each of these segments is full of multiple towers all bridged together. Most of our customers have NAT enabled in the Canopy module but we have many customers paying for statics and need to be un-natted for VPN’s etc. The problem is every now and then somone creates a network loop in their office and brings down an entire segment. I also want to elimate the ability these clients have to browse the network if they get bored.
So how do you guys combat this? I can either put routers at every tower and subnet everything but we’re talking about 50 towers at this point. I’m also thinking about setting up a vlan for each customer that is un-natted. But that could get messy if we get big enough. Any suggestions?
We bit the bullet and put routers at every tower. Weighing that versus vlans it was the easier option for us. Our smaller towers with ~100 customers we’re using MikroTik 493’s and the larger towers we use the 1100’s or at the towers with over 300 customers our home brewed x86 quad core xeon w/pci-e and 4 port gig-e cards.
Did you create subnets for each tower and add IP’s as needed or are you using a dhcp relay?
We use routers at most of our towers , DHCP at each router , with OSPF for inter tower redundancy via back-hauls.
You will never look back !! and mikrotik is one of the best and most cost effective ways to go!
THEBRAD wrote: We use routers at most of our towers , DHCP at each router , with OSPF for inter tower redundancy via back-hauls.
You will never look back !! and mikrotik is one of the best and most cost effective ways to go!
+1
THEBRAD wrote: We use routers at most of our towers , DHCP at each router , with OSPF for inter tower redundancy via back-hauls.
this could be the right choice but...
1) do you assign public IPs to every customer?
if so, you have to subnet your public ip space among every tower (ip waste)
2) do you still NAT on customer side? (canopy SM)
what if the customer doesn't want NAT? Do you configure his SM in bridge mode?
if so, what if he's not using the DHCP released IP, but configure another public IP on his router/pc?
Thanks for sharing
Massimo
We release public IP’s to everyone. Thats why I am hesitant to to subnet 50 towers. Major IP waste. Especially if I try to route “router to router” with publics. Yes we Nat everybody with the Canopy module unless they are a business that needs their own router for vpn etc. Then we assign them a static and bridge them.
Coming from a DSL world our infrastructure is built around hauling all traffic to a central termination point. Currently we run a Redback SER-400 to terminate all PPPoE sessions. It keeps the gear in the field very simple as anything ‘complex’ is located in a real datacenter, and of course our IP usage is very efficient. Our network is built so that there is complete layer 2 isolation between all subscribers through the use of 802.1q VLANs, “protected ports” (similar to Private VLANs)at sites, and port-based VLANs on CMMs. Once up and running properly, a customer’s NIC can only talk to the Redback or whatever other equipment we have connected at the CO.
Of course there are various different approaches and some of them fall in the middle… such as placing a router at each site that will terminate PPPoE and send PPP with L2TP back to a box at your CO. Or depending on what MikroTik offers, another possibility is terminating (via DHCP or PPPoE or something) customers at each tower and having a centralized IP pool that everything grabs from. Don’t forget that OSPF is quite capable of supporting very many /32 routes so there is no reason to assign an entire block to a tower site.
Whatever you end up doing, though, I strongly recommend keeping management and subscribers separate. All SMs and APs should have separate VLANs configured for their management interfaces versus the path that subscriber data takes. This sort of thing is really easy to do with cheapo switches like a Cisco 2924 ($50 on eBay). And you can make the choice for yourself if you want to trunk VLANs around or just stick them into a router on site.
What about using a MT router as a PPPoE concentrator and enabling
PPPoE in the radios?
We use all juniper gear for our routing, each customer is given a public IP and the SM and APs are on private lans
if you are in the thousands of customers you might want to look at something a bit more robust than the mikrotik, not knocking the guys who use that gear, but it won’t hold up to some nice Cisco or Juniper routers and service gateways and looking at the service load, save a few dollars on a router could hurt much more than its going to help. Granted I haven’t used mikrotik much, but our out of business competition did and the customers we picked up from them complained about packet loss horribly, not sure it if it was thru his routers, WAN link or wireless, but they were avid mikrotik users.
At each of our towers we use 8 port service gateways, clustered if needed to have an individual subnet for each AP, a /26 public and a /24 private for our gear. We use DHCP rules to automatically give the canopy hardware privates and customers public. Also the routers we use can give a great deal extra insight to what the APs are really getting performance wise like PPS throughput. Also the routers can control packet fragmentation to help stop applications like lime wire kicking your discards thru the roof. Our edge routers have the wan acceleration added to them and I think that’s the best money we ever spent, we have a 50Mbps fiber Ethernet connection to one carry and a DS-3 from ATT with a little over 2,000 subs and our wan usage rarely gets over 60% the wan acceleration feature caches websites and commonly download files and rather than pulling them thru the Wan links it will simply access the site once every what ever time frame is setup for that particular site or file and that’s like windows update for example, all 2k users get updates at 3 AM and rather than it choking out our WAN links it hardly touches it same goes for the games downloading there patches ect ect ect
Before we implemented this setup we were using all layer 2 with vlan and it did ok, but our wan links were hitting max with only 800 subs, packet discards were getting to be a problem games would complain of lag ect ect. We almost used a lesser layer 3 system, but we defiantly made the right choice….
Spend some time and make a list of all your weak points in the network and see if the routers you are looking at will solve the problems and give you room to grow, the biggest thing that scared us away from the mikrotik was no benchmark data, I know they print advertised hopefuls of max throughput. But most of the big vendors will only give you the benchmark results in real world setups. And the 2nd was the price tag… the sales rep I spoke with about the mikrotik routers was trying to sell me the RB 1000 I believe it was for $800 trying to tell me it would out perform a cisco 3800 series router…. If your not familiar with the cisco routers… the 3800 series are service routers that came carry mpls frame replay ect. Ect. Not exactly comparing apples to apples.
The setup we have at each tower is:
Each port on the router has 2 subnets, public and private, the router is 10.9.x.1 and the AP is 10.9.x.2, SMs and phone devices get private IPs and the “unknown” macs get public IPs this is repeated for every AP
The backhaul ports are set to switch mode and share 2 ip interfaces, public and private from there we attach our backhauls witch are all in the same subnet (well one private and public carries thru them) to our network edges.
Pretty simple, easy to manage fault tolerant and extremely efficient. And considering we have gotten more than double the subs on the same wan links, its save us a TON of money.
Oh and 2nd the OSPF we went with BGP, but OSPF is clean and simple BGP is well… mostly clean and simple the BGP was a little better at balancing load based on latency and capacity of backhauls is why we went that way.
For us this is not an IP waste -
We get our IPs assigned to us directly through ARIN
We typically use an entire /24 network for large towers (sometimes 2) and /25 network for our smaller networks
The only waste we have is our OSPF network as we are currently using public IPs and I am working on converting to a private IP. Down side there is any looking glass utility from outside the network will fail once inside your network.
Every Single customer module is bridged and has protocol filtering to filter out several types of traffic.
For every network you make you are only ‘wasting’ 2 IP addresses ( broadcast and host ) If you have the IP space this is the way to go.