I am setting up management and traffic vlans for traffic flow across wireless links to customers. Here’s what I’m trying to accomplish:
I have the trunking and routing working at the router/switch level, and the ePMP management vlan 9 reachable.
How do I allow the customer routers to use DHCP and have the ePMP put them in the traffic vlan 10 and be reachable across the network? Here’s the config I have so far, what should I be doing here to accomplish this?
bridge mode on your backhaul radios with management vlan enabled and set.
Either nat mode with management and data vlan set to each vlan (do not fill in the vlan list) or bridged mode with management vlan set and the data vlan in the vlan table.
you could also do a routed setup which is more complicated but is not much different than bridged mode for this setup.
I want groups of subs’ traffic (by tower) on a small subnet then sent back to the core.
Can the epmp tag (access port) customer traffic and encapsulate it in a traffic VLAN trunk and send it upstream to the core Juniper, or do I need another network device inline to do that?
To just use per tower vlans, you need to have The APs and the SMs management vlan set and should be the same. You will need to set the data vlan on the SMs, but NOT on the APs. This will tag all customer traffic to that vlan.
At the core, you will need to do something to get the discrete vlans to be able to get to the internet.
THIS IS A FLAT NETWORK DESIGN and should be avoided.
A better setup would be to place a small but powerful enough router per tower and route all traffic to the gateway. This has a couple of up front benifits: 1) you will not have to worry about having multiple paths to a tower, your IGP will handle that and even balance the traffic for you. 2) Each tower would have its own subnet for device IPs and each customer could be served from a single subnet or you could give each tower a customer subnet. 3) Your network will not be a vlan’d L2 network which has its own issues (dont ask how I know) 4) Business customers that need static IPs routed to them can easily be done.
Any enterprise grade router would work, we use modified Cisco routers at each tower, but I know there are Microtik’s doing this just fine.
If your looking at pppoe or eap-ttls then having a router at the tower actually speeds up the feel of your network as your LAC will be local to the client and your LNS at your gateway will just be concerned with LAC connections and not client connections, this increases your networks ability to expand before major upgrades are needed.
So if I have a per tower router, is that a gateway that NAT’s and then sends bulk traffic back to the core, perhaps over a traffic trunk vlan? Something like this modified diagram? If so, that would seem to double the number of subnets floating around, and the corresponding routes for instrumentation (i.e. ping customer routers as heartbeat, etc.) no?
Of course, I could be missing parts of what you’re saying.
Is customer’s TP-Link managed by you or bu customer?
Just a placeholder for a cheap router. Lately we’re moving to routers that can be remotely managed. I think the cambium ones have the ability to set the WAN with a VLAN, but don’t really remember.
Sorry for the long delay.
The part I was refering you to was to move to a vlan per tower with per tower routing. A large vlan based L2 network will cause you headaches as though you did shrink broadcast domains, you also multiplied them. Routers actually delineate these broadcast domains and reduce all broadcast traffic. This also lets you setup additional layers to provide additional services such as mpls, L2 tunneling, traffic management, redundant link load balancing with automatic failover.
You have chosen pppoe as your access control, now look at lns-lac setups. This moves the point of termination from your core to your tower but tunnels to a core gateway to then be routed. If we take this one step further towards an eap-ttls type setup, we get rid of the lns, have the tower router act as a customer gateway for pppoe and route to wherever the data needs to go but authenticate with a radius server. You can use the radius server to push the connection specifics including pppoe virtual port speeds, thus giving you a more central method of control and package management.
Is L2TP supported by vanilla Mikrotik/Juniper SRX and commodity hardware in general (same with EAP-TTLS) without special licenses (Cisco does this a lot)? Would I see a perfomance hit due to the encapsulation and would intermediate routers have problems transporting that traffic across them transparently?
I see EAP-TTLS basically having Radius AAA per tower, but wouldn’t that mean I’d have to run a per tower server that somehow communicates (via mgmt vlan or some such) to the core? Not sure I’d want a radius server at each tower, would make management, um, more complex, no?
Also, with either, would there be issues with core instrumentation being able to reach endpoints for things like snmp/heartbeat over weird routing?
There is no real performance hit as long as your equipment can pass packets with an mtu larger than 1500. We use Cisco routers and switches and set everything to 1700 to match the APs and set the SMs to 1500.
Not a Mirotik guy, cisco requires you to have a service provider or higher license. You can get a good used ASR or ISR G2 from an datacenter pull and it will have at least a SP license. This is what I do.
You would not usually have a radius server at each tower, that would be way too much to keep straight 
. You have one in your core and one at a backup location just incase the main one isnt available. With an SQL database for easy client config changes, all you would have to do is setup database replication and the other server would be at most 5 seconds behind. We use freeradius3, Mysql and Daloradius web frontend to setup the database. We pass VSAs to the SM to configure package bandwidth, etc.
Eap-ttls does not require anything more than a router to act as a network access server. An IP service license can do this but Id go for a SP license which opens mpls.
Your routed network will pass all of your management communications just fine. Each tower will have its own management subnet and vlan for tower equipment and your routing protocol will do the rest. You can even pass vlans to the SM for service provisioning and have them have different priorities accross the network regardless of the dscp bits on the packet. To prevent client traffic from entering the management network, you should use different vlans and block routing between them. There is about 20 different ways to do this including q-in-q
Looks like Juniper SRX doesn’t support L2TP as a LNS, and I can’t find anything at all on JunOS on implementing EAP-TTLS. It looks like Mikrotik does some version of it, but it sort of looks like it’s just authenticating .1x traffic across some VLANs hooked into a radius server for auth, then passing it to some upstream router if I understand what they’re saying here: Dot1X - RouterOS - MikroTik Documentation
Some of our towers are solar powered, so I wouldn’t want to put a power hungry Cisco there I think, but also it would require full hardware swaps end-for-end possibly. Is there something I can do with my network for now and then try to implement that later once I get the hardware and test it?
Nothing that wont be costly later. Best to do it now before it gets too big.
Most Ciscos are 12dc inside, we power them with a 200w buck converter from 24v systems. So the idea they are power hungry is partly a misnomer. Some devices are power hungry when they are fully loaded, c6500 or c4507 with dual sups and full of 48 port poe line cards will suck back the watts. A isr2951 pulls 65w passing 200mbps and its older brother isr3825 pulls near 80w passing 100mbps. The ASR series is even more power thrifty. Since most of your traffic will be hardware routed, CPU usage will be low, we are seeing 30% up to 50% for short periods and its the CPU that hogs power until you add HWICs or network modules like switch or service modules. Cisco does tend to put larger than necessary power supplies as they have no clue what we do with the hardware, configuration wise, but that does not mean its fully used all the time if at all.
A typical solar site for us has a c2960s and at least a c3825 but lately c2951s are more common since they are better performing and we do not have to change how we modify them. We also have tossed out the “book” when it comes to how things are done at solar sites. We do not have a south facing centric system, infact we usually have 600w facing south where we have 900w facing east and west. Our batteries are huge 500ah (we calc from 400ah for derating and ensured reserve) Rolls-Surrette locomotive batteries and our whole goal is longer more consistent recharge time vs thebmore common higer current charge for a short time. Our system is off battery usually 2 mins after sunrise starts and doesnt go to battery until just before the sun sets. Actual runtime with 3 ptp links, a single omni AP, sync injectiin by packetflux and sitemon2, c2960s and a isr2951 all at 24v is 3 days (we do not get the sort of heavy overcast days that would kill this setup and we do not get snow buildup due to the combination of steep mount angle and the winds we get). We are usually fully charged by noon as of the current winter trend or before 10am during summer. We can also easily add another set of batteries in parallel to get 800ah (1000ah if you use the manufacturers rating) if we need more run time or are planning to add more hardware. Our heaviest loaded site with 10 radios, Packrtflux sync injection, a c2960s and a isr2951 and with 400ah we have 34hrs of battery run time, all powered by a 30a 24vdc psu (not a solar site but fully dc and except for a mppt charge controller, identical power system). For devices that need 48v (epmp3000), we add a dc to dc boost converter rated large enough to support the current needed load plus room for planned expansion though these can usually be paralleled to provide more avalable current as needed. Our sites have 12v, 24v and 48v loads, we chose 24v as that is the median voltage where most of our loads are. The ciscos are 12v and IP cams (we also have 12v ip cams) and e3k APs are 48v.
We are also toying with adding something like this: ebay ID: 313794988883 to our towers to take advantage of the fact we usually have a 2m/s wind all the time.
My point is, this is very doable once you break from the notion of an inverter must be used and actually looking at real consumption and using that instead of power supply available/max wattage. There are many ways of doing this, Microtik is mostly 12vdc inside though anthing with poe will have a 48v input too.
I have a c4506 and it is a power pig. Presumably it should work at the core end for EAP-TTLS (will have to check license), or also I have a C3560, don’t know if that will work.
I wonder if Netonix has something, we have a few of those sitting around. There must be something super low power. I guess I could run a raspberry Pi compute module made for industrial applications running Debian drawing around 7 Watts max, they have models with dual NIC and good specs https://www.seeedstudio.com/Dual-GbE-Carrier-Board-with-4GB-RAM-32GB-eMMC-RPi-CM-4-p-4898.html for $108.
It looks like vanilla Debian has wpa_supplicant package which supports EAP-TTLS like wpa_supplicant.conf(5) — wpasupplicant — Debian stretch — Debian Manpages so I think it could work, I’ll configure one port facing AP and one facing backhaul at a tower site.
We run 24VDC, but still 65W is about 10 epmp1K radios. We also upconvert to 48vdc with a charge pump for epmp3K.
Your tower switch really doesnt matter as long as its vlan aware. your tower router however does matter as it acts as both a NAS and a route point. If going Cisco, then you will need a service provider license. On EOL devices, Cisco does play nicer if you need to upgrade but also keep in mind that with the new universal IOS structure, licensing is on the honor system, you just get nagged in the syslogs.
Yes, your c4506 is a power hungry gremlin but it also has the switching capacity to meet most medium enterprises needs! That is most definitely not solar friendly though not impossible, but I would look at a 48v array rather than 24v.
Unless you have a Sup7e or newer, it will not act as a good router for eap-ttls, but its also not needed in this regard.
65w on a solar tower is actually not a lot if you design around your needed loads, we have 2400w of solar with a voc of 32v and a Midnite solar classic 150. You need roughly 10x the wattage on the array as the amount of load to always be ahead of your needs. It is all about time on charge rather than total amount of current available. Our design has all panels paralleled and they charge the batteries with what is left from running the tower. Try not to micro size everything as you will be disappointed in its performance. Tried this so I have a bit of experience. Plus a rPi is not a good platform for this, they are capable but everything is process switched and you will run into the CPU wall.
The deian package for wpa is for acting as a wifi AP not for acting as a network access server (nas).
We started with pppoe and a vlan bridged network. This worked very well, minimal equipment and only one LAC in the core. Then we moved the LAC to the towers and noticed a very large improvement in our backhaul performance as we minimized the amount of control data being passed and we then moved to routed between towers. We overlayed eap-ttls as it authenticates the hardware not the user and if a device gets stolen, we can force it to display that fact when it reconnects. All new connections are eap-ttls and we have moved almost every client to eap-ttls while leaving pppoe running. Both systems use the exact same radius servers and backend database, just a little different VSAs sent on connect. This has been a very stable approach and the only thing I wish was done different is that we started routed and skipped the vlan bridge mess.
A straight charge pump to get to 48v will leave you with a problem of stability. They work for low current uses but you will need 20w and more to boot up a e3k from cold, its short duration but you do not want to drop below 36v or they shut off. We use these (ebay id 143686772450) and find that one if more than enough for 4x e3k’s with power to spare. And the best part is they are a regulated output.
Right now I don’t have the ability to upgrade the solar on the most offending tower, and don’t want to do the mystical dance to get Cisco licensing to what it might be, so what can I do with low power in the meantime? PPPoE traversing across a VLAN to a core SRX router (which doesn’t do LNS anyway)?
It looks like Debian has an L2TP implementation for both LAC/LNS l2tpns(8) — l2tpns — Debian testing — Debian Manpages , could that work?
If I run out of CPU on the RPI there are other lower power options in the <20Watt category that don’t involve upgraded charge controllers, batteries, panels and mounting structures I think. At the core I can run whatever horsepower on a Debian server that’s needed, and then hand off the routing to my Juniper SRX?
Have you run into issues with some radios not supporting MTU of 1700?
Again, not a juniper or a microtik guy, cant help you with which to use since I only know how to break them.
If you have any cisco routers, then any of them can act as a pppoe server. IP services or advipservices is all you need.
Debian CAN act as a pppoe server, but you are process switching every packet and will need a big beefy CPU and lots of ram to do this. This will not be low power as it is CPU intensive!
Without knowing more about your towers actual power setup, actual loads, solar panel ratings, facing direction, wiring configuration, charge controller, battery type and AH ratings, All I can do is tell you that most of the devices you seem to think are not low power actually are when setup correctly.
Keep things simple and routed where you can. You are hooked on pppoe, I am not sure why since you can use a number of other systems to do what you want. PPPoE does not have any advantages over IPoE or EAP-TTLS. Essentially they are equal with EAP-TTLS being slightly ahead due to the added security.
If your core router can act as a LNS for pppoe, it can also act as a NAS for EAP-TTLS. Problem is when you finally understand that this is not scalable nor controllable like it should be, you will be in a position of having a large flat network (pppoe requires L2 connectivity between client and LNS) that will need to be routed to add secondary links (not redundant, just alternatives) and you will start seeing how pppoe falls on its face.
For your power strained towers, you can just make them a child of another tower and have the other tower host the required services. But be warned, this means if one goes offline, both go offline with no hope to figure out what happened remotely.
All epmp radios support MTU of 1700, the only radios I have that do not support an MTU of 1700 still support over 1600 so there has been no issues. For pppoe with no mtu adjustments you need an mtu of 1528, for MPLS you need 1540, for MPLS-TE you need 1596. BUT you can also limit to 1500 and use MSS-Adjust to fragment packets down to serviceable sizes at the cost of bridging and routing PPS.
There are many places in a WISP where you can use “crafted” solutions, but if you are looking to save $$ or time then you have already lost both!
I’m not stuck on PPPoE, actually EAP-TTLS seems like a better idea, but to even test that experiment I’d have to revamp tower sites and tear apart Cisco boxes (neither Mikrotik nor Juniper supports this), then go get licenses for each piece, so what do I have left besides tower-based PPPoE (or some auth) with a router that calls back to a core Radius server? I haven’t heard of many (or really any) operators doing 100% Cisco topology to get it to work, what’s everyone else using that’s working?
I’ve also had friends who say they have nightmares trying to get 1700 MTU sizes to work across multiple vendor backhauls, they say they wind up reducing all the MTU’s on the clients to make them fit on 1500, don’t know.
My friend suggested OSPF to get back to the core, which would provide redundant paths and quiet the network down, is that a good option?
For IGPs, there is little difference between ospf and is-is, so whichever you know best is which you should use. Simply having a routed network with each tower having its own router will enhance your networks abilites 10 fold over straight bridging.
Just by setting the max MTU to 1700 does not mean that your packets will be 1700 long. What it means is that you wont fragment a packet that is over 1500. It also means you will not have problems with MPLS or other overlay techniques that will improve your service when you add them.
Most ptp hardware supports MTUs larger than 1500. If you find one of your links has a smaller MTU capacity, then you set your network MTU based on the largest MTU that all of your devices can support.
Another question to ask yourself: do you need an authentication layer right now? For security you have the shared security key in each CPE radio. You have cnMaestro zero touch on the AP to force your device onboarding.
You can always add the authentication layer after you get your service up and running and your network routed and stable. Adding EAP-TTLS is as simple as uploading the certificates to each device, adding the VSA profiles to the database, setting your authentication details in each device and telling freeradius (can be any radius server) about the NAS at each tower. This can be done in stages without anybody knowing its happening until you force the AP to only use EAP-TTLS.
As for many 100% cisco operators? I am one. Got tired of fighting with things that didnt do what I need them to and making everything the same has helped with keeping config errors to a minimum, only one cli to know the cryptic error messages for. Licensing has never been an issue, If the cisco device you get does not have the entitlement (cisco speak for license), you simple activate the trial. We get datacenter pulls and they have yet to not have a sufficient entitlement.
I know some other Wisps that are using Netonix and other switches, some with Microtik routers per tower and some with a few towers per router. There are many ways to skin this cat, some are better than others, but the game here is to spend the money wisely and not piss off customers with down time.
I would like to have AAA defined per tower to quiet down the backhaul, with just a remote call to the core radius server for ease of centralized customer managment through a daloradius instance (thanks for the dalo tip 
)
I will look into the MTU size options. I assume my OSPF will carry a management VLAN to the AP/SM along with customer traffic just fine.
I guess I’ll see if Mikrotik will behave with some basic stuff like basic routing my subs back to core routers over OSPF, handling management VLAN, and doing a radius call back to the core router. I don’t trust Mikrotiks for anything major, but maybe these aren’t that major.
I understand the Cisco only comment, I can definitely see it, though we lean more toward Juniper, especially for core routers, and I don’t know why those don’t seem to support EAP-TTLS, that’s the major issue, and neither do Mikrotik, which are stupidly low power which is nice for solar sites. Juniper would need to go to MX level to do EAP-TTLS, which is massive overkill per tower. Later we might burn the Mikrotiks for being - well - Mikrotiks , but it might get me up and running for now?
rather than carry a vlan (wrong concept) you route your management traffic to the tower. The tower has the vlans that separate management traffic from client traffic. Your routing protocol will figure out how packets get from point a to b and if you need to, you can assign path priorities ect.
Daloradius is a nice stepping stone but is very unfriendly to people whom do not work in the AAA terminology. We use it to prime the database in a way that freeradius understands and then have our own web interface to modify the database. You can look at phpMyadmin for clues on how to do this.
Yes an MX would be like me placing a 7206vxr at a tower, doable but why?
Your routers do not need to do EAP-TTLS directly, just act as a NAS for the radius server to use as an authorized device. On a Cisco, there is very little that has to be configured to make this work. Just a bunch of AAA commands. There may be another advantage in that this also enables the Tacacs functions and allows for unified user ids without having a local user table (though you better have some local users listed just in case you cant get to the radius server!)
Dont knock your tiks until you’ve really used them. Personally, Im not a fan of them, but they are a good little product once you get to know them. Hence why the WISP community loves them so much. But I can get a good near eol cisco router from a datacent for the same price as a new tik and have 10x the power and functions in a format I know very well.
If you need help modifying a juniper to low power/Direct DC, PM me. And I will work with you directly.
