Hi. I need some help please. :) I've searched the forums and didn't find any specific place, but is there a place where networking/switches/routing questions are covered?
Here is our situation... our network used to be all StarOS. For those who don't know, StarOS is a bit like MikroTik, in as much as the typical access point is a multi-port board, with mutiple radios, multiple ethernet, and with internal routing, DHCP, QOS, etc, etc. A single StarOS board may only have 1 ethernet and 1 radio card, but they may also have 3 ethernet and 4 radio cards.
So - on our network, with StarOS, we basically have each access point or backhaul connected to each other, and internally - we can do all the routing internally. So, it's not uncommon for us to have a 4-radio board, where we use 1 of the radios for a backaul, 1 of them for a 'forehaul', and the other 2 radios can be customer-facing PMP access points. On some of our smaller capacity towers, this is all we need and there is no switch, nothing more than a single POE in an enclosure and a single ethernet cable to the board. Likewise, on other towers, we have 2 or 3 boards (maybe supporting 8 or 10 radio cards) and that's all that we need for all the backhauls, and all the PMP Sectors. And we do all the routing, DHCP, QOS, etc, etc all internally in those boards.
So, this is kinda cool - BUT it also means that when we transition our network to ePMP, it's causing us to have to rethink how we interconnect everything. Basically, everywhere on our network, we either just connect the boards back-to-back (connecting directly Ethernet-to-Ethernet) or, we connect all the boards at a tower into a simple switch, since the Ethernet ports themselves have the IP addressing and the boards know how to route.
So - my question is: What are people using at the towers equipment in the shed/enclosure? Is it MikroTik or is it Netonix or UBNT's EdgeSwitch... or what are people doing?
AND - is there a forum on here to discuss such a thing? I went to a Cambium Certification Course, but none of that was on the docket there...
We use a combination of Packetflux to provide power and GPS sync to ePMP and PMP450 radios, and then depending on the number of ports needed, we'll use just a Mikrotik cloud core router... and if more then 10 ports are needed, we'll use a Mikrotik plus a managed Cisco switch to get us any extra ports we need for a large site. Why would we bother using a PacketFlux sync injector when ePMP 1000/2000 already has built in GPS? We use it to provide backup in case the onboard GPS fails on the 1000/2000, and it also provides a common timing platform and power at gigabit speeds for all ePMP/PXP450/PXP450i/PMP450m radios.
You can use Netonix although you will still need a router if you're routed at each site, and the Netonix doesn't get you GPS sync. So again, we've found an easier/cheaper combo to use PacketFlux + Mikrotik
Avoid UBNT toughswitches at all cost... they tend to have an extremely high failure rate and many many people (including myself) have switch to Netonix.
Cambium tends to only recommend and offer training for only core products that they produce, which is a common practice among hardware manufacturers... that's why you see little to no information on PacketFlux, LastMile CTM's, Mikrotik, Netonix, etc. Although PacketFlux and LastMile work very very closely with Cambium to ensure their products are compatible. Your best bet to find out more information is join the AFMUG mailing list, or post questions to the WISP Talk FaceBook group.
That's all OK by me. Other than some AirFiber24's (which have been really good) we don't much like to put any Ubiquiti stuff on the network. We're trying to be all Cambium, but of course they don't make as much stuff.
So, the attractive thing about Netonix is the ping rebooting features. Does PacketFlux have such a feature, to be able to reboot a radio when it's not responding to a ping? We have been using Digital-Loggers Web Power Bar and their DIN power relay... and those are a lifesaver, but they also add another variable into the mix too. It would be great to have something like the Netonix which had routing.
However, If we need to learn MicroTik, then that's what we need to do. We had chose StarOS years ago because it was pretty simple (and the expense of some features) compared to MTik, but we really need to cut-the-cord and figure out the best way to interconnect our ePMP's.
We also need to figure out how to have redundant backhaul links. With StarOS, they have OLSR and OSPF routing, although those have been weirdful for us. Regardless, that's another thing we need to figure - how to have redundancy on the backhauls with AirFiber and/or Cambium backhaul links.
The PacketFlux's have basic remote network management and you can remotely turn on/off/cycle individual ports, look at voltages, etc. I don't believe it has a watchdog function, although IMHO I don't really want anything that does that because it can cause lots of problems... e.g. OSPF incidents, which cause some packet loss during the change, which in turn might trip a watchdog, and cause a power cycle of a BH radio that then causes further interruption of service. Firmware updates that take longer then expected which trigger a watch dog to power cycle and brick the unit, etc. If you're using a hardware watchdog to power cycle equipment, then you have bigger problems that you should be trying to address first.
The Netonix's provide power and switching... but no routing/L3 functionality. So you're not going to be able to do OSPF... you'll still need a router, like Mikrotik to do that. We typically have 2 or more paths to each site and use OSPF on Mikrotik to accomplish this. Each site is fully routed, we got away from a flat/bridge network years ago.
We do it just like Eric Ozrelic, packetflux and mikrotik switches. We have also always used the McCownTech: DIN Rail Chassis Modular Suppressors. They hold 4 modules each so they are a nice fit for the 4 port packetflux sync injectors. Although we have repeatedly had trouble with them and the ePMP rados and I'm about to give up on them. I have two towers right now that have no surge suppression on the ePMP gear because we had to bypass the McCownTech suppressors due to various ethernet problems.
OK, well MikroTik is what we'll need to learn then. As I said, the bulk of our existing network the radios have routing and multiple ethernet's in each of them, so we have a totally routed network and not flat bridged - but each radio board is a member of that routed network.
WIth ePMP Radios, we'll have to switch our logic to doing the routing in a MikroTik switch - so that's what we need to learn I gather.
So - I'm putting up a new 8 sector tower (4x ePMP1000 @ 2.4 Ghz and 4x ePMP2000 @ 5Ghz) so what would you guys recommend (specific model numbers would be great) for the switch, GPS/POE, etc?
@Eric Ozrelic wrote:If you're using a hardware watchdog to power cycle equipment, then you have bigger problems that you should be trying to address first.
LOL, yes, that's why we are switching everything to Cambium.
@Eric Ozrelic wrote:The Netonix's provide power and switching... but no routing/L3 functionality. So you're not going to be able to do OSPF... you'll still need a router, like Mikrotik to do that. We typically have 2 or more paths to each site and use OSPF on Mikrotik to accomplish this. Each site is fully routed, we got away from a flat/bridge network years ago.
OK, that's good info. I suspected that the Netonix was just a POE switch and not a router. I have heard people on here and other boards swear by them, but yes - we want to have increased redundancy everywhere. We have it with our old (non-Cambium) gear now, but with manual routing entries. So, we're able to route around problems without driving, but that still ties someone to a keyboard somewhere... which means 7 days a week on call and that's getting old after 20+ years of that.
So, yes, we'll have to learn MikroTik. We have lots of routing experience, so addressing and netmasks and whatnot are not an issue. But, MikroTik's interface will be new, and that's what I'm asking for recommendations on. If you can suggest which MikroTik router/switches to buy, and if you can be help to on getting over the MTik GUI hump with OSPF, that would be appreciated. :)
OK - I'm diving back into this... trying to switch our routing over to MikroTik, and looking for some guidance. :)
I know MT has it's own forums, but I was hoping to get help from this community if possible... because everyone here has a more similar total setup to us - with Cambium AP/SMs and with the same wants and needs as we do, and with the same little idiosyncrasies that may be present with Cambium/MTik.
So, as mentioned previously - almost all our our routing is all manual/static routes. We have over 100 access points, all manually routed. That works well - until it doesn't work. We have no automatic redundancy.. we have to log in an manually route around a problem. So - I'm mainly wanting to get help with MikroTik on adding redundancy, failover, bonding - I guess bonding and/or OSPF, etc. I know lots about RF and routing/subnets/etc (25 years worth of that) but next to nothing about MTik. So... be gentle please. :)