I have around 15 CMMs deployed. A few days ago all CMMs froze and nothing we did would get them to work until we switched the power off then back on again and everything went back to normal. This happened to all CMMs at the same time. It happened 4 times in the past 7-days. Fortunately we have security guards at each site so we call them on phone to power off/on. We found nothing in the log. Can’t be earthing cause it’s happening everywhere. We did not change the firmware version either. Very strange!!! Any clues???
if it happens to all the units at the same time, perhaps there is some malicious traffic on your network directed at their management IPs?
We use OSSIM for network management. I will flag all traffic to/from the management IP of the CMMs to see what comes up in the next 1-2 days. Thanks!
you using SNMP monitoring?
yes, greg is using SNMP monitoring? We are using Prizm
Gregory,
has it happened again?
If so, go through the log files and see if there is a common warming or error.
Jerry
No it did not happen again since I posted this message on the forum. I hate it when this happens. Not that I want the problem to happen again 
But it makes me very uncomfortable when a problem just “goes away” without knowing what caused it because I can never tell when it will happen again. Anyway there’s nothing in the log files. Absolutely nothing!
Since this happened I had several meetings with all concerned in the Company and the general consensus is that we have to move from a flat network to a more segmented one using routers, firewalls, etc. I have tasked Alex to coordinate this project and to come up with a two phased plan.
Phase-1 (next 3-months): Improve network security on a flat network. Enable protocol filters (IPv4, SNMP, BootP, etc.), enable SM isolation, create VLANs, etc
Phase-2: (next 18-24 months): re-design network into smaller subnets separated by routers at each site
I can’t wait to start with phase-2 but I have to take my time because of the cost involved. We now have 33 sites (35 by the end of November 07) in 7 Regions. The cost of 35 routers (not to mention at least 2-3 spare routers) is huge.
All of you guys who have contributed to this forum have done a great job.
Thanks
Greg
Gregory,
I agree… Those mystery bugs are the ones that keep me up at night.
I have been pleasantly surprised by the performance of the MikroTik. At 220US each they are a very very good value.
MikroTik does do well- especially the new boards that will have more power to them. I think that it is a huge consideration for those locations that have no building to put the equipment it.
I’m not a networking person and this question may sound silly but I must ask it
How do you connect 6 APs and 2 BHs at one site to a router (or microTIK)? Connect the Canopy devices to a switch and the switch to the router? Or I should have one network interface in the router for each Canopy device?
Silly question???
What are the BH’s?
There are many ways that you can do that. Many depend on what you are using and what you might need. if you need timing, then usually you would use a CMM and have one port go to the router. (with 8 on there, you will have to find an alternative source of timing for one of the units) unless your BHs are slaves then there’s no problem.
You could have the cmm (with all APs on it) going to the router on one port, and then each BH to it’s own port. again, this depends on what the router is going to be for. If you are not using a CMM, then sure, all APs could be on a switch then to a router, or a layer 3 switch (don’t I wish).
Sorry if that’s all confusing- it’s just that there are many ways that it can be implamented. I think the cmm/switch, to a router with BHs on their own port is the easiest. Then, a layer 3 switch with vlans and routing would possibly be the most ideal.
For Jerry: I am using Canopy 20Mbps and in some hops the Canopy/Orthogon 30Mbps
For LP: I have CMMs at all sites. The example I gave was not very accurate because most sites have max 4-6 devices (combination of APs and BHs). I have only 2 sites, each site has 2 CMMs & 8+ Canopy devices connected.
Timing is important (I learned that from Jerry) therefore I guess all devices (APs & BHs) on all sites must connect to the CMM for timing then we’ll drop a cable from one port on the CMM to the router.
In this case how do I prevent on the same site device to device communication when all devices are connected to the same CMM switch? It would be nice to control/manage traffic on a per device/site as well as site/site basis.
The CMM Micros have a vlan selection where you can pick your uplink ports (ports that all other ports can comunicate with) and other ports will be “invisible” to them. in most cases, you just select your BH as the uplink port and all traffic will go through that and not to the others. You can also configure them yourself but for simple restrictions, that will work great. Otherwise, you would have to have your BH connected to your router seperately making the router the “middle man”- and to do that you would have to make your BHs slaves (so they don’t need the CMM for timing as they would receive the timing from the master).
The reason I asked about the BH’s is that BH20’s need timing, Orthos don’t.
I don’t recall, are your units the CMMmicro (managed), or are they the older CMM (unmanaged)?
If they are the CMMmicros, with V2.2 software you can enable Port Based VLAN so that each AP and BH feeding remote towers can only talk to one port (uplink) on the CMMmicro. That uplink port connects to one Ethernet port on the router.
The BH to transport traffic to the Internet connects to the second port on the Router. If the BH connected to the internet side needs timing, you can take the timing off one of the AP’s RJ11.
All CMMs are micro v.2.2.
We plan to test VLAN in the lab to better understand it, its advantages and disadvantages. We are unsure at this stage how to implement it on our network in terms of how many VLANs to create. Should we create one VLAN per group of client (i.e. Internet clients on 1 VLAN, WAN/VPN clients on another)? Should we create one VLAN per site? Should we create multiple VLANs per site?
Clearly we lack the knowledge and expertise but we plan to acquire both in the next few weeks. We set up a project team to handle this, created a long list of tasks which we plan to cross out one after the other until our network is 99.9999999% under control
We finally found the reason why all the CMMs on greg’s network was “freezing” and why the entire network went down.
We ran vulnerability checks using nessus for the network. We found out that whenever we start the vulnerability scans, the entire network went down. We compared the dates we started the scans and the dates we had complete network outages and they tallied.
DATE OUTAGES VULNERABILITY SCANS
26-10-2007 8:46am
27-10-2007 10:58am - 11:25am 10:46am
29-10-2007 8:15am - 9:05am 7:44am
05-11-2007 6:55pm - 8:05pm 6:42pm
The million dollar question is why would my radio devices “freak out” under a vulnerability scan?
alexdehaini,
Now that you mention it- I have crashed my whole network (wireless side) by doing a ping sweep, or auto-discovery proceedure. funny how that works…
As for the vlans… if you do port based vlan (what the cmm micro does unless you actually select to tag the traffic) you are just segregating them. By this I mean that if you have a cmm and set only your BH to the internet as the uplink, then all your traffic will only go to that port. If port 1 was your uplink then it would be like having port 1 and 2 on a switch, port 1 and 3 on a switch and so-on. So 2 and 3 would not be able to pass traffic. Now, if you tag them, it’s the same basic thing only you can then have port 1 on a cmm be on the same vlan as port 8 on a cmm 20 miles away (they would virtually be on the same switch). With tagging, however, it gets more complicated…
I would refer you to
http://net21.ucdavis.edu/newvlan.htm
or
http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/sw_ntman/vlandir/vdir1gsg/overvw.pdf
It is shocking that a ping sweep or a vulnerability scan will take out an entire radio network. This is the reply I got from moto
"Alex -
As I’m sure you know, we do not support third party applications. However,
based on your description, it sounds like your “scanner” appears to the
radio like a DOS attack. We would recommend that you not use this tool on
the radios"
WOW!
I was kinda figuring it was something along those lines. It was an interesting lesson to learn :roll: . Too bad it does that though.