Some of this may be vague, but I’m looking for known causes or if someone else has had this problem.
Our customers are calling in, from all across our network, saying they lose their connection to the internet. Signals between radios and tower are solid, with strong signals, low jitter and no re-regs.
We thought we found the problem, that our router was set for 4 hours to clear the ARP and the radios were all still set to the default, 25 minutes. We changed this throughout the APs, BHs, and SMs, setting them to 500 minutes.
All equipment is on 10.3.2.
We also have a Cable TV system with HSD using the same router, switch, DADNS etc and the Cable Internet customers have no problems.
I have three backhauls pulling data in from three different directions, coveing our county. All three BHs go into a CMM and the CMM goes into the switch. We see no CRC errors at the switch and I have replaced the cable connecting the two.
Any thoughts or ideas???
Can you draw up a quick network layout?..Could be a few things.
I’ll try to put something together but I’ll tell you it is a flat network. Public IP goes to the customer via DHCP. Filtering SMB, BootP Server, and IPV4 Multicast. QOS is Broadcast enabled with a data rate set to 15.
We thought our problem was ARP related. If we cleared the ARP Cache on the router, everything would come back up but go down again a little later. We changed the ARP Cache from the Cisco default of 4 hours to 5 minutes and it seems to almost completely fixed the problem. We then found the suggestions to set the Bridge Entry Timeout on all radios, BH, AP, SMs to 500 minutes (8 hours 20mins). We did not enable anything else or change anything else.
When we set the Router ARP Cache timeout back to 4 hours, it started happening again. I turned it back down to 5 minutes and all seems calm.
This just started within the last month or so. We’ve been operating for a couple years with no problems . (Like this)
Are the SMs in bridge mode? Have you tried a packet capture to see what the router is seeing? What sort of router is it?
salad wrote: Are the SMs in bridge mode? Have you tried a packet capture to see what the router is seeing? What sort of router is it?
They are in bridge mode. All MAC addresses and IPs are associated to the customers equipment, the SM is basically invisible. On the APs, we have enabled the DHCP Relay Agent with insert only option 82 so we can track down users. Even though I've been to the Canopy School, I still feelt there is a lot I don't know about the various settings and options on the radios. I hope we haven't overlooked something.
The router is a Cisco 7200 VXR with G2 engine.
I have not ran a packet capture. What would you recommend I use to perform it?
APs function as a DHCP relay in 10.3.2? Interesting… I’ve never encountered anything router-like in the APs (though I’m only on 9.5 still).
If you have a decent switch there try and set up a span/mirror port and look for ARP with Wireshark that’s making it to the 7200.
Another idea - what is your DHCP lease period set to? Have you tried setting reservations?
salad wrote: APs function as a DHCP relay in 10.3.2? Interesting... I've never encountered anything router-like in the APs (though I'm only on 9.5 still).
If you have a decent switch there try and set up a span/mirror port and look for ARP with Wireshark that's making it to the 7200.
Another idea - what is your DHCP lease period set to? Have you tried setting reservations?
The APs function as a DHCP relay but you have go to version 10.3.2. 10.3.2 fixed a problem with the DHCP Relay that would stop the customer from receiving their IP address. In 10.3.2, they fixed it and it seems to work well. Now, when we receive copyright violations, we can actually find WHO has the IP address they're making a fuss about.
I'll look into the switch. It's a Cisco Catalyst 2960G. We are a small cable system and outsource much of out engineer/technical support. Much of this is Greek to me and Wireshark, I don't even understand what I'm looking at when I see it.
Can you plug a PC with a Packet Analyzer into a CMM Micro port and see all the traffic coming in? Everthing comes into a CMM micro with port 1 going to the switch.
BTW, I appreciate your correspondence.
Very interesting - I thought that the Canopy APs were pretty much bridge-only devices.
2960 will definitely support that so you can see everything anything is doing. The big nasty guide is here: http://www.cisco.com/en/US/docs/switche … wspan.html I believe that firmware 3.0 introduces support on the CMM Micro for a mirror port for troubleshooting.
What you’d be looking for in Wireshark is anything nasty like IP conflicts, gratuitous ARP, or things of that nature. A pretty decent explanation of all things ARP can be found at http://en.wikipedia.org/wiki/Address_Re … ouncements, but the relevant section you’d want to check out is that bit on announcements and basically how it’s supposed to work.
Are you running Prizm by any chance? If so it might aid you in troubleshooting, though I doubt there would be any loss of traffic to the management IPs of the radios.
We were plagued with these types of problems for years and they way I fixed it was by turning on NAT in the subscribers. This eliminated many things such as broadcast storms, backwards routers, customers hacking network, etc. 90% of our network is now natted. This also makes it very easy to track down DHCP customers as the SM is now the device with the public IP. We have some customers who we do not nat for vps etc, but these are generally businesses who don’t jack with their network. I would say nearly all of our network problems of the past came from residential customers. Nat em and forget em.
jmaysww wrote: We were plagued with these types of problems for years and they way I fixed it was by turning on NAT in the subscribers. This eliminated many things such as broadcast storms, backwards routers, customers hacking network, etc. 90% of our network is now natted. This also makes it very easy to track down DHCP customers as the SM is now the device with the public IP. We have some customers who we do not nat for vps etc, but these are generally businesses who don't jack with their network. I would say nearly all of our network problems of the past came from residential customers. Nat em and forget em.
My experience with Moto Canopy NAT was not good. Did Moto fix the problem and the NAT feature works now? I was always aggrevated that a $30 router from Walmart could function better than a $$$ Canopy SM.
If it does seen to be functioning GOOD, please refresh my memory of the settings I should put in so I don't overlook anything.
We’re using 10.3.2 but have been natting since 9.4. It works pretty well. If you plug a switch into the other end and several computers I have seen issues where if 1 computer is running max download the others will run a bit slow. The processing is not quite a good as a $30 router. However the majority of users have a wireless router on the other side anyways and while I’ve noticed the issue we have never had any calls about it.
All you have to do is enable it and your done if you are using DHCP. By default the lan IP is 169.254.1.1 so we change that to a 172.16.0.1 just for preference. Just dont use a range that routers will want to use or your customers will have conflicts when they put em in.
How about the NAT Protocol parameters? I was under the impression that your ARP Timeout should be greater that whats on the router but I’m seeing that it only goes up to 30 minutes.