We have been having some odd issues lately with DHCP and our Canopy network. Most of our residential clients are running NAT on the radios. We statically assign them a management IP (10.1.0.0/16) and then have our DHCP server assign them addresses from our pool (We have 4 C class subnets). As of late, we are having an issue where the radios are getting an address of 0.0.0.0, while all the other info is obtained correctly (Subnet Mask, DNS servers, DHCP server, and the Default Gateway). If we reboot the radio then it will usually obtain a proper address, however sometimes it does not.
This problem is happening on all types of equipment. We are running a combination of 900 & 2.4’s, and everything from P7-P10. 95% of the radios that can run hardware scheduling are running software version 9, while the rest are running a mix of anything between 7.0.7 and 8.2.7 (They probably weren’t powered up, or installed the last time I did a software update). We have some 2.4 AP’s that are only capable of software scheduling, so the SM’s on them are running either 7.0.7, 7.2.9, or 7.3.6.
At first we suspected the DHCP server was the culprit. It was an old Dell workstation running Windows Server 2000 SP4 using the built in DHCP server. It has been in operation since we started up around 4 years ago. I was getting sick of Win2k anyways, so on Christmas Eve I pulled another workstation out of the box and installed Server 2003 on it, and got it setup the way it was before. I pulled the other machine off the rack and threw this new one in its place. Then I issued a mass reboot of all radios using the tool in CNUT and after a few dozen radios came back with proper addresses I took off for the day thinking the issue was resolved.
On Boxing Day I came in to the NOC to check on things, only to discover that I had about 50 voice-mails. So I rebooted all the customers that had called in, and called it a day. Since then there have been another 75 or so that I’ve had to reboot to get an IP, and everyday there are more and more. There seems to be no rhyme or reason to it either, however is doesn’t seem to be affecting anything but NAT enabled Canopy radios. Routers and computers seem to be getting the proper addresses from the server without a hitch.
I’m currently debating trying a different DHCP server software, or possibly ditching DHCP all together and assigning everyone a static IP. However I would like to leave both of these options as an absolute last resort. I couldn’t imagine having to go into 700 radios and giving them statics, and then updating the client records. It would be a nightmare.
If you need any more info, let me know.
Anyways, I’m open to suggestions. Thanks in advance.
Still having these issues. Any thoughts?
Excessive broadcast traffic drowning out dhcp requests?
I agree. Make sure you have the Broadcast limit (on QOS tab) enabled and set to about 5kbps on ALL SMs. You may find a couple that have to set a little higher (10 kps or a bit more) if they dont pickup the address properly.
Also, look at your DHCP logs for routers that ask constantly for DHCP registrations. Upgrade their firmware, or get rid of them completely. that definately helps lower your DHCP “noise floor”
-Paul
Paul McCall
PDMNet Canopy Repairs
658 Old Dixie Highway
Vero Beach, FL 32962
772-564-6800
RyanR wrote: Excessive broadcast traffic drowning out dhcp requests?
just thinking:
How could be the broadcast traffic so high if the SMs have NAT enabled?
Massimo
First of all, I would like to thank you guys for the responses.
We have Wireshark running at the moment capturing all DHCP traffic. Unfortunately we haven’t had this happen the entire time we’ve been running Wireshark. We even re-booted a couple of heavily loaded AP’s and then checked every SM to ensure that the had a valid IP, and all was good. However we did find an SM that seemed to have a pretty flakey connection. It was sending out DHCP requests every 15-20 seconds. We assigned it a static IP, and since then to my knowledge we havent had any issues. I cannot see how one SM could cause our entire network to goto crap, but I suppose it’s possible. I’ll keep you all updated on our findings.
paulchops wrote: I agree. Make sure you have the Broadcast limit (on QOS tab) enabled and set to about 5kbps on ALL SMs. You may find a couple that have to set a little higher (10 kps or a bit more) if they dont pickup the address properly.
Also, look at your DHCP logs for routers that ask constantly for DHCP registrations. Upgrade their firmware, or get rid of them completely. that definately helps lower your DHCP "noise floor"
-Paul
Paul McCall
PDMNet Canopy Repairs
658 Old Dixie Highway
Vero Beach, FL 32962
772-564-6800
I have a question about this setting. If that option is set to Disabled, as it is on all of our SM's, will they essentially broadcast however much traffic they want?
yes… you are correct. Definately need to limit that when using DHCP.
Paul
BigTrumpet wrote: [quote="RyanR":2ev1k7zy]Excessive broadcast traffic drowning out dhcp requests?
just thinking:
How could be the broadcast traffic so high if the SMs have NAT enabled?
Massimo[/quote:2ev1k7zy]
Even with nat enabled your canopy can be requesting dhcp leases excessively as well. Wireshark would be able to pick up these requests but it looks like you already figured out the problem :)
We use NAT and the SMs get their WAN address assigned via DHCP.
but, yes you are thinking correct. The only broadcast traffic emanating from the SM should be minimal. Normal layer 2ish type stuff
Paul
I have the exact same problem on two different networks. One network has a cisco router doing DHCP and only about 100 SMs on the whole network. The broadcast traffic is very low. We are moving away from DHCP because of this.
We think that we found the solution to our problem. Using Wireshark we discovered a very high retransmission rate coming form one of our AP’s. Upon closer inspection, this old AP was running through a Motorola grounding box. We removed the grounding box from the equation and our network seems to have improved significantly. This spring we intend on replacing all of our tower cat5 with STP gel filled cable instead of the UTP outdoor rated cable we have been using (why it was used over STP I’m not sure) Should be a fun experience replacing about 100 individual cables across 18 different towers.