P10 8.1.4/8.1.5.1 NAT broken

Motorola released an advisory about a certain manufacturing run of P9 boards that had problems with 8.1.4. The problem was related to some piece of the OS that Motorola rewrote to avoid royalty payments (or something along these lines…) Apparently, Motorola forgot that certain P9 boards used some chip and didn’t write the right whatever blah blah blah.

The upshot of all this was that some ethernet ports, particularly those on various linksys, d-link, and other cheap routers couldn’t talk with the P9 boards, but ONLY in NAT mode…

So, now we are starting to get P10 boards with 8.1.4. Guess what, I have the same problem with many of the subscribers. They can use the connection after the SM reboots, but within 1-2 hours, they can’t talk to the board anymore. I am unclear exactly what is going on, other than the ethernet link itself still seems to be active, but their router can’t get DHCP from the Motorola SM.

Upgrading the P10 boards to 8.1.5.1 doesn’t fix anything, the behavior stays the same. Upgrading the APs to 8.1.5.1 doesn’t change the behavior, either. (Not that it would make sense for the APs to be part of the problem, as the SMs can talk fine to the APs the entire time.)

Now I have been moving all of these customers to a VLAN with NAT enabled on our side and with SM isolation enabled. This seems to be roughly equivalent, and so far the P10 boards seem to work fine with NAT disabled.

Last time I started to deploy 8.1.4, it was a disaster because most of my customers use NAT. Now I am just disabling NAT entirely and providing it on my side, or I could even provide public IPs (but I don’t really have enough public IP space at the moment to do this and none of the customers really care, the ones that do get their own subnet and VLAN) and it seems to work fine, so far.

The disaster was exacerbated by the fact that Motorola support kept denying that I had any problem and kept telling me I must be the one doing something wrong. (Hey, thanks, after buying hundreds of thousands of dollars of your gear, I like getting the most clueless, worthless tech support on the planet…)

What the hell is up with this? Why can Motorola not do NAT decently? Even when the NAT setup works fine under 7.2.9/7.3.6, it can’t NAT half the types of traffic that any common Linksys router can (like differentiating ESP streams by SPI for IPsec VPN users, or GRE traffic for PPTP VPN users, or many other non-TCP/non-UDP protocols that everyone else supports.)

I give up using NAT with Canopy anymore. It’s worthless. At this point I would really LOVE to talk to somebody on the development team. In a past life I wrote code for kernel-level networking software and worked with guys that wrote VHDL for FPGAs on T1/T3 boards. I have some idea of what happens at the lower levels, and I’d really like to see what’s happening at Motorola.

What about NAT on P10 hardware? Any problems there?

EDIT: NM I “read” the topic

mattmann72 wrote:
What about NAT on P10 hardware? Any problems there?


Look at the topic... :D

To answer the reply post, absolutely, yes, I have the problem with P10 hardware. That was the whole point of the post, that a very similar problem exists in P10 hardware that “did not” exist in P9 hardware, yet was later confirmed by Motorola. It’s really an 8.1.X issue, not a hardware issue, from what I can tell. The only solution with new SMs from Motorola is simply to NOT use NAT…

So, the problem that motorola acknowledged was an SM crash related to the ethernet code, not the NAT-mode-broken issue that i described.

I assumed that they were the same issue, but, maybe not…

In my experience, the port negotiation and the NAT problems are separate issues. We are frozen on 7.3.6 until NAT gets fixed. My whole network utilizes the NAT feature of the radios.

BAD NEWS:

Unfortunately, I also have to agree with the guys above: we bought around 25 pieces of new SMs and most of them are badly misbehaving doing NAT. Random and repeating unit restarts (the event log is filled with some stack dump), ethernet disconnects, packet loss, non-fuctional port forwarding, non-functional NAT for high ports.

These new SMs are totally useless and only discredit our company’s name in the customers’ eyes :-(((

All units are P10 SMs equipped with 8.1.4 firmware and firmware upgrade to 8.1.5 does not solve anything.

The only thing that probably helps is to disable NAT.

Disabling NAT fixes the problem entirely for us. (The problem is characterized as DHCP not working, the radio’s own internal NAT address becomes unpingable, and no IP traffic is passed to the wireless network. All this while I can still login to the wireless side) Sure, this would make me wonder if the “fix” actually solves all of these problems. The reality is that there is a real problem with NAT enabled on v8 and it really goes away when you disable it. We also had very occasional ethernet port lockups on NAT on v7 where the radio would become unpingable (again, only from the wired side) and DHCP would stop working, and rebooting the radio would fix it. With v8, rebooting the radio does not fix it.

I wouldn’t expect Motorola to fix this any time soon. To deal with the v8 NAT problem, I setup a separate vlan with SM isolation turned on and NAT at our ‘head end’ using OpenBSD with pf cranked to allow for tracking 250,000 states across four IP addresses. pf can track many different types of traffic, so this ends up being a MUCH BETTER solution than Canopy NAT which can’t even track ESP SPIs (IPsec VPN traffic.) Hopefully this solves the v7 ethernet port lockup as well. The fact that Motorola has this problem after all this time and has barely even acknowledged it does not look good for the future of the Canopy product line. I wish anyone at Motorola gave any sign of caring about this problem at all. At it is now I can barely get a response without trying to strongarm the support line. I shouldn’t have to spend so much time and effort to work with a supplier like this.

SM isolation really only works right if your switch will cooperate (and not send broadcast or unicast traffic between AP ports, which is an option on the CMM2). Or you can run each SM on its own broadcast domain (royal pain in the ass) or come up with some other hack (who knows)