450s chewing up the spectrum..

So I've hesitated posting any of this as all I have is anecdotal evidence, but it's overwhelming..
We collocate with a large WISP who about a week before Thanksgiving upgraded to 450m. Immediately our 2000s did a nose dive. Unfortunately, no one said a word to us until the GM at the tank told me they had upgraded. (Our holidays sucked!)

We contacted the other WISP, they shared their frequencies and we got a hold of a Cambium engineer. After weeks of working with us, he figuratively walked away. Nothing worked.

We’ve got the spread on the frequencies. They’re saying they’re on channel 20, but between 7 PM and 10 PM our signals drop to next to nothing.

I’m avoiding specifics as they really don’t matter, this is an issue between the 450m and ePMPs. (We’re only running 2000s, standard 90 degrees, with two sides “stacked”.) They can’t sync either.

A customer pulling 70 megs during the day, crashes to 3 megs at night and right at 10 PM shoots back up to around 20.. by AM, it’s back to 70.

We’ve done all the settings the Cambium expert has suggested. We’ve got about 10 feet horizontal spacing, still a problem. We’re loosing customers left and right. At this point, I’m seriously considering swapping out the panels and going to a dozen “horns” and just flooding them out.

If anyone has any insight or suggestions, please! I'm all ears.

Thank you.
J.

Are you running on the same frequencies? If so there's not much you're going to be able to do, even if you were both running 450. 

Nope.
We tried to sync with them, didn't work.
And we've got about 20-40 channels seperation between them.

Signal strengths will still show pretty good, 13-15s but there's just no bandwidth getting thru.

It makes abosultely no logical sense.

J.

Sounds pretty strange. Can you post some SA's?

Ya. I'll need to wait till tomorrow night for full effect.

J.

What firmware are you running on your AP's and have you used Cambium's 450 / ePMP sync tool?

ALSO, I assume that you're using GPS sync and they are too? What source(s) are you both getting sync from?

When you say "They can’t sync either." what do you mean? If they cannot even sync their own AP's or are not using (or even getting) a sync source this is cause for alarm.

3.5.6 now. (We've upgraded every offering.)

We did when we tried syncing with them. Turned up distance to 20 miles, set to 50/50, blah, blah, blah.

There in may lie the problem.
They're running CM 5 (I believe it's 5. Latest one.) and we're running external ePMP GPS. (Cable)
We also tried syncing without internal GPS, (no cable, per engeneer) didn't do a thing.

We were told the CM 5 might be the issue, it's throwing off the ePMPs.

But even at that, I doubt that would explain where our bandwidth goes at night.


For example, tonight, AP "6" was set at 5750 channel 20, They're on 5815, channel 20.
Customer "Jones" pulling 40 megs today, tonight, 3 megs, signals still hovering around 14-13. -67.
I started messing with the AP, set to 40 (very dangerious) settled on 5615, still only pulling 10 megs, ping around 100-200. 5615 was "open"


@Jackhmf wrote:

...But even at that, I doubt that would explain where our bandwidth goes at night.


For example, tonight, AP "6" was set at 5750 channel 20, They're on 5815, channel 20.
Customer "Jones" pulling 40 megs today, tonight, 3 megs, signals still hovering around 14-13. -67.


There is a lot of missing information needed to really help here, but one thing to think about....

You say in a previous post that your bandwidth drops drastically from 7pm-10pm. You also say in the quote above that the if the CM5 (sync issue) is the cause it does not explain where your bandwidth goes. Actually it does, 7pm-10pm is prime time internet useage. If the other WISP is interfering with you, the interference is going to be heaviest at these times.

Also, you list an AP 6 above? How many AP's do you have at this location? What frequencies? And are all of them synced or not synced? If they are all synced, are they all synced the same, with the same settings? If not, you could be a lot of your interference. 

1 Like

I assumed when I said we've tried everything, I'd be taken at my word.. This isn't a simple matter of us not knowing how to sync our radios. As explained, we've had a Cambium engineer engaged and we did the sync thing. Didn't work either.
And yes, I believe I explained, we have 6 APs, standard 90 degree deployment with 2 sides "stacked". So that's 0, 90a, 90b, 180, 240a, 240b. (Sorry, 0 is actually 3 30 degree cones.) I'm even getting lost in this mess. :)

What I believe the more relevant question is, are the "450ms" chewing up the frequencies during heavy usage? I've done a  spectrum analysis, and during the day the spectrum looks pretty clean, come night, almost every thing is full. At least as far as the ePMPs saturated are concerned.
Could it be a CNM 5 issue? Could it be the way the ePMPs read the spectrum?
Funny thing? Our cheap junk UBNT M5s could care less. They don't seem effected by this. We dumped our UBNT gear for the ePMPs so we only have a handfull of customers left on them.

your posts feel more like a rant vs looking for help.

the details are what's needed and you commented intentionally leaving them out, we'd all like to help, but we need information to do so.  

the 2 platforms can be synced together.   guessing and trying things will never work. your talking about thousands of possible combinations.     Attempting to flood out a competitor will end up causing you to be the person out in the long run.    EPMP2000 attempting to drown out a 450M isn't going to go well not to mention you'd be breaking the law. 

back to your problem. 

on your competitor,   he shared his sync info with you, did you do as Eric suggested and use the sync tool?   unfortunately for you, most of the tweaking needs to be done on the 450, the epmp 2000 doesn't have a lot of sync options.    you can mix cmm5 timing and puck timing. you don't need to share resources. 

next, the epmp2000 and the 450 both have filtered in and outputs.  so likely the "noise" your getting is actually on channel rather than co-band.   I'd dare say as well if you're getting noise, it's showing your uplink modulations being completely tanked vs your downlink modulations looking good.    that can be seriously improved by adding the beam steering antenna. horns may or may not fix it.  a lot of variables there.   throwing different types of antennas at a problem doesn't necessarily clean out your noise. it depends on where it's coming from and what the needs are to fix it.  if the source of noise is 10 degrees away from your sub, and same power level as your customer, it will not help. a BSA will give you some improvement. vs if the noise is 50 degrees away, a horn will do a better job. but you need to truly identify the noise before you can determine that.

if you truly are running 20 channels (100mhz apart) they are not the cause of your headache, and the timing is nearly not relevant.  I've got EPMP2000 APs sitting with an AF5 that runs nearly full bore every night just 2' away facing the same direction only 40mhz off, not a bother in the slightest.   10 feet of the side of the 450 will put you well outside of the full power center of that radio.  

as mentioned above, the noise will increase during peak time because of usage increases. this is something we all face, and we all have to mitigate.   you can test this theory on the bench if you'd like.   setup an AP and SM at the weakest possible signal you'd have in the field realistically.  get them locked at say -65.  with a 20mhz channel.  let them the link and watch.   now set a router between them on the same channel and width connect a device, but don't have it pass any traffic. you will notice some modulation bouncing, nothing bad.   now download a file, you'll see it tank and possibly de-register. 

next possibility is your simply saturating your frame times.   during peak, what is your frame utilization look like?  (monitor > performance, the bottom of the page)    if you're sitting at 90% or better, you need to add capacity, it's just full.    

if you can post screenshots of your wireless tab showing your RSSIs and your MCS states, we can help you resolve this problem.     if you've got an inactive AP you can post a spectrum scan from the same direction, that would be of extreme benefit.  or if you'd be willing to take an AP offline and the SA scan during the problem period, that would also be helpful in solving this issue. 

another possibility altogether is your backhaul being bothered.    what are you using to feed this site?  is it a regular af5x?  they don't do well with extreme noise right on them.     Is it in a different band altogether?  or a different radio? have you validated that it is not slowing down or being overwhelmed at peak?

what are your switches saying connected to these radios?  are you getting crazy pause frame counting? so many peices to a puzzle like this.  making an assumption and sticking to it can stop you from finding the source of the actual problem. 

6 Likes

I'm not aware of any "law" that says I cannot take down 2 panels and replace them with 3 horns.
Perpahs you can site it for me? I'm simply trying to provide service to exsisting customers and stop bleeding current ones. No law against that, is there?

I'm not "ranting", quite the contrary, I'm seeking input. I've sat on this problem for six months, trying to find workarounds and solutions and pulling my hair out, while loosing customers. This was my last attempt. I hate going to forums because no one ever seems to believe the poster when they say, "Hey.. I've tried all the standard stuff.. "
And "yes" the 450s do chew up the spectrum, simply by how the packets are encrypted, if anything.. 
But I doubt I'm telling you anything you don't already know.

That being said, you have made a vailid point which I'm now going to look at. The first being our total "frame time used" on one of the APs, the one we're having issues with, was 94.7%. (It's running in the high 90s.) That would explain it.. hadn't even thought of that, as we have less than 50 subs on that AP.

We're not "feeding" this tower. We have fiber at it.

Thank you.

You are putting nearly 50 subs on a 20mhz AP ?  I see why customer is dropping from 70mbps during the day to 3mbps at night. That AP is is only capable of around 80mbps total download under perfect conditions. We are putting less than 20 subs per 20Mhz AP as many of our customers are heavy streamers. This doesn't sound like interference, it sounds like an overloaded AP.

If you have that many customers then yes some 30 degree horns would be your best bet...without disturbing other WISP. 

" take down 2 panels and replace them with 3 horns."

That will probably help as far as splitting the number of users between 3 ap's instead of 2. Hopefully you have enough free spectrum (channels) for the increased number of AP's you will have.

>Perpahs you can site it for me? I'm simply trying to provide service to exsisting customers and stop bleeding current ones. No law against that, is there?

Not sure if they would have any recourse if they could prove you were doing it with the intent/hope to wreck their service (possibly if they post/read these forums). However it sounds like they have tried to work with you so it seems they are "good neighbors" and if you start doing things that you know will cause them grief and won't work with them then that makes you the "bad neighbor".  Especially if it turns out your own network was just overloaded and they wasn't interfering with your equipment.


" I hate going to forums because no one ever seems to believe the poster when they say, "Hey.. I've tried all the standard stuff.. " "

While I also find it annoying when people ask me if I did the obvious I understand that there is no way for them to know my level of experience and there have been times when someone ask me if I did obvious thing X and instantly my brain when "DOH ! OMG how did I forget to try/do/look at that !"  So , yeah, annoying but that's how it has to work.

" That being said, you have made a vailid point which I'm now going to look at. The first being our total "frame time used" on one of the APs, the one we're having issues with, was 94.7%. (It's running in the high 90s.) That would explain it.. hadn't even thought of that, as we have less than 50 subs on that AP."


See, looking at your own bandwidth/AP being overload is what many would consider "trying the standard stuff" . But we all do this, sometimes a problem starts at the same time something unrelated happens and our brain just assume A+B=X and doesn't even consider other options.

>We're not "feeding" this tower. We have fiber at it.

Then you are feeding the tower with fiber. That fiber could be 10Mbps or 10Gbps , just because it is fiber doesn't mean it can't be the bottleneck. 

3 Likes

+1. 50 people on a 20MHz channel is pushing it with epmp. 

Don't suppose you want to forklift your epmp to 450m? Maybe put a 450m up and then migrate certain problem customers over to the new AP. 

What's your traffic numbers look like on the AP during the problem times?

Here is a view from a mimosa a5 real time spectrum analyzer sitting on a tower with (6) 450's. Spectrum use on the 450's is mightly clean usually so I'd rule out the 450 complaint.

57.png

What total bandwidth usage do you see at peak time on your AP? Like the other posters I'd look to total sector capacity as the primary issue first. If you've got 50 subs on a tower at night using netflix at 4 mbits each that's 200 meg on a sector that can do 80-100? There ya go. 

Tim

There is no law regarding antenna types, so yes you can take down a panel and place a horn in its place as long as you abide by the regulations on EIRP for your counrty. In Canada where I am there are regulations regarding the "bad neighbor" experience. Some of these are also in the FCC regulations too, so make sure you are not a bad neighbor.

We in the forums do try to help as best as we can, but if you withhold information then you will only get basic help.

Also, forums should not be a last ditch effort, if you wait to ask for help then you loose all that time that others could have been helping you. And yes you must endure the inane questions of basic simplicity, this is basic troubleshooting and if we didnt do it, we would not be the professionals that we are.

This what is needed to be done (not tried already and didnt work, make sure it is setup and done):

Make sure your radios are not within the radiating centers of their radios, make sure you have some virticle and horizontal spacing. If you have to share a "pin-wheel" mount, then place your radios on the oposite end of the virticle.

Once you are sync'd with the other wisp, this means working with them to get a channel use plan, channel width plan, and stream ratios set. You can be assured that the problem is not from them. Use the the co-location tool and work with the other wisp to get things set properly.

Make sure you have enough backhaul bandwidth, just because "its fiber" does not mean its not 10Mbps. Fiber is a medium not a speed. T1, OC1, OC3, DS4 are speeds. Your AP's need 100Mbps per AP at that site with a router/switch capable of that bandwidth and the packets per second that your clients will be sending. Having a router that can do 1.5Gbps is nice but if its rated with packets sized 1700kb then with 400kb sized packets it will have a hard time keeping up, this is why the big companies like Cisco and HP have rated their offerings in PPS. I know a lot of small wisps are using more affordable offerings, but the backend of the network is just as important as the radio system you use to get to your clients.

Monitor your frame utilization (MRTG/Cacti can do this and graph it for you), this percentage is based on your stream ratio setting so make sure you understand what it means. Eg. your stream ratio is 50/50, your frame utilization is 50%, this is 1/4 of the total bandwidth the AP is capable of being used. The ePMP1000 is capable of 200MB total or 100MB full duplex at 40MHz channel, so at the previous configuration settings of 50/50 and frame use of 50% the AP is using 25MB of its bandwidth. Be careful of how many subs are set to use large QoS settings as this will affect your frame utilization and effect other users. A good Fair Access Plan and method of applying it is a must these days.

If you are running a high frame utilization rate and are only using a 20MHz channel, try a 40MHz channel.

If you have too many subs with high use packages, you can always add more Ap's to a sector by sub-dividing, but this will increase your RF noise. Another alternative is to switch to a 450 yourself, these are designed to provide commercial service and are the next logical step up. The ePMP is not designed for a lot of heavy use subs on a single AP. Yes it is a forklift solution, yes it is expensive, but if you want to stop bleeding clients and even entice them back, you need to be concerned with their experience too. A unhappy client will cost you other clients before they even connect.

1 Like