Ask the Experts: ePMP Troubleshooting

Sakid_Ahmed · June 17, 2015, 2:06pm

@Guilherme wrote:

i meant EVM in the interference evaluating sense, like in the ptp250 graphs / other vendor's constellation.

But thanks for the attention!

Question on this interesting topic -

Would something like a trending graph showing per packet EVM (maybe represented by a simpler term) be beneficial ? This is very preliminary thought but something where you see EVM go up/down could help. Keep in mind that it may not track to our MCS changes as that would use a more sophisticated algorithm and this would essentially be raw packet EVM trend.

Sakid

Guilherme · June 17, 2015, 2:29pm

Question on this interesting topic -
Would something like a trending graph showing per packet EVM (maybe represented by a simpler term) be beneficial ? This is very preliminary thought but something where you see EVM go up/down could help. Keep in mind that it may not track to our MCS changes as that would use a more sophisticated algorithm and this would essentially be raw packet EVM trend.

Sakid

I think so, if that can be correlated to local noise to a higher degree than to good aiming of the antennas.

if it would be graphed in the eAlign window, we would see a stable RSSI line and a spike in the EVM line in case of spurious noise, right?

I think that showing the noise floor of every station in the "monitor > wireless" tab of the AP could be benefical too.

Not every station "sees" the same RF environment as the AP.

Changing subjects a bit, not every of our stations is perfectly aimed, especially because at the time of the initial deployment of the network, we had no eAlign tool, and the ATPC algorithm kept changing the MCS/TX power. It would be very nice to have a "Alignment mode", where the MCS rate and the TX power are locked during the alignment procedure.

Sakid_Ahmed · June 17, 2015, 2:43pm

@Guilherme wrote:

Question on this interesting topic -

Would something like a trending graph showing per packet EVM (maybe represented by a simpler term) be beneficial ? This is very preliminary thought but something where you see EVM go up/down could help. Keep in mind that it may not track to our MCS changes as that would use a more sophisticated algorithm and this would essentially be raw packet EVM trend.

Sakid

I think so, if that can be correlated to local noise to a higher degree than to good aiming of the antennas.

if it would be graphed in the eAlign window, we would see a stable RSSI line and a spike in the EVM line in case of spurious noise, right?

I think that showing the noise floor of every station in the "monitor > wireless" tab of the AP could be benefical too.

Not every station "sees" the same RF environment as the AP.

Changing subjects a bit, not every of our stations is perfectly aimed, especially because at the time of the initial deployment of the network, we had no eAlign tool, and the ATPC algorithm kept changing the MCS/TX power. It would be very nice to have a "Alignment mode", where the MCS rate and the TX power are locked during the alignment procedure.

We will have to think through this a bit more. We actually need registration and packet transfers to capture that level of information but I see how localized interference representation at the time of alignment could help you. We do have eDetect that would actually give you same channel interference sources and this could be used in conjunction with alignment activities. As far as per SM interference level, we have link capacity & link quality as Sri mentioned before but maybe something like a DL interference metric that represents packet EVM could be useful.

On your second topic, you can lock the max MCS rate and set the max tx power manually. Not sure how locking the MCS and Tx power to an exact level will help with the alignment.

wvvamike · June 17, 2015, 5:16pm

Has anything been done or looked at into this issue? http://community.cambiumnetworks.com/t5/ePMP-1000/75-25-ratio-low-client-s-upload/m-p/40407#U40407

Cambium_RayS · June 17, 2015, 5:48pm

Our ePMP development team will be joining us in a few minutes. Please post your questions now.

Also, if you like the response, be sure to click to give the post a kudo. If the response solves the problem, please click on ACCEPT AS SOLUTION.

Sakid_Ahmed · June 17, 2015, 6:10pm

@wvvamike wrote:

Has anything been done or looked at into this issue? http://community.cambiumnetworks.com/t5/ePMP-1000/75-25-ratio-low-client-s-upload/m-p/40407#U40407

Yes, we are characterizing this behavior in our labs and hope to have an update either way for you.

Sakid

ninedd · June 18, 2015, 12:31am

Hi. With the 'eDetect' feature, I thought that was supposed to list all AP/CPE's that the unit could hear? I understood it would include all widths, so if I'm in TDD Mode and 10 Mhz, I thought it would still show me the Clients' router (in 20 Mhz) if was visible to the SM? I also thought that the eDetect would show all systems (Cambium, WiFi, UBNT) that it could hear.

The Site Survey tool is powerful, but having a eDetect that shows everything that the SM or AP can hear, and being able to sort it by frequency proximity and by RSSI, that's what I understood from the Cambium sales guys that it could do?

So, if it can't do that, then as a Troubleshooting help - what do we do when a client calls with slow internet, yet their signal looks fine. How do we detect interferers which may be foreign or which may be on different channel widths?

ninedd · June 18, 2015, 12:36am

Hi. With our older (non Cambium) gear. About 80% of our support calls for ''slow internet'' are really because the client has an unrestricted Torrent program that they don't understand, or they have devices with Cloud storage aps that they don't understand. Right now, we can look at our AP's and see all stations and their throughput up/down in real time.

That allows us to see "at a glance" what everyone is doing on the AP, and every day, we probably have a dozen calls where people report 'slow' and we log in and they have 300 KBytes of transmit going on constantly that they are unaware is even happening. That's out-competing their other traffic and they feel ''slow'' when it's really just 'occupied''.

So, with ePMP - how can I look at an Access Point and see a real-time. constantly updated list of all associated stations and their throughput? I know we can't do it with our Canopy 900 gear (PMP100) and I don't see how to do that on the Cambium ePMP1000 either.

Cambium_RayS · June 18, 2015, 1:27am

Hi ninedd and others. Our ePMP development team will be joining us in 30 minutes, so start posting your questions here.

Dont forget to kudo posts that you like, and also click on ACCEPT AS SOLUTION those posts that fix your problems.

Sakid_Ahmed · June 18, 2015, 2:13am

@ninedd wrote:

Hi. With the 'eDetect' feature, I thought that was supposed to list all AP/CPE's that the unit could hear? I understood it would include all widths, so if I'm in TDD Mode and 10 Mhz, I thought it would still show me the Clients' router (in 20 Mhz) if was visible to the SM? I also thought that the eDetect would show all systems (Cambium, WiFi, UBNT) that it could hear.

The Site Survey tool is powerful, but having a eDetect that shows everything that the SM or AP can hear, and being able to sort it by frequency proximity and by RSSI, that's what I understood from the Cambium sales guys that it could do?

So, if it can't do that, then as a Troubleshooting help - what do we do when a client calls with slow internet, yet their signal looks fine. How do we detect interferers which may be foreign or which may be on different channel widths?

Hi,

eDetect will only give you what is running in the same channel and not in channels outside of your operating channel and bandwidth. Note that eDetect is running while your AP & SM is still in service. Unlike site survey tools it doesn't go into Rx mode only to scan all channels. When a customer calls in slow speeds and you are suspecting interference, you should run edetect and check what MAC adresses are the sources. We list the MACs of the interferers and their levels represented in colors. Its a pretty good start to the troubleshooting stage.

Sakid

Daniel_Sullivan · June 18, 2015, 2:14am

eDetect uses free time within the frame and enters promiscuous mode. This allows the SM to hear other 802.11a/b/g/n devices on the channel. If the 802.11 interferer is not on channel, then it cannot be detected with eDetect. 5 and 10 MHz channels cannot be heard by a 20 MHz channel. I believe 20 and 40 MHz channels can hear each other as long as the beacon information is on the same 20 MHz channel.

In order to hear other interferers (e.g. Canopy), the Automatic Channel Selection (ACS) tool could be used. ACS can only be run in AP mode. When ACS is enabled and run, it will scan for energy across the whole band and display the overall level on a normalized scale for each channel in the band and also pick the best overall channel for the selected bandwidth. A person can use this information to for a variety of APs to perform channel planning or plan a single AP.

Sakid_Ahmed · June 18, 2015, 2:23am

@ninedd wrote:

Hi. With our older (non Cambium) gear. About 80% of our support calls for ''slow internet'' are really because the client has an unrestricted Torrent program that they don't understand, or they have devices with Cloud storage aps that they don't understand. Right now, we can look at our AP's and see all stations and their throughput up/down in real time.

That allows us to see "at a glance" what everyone is doing on the AP, and every day, we probably have a dozen calls where people report 'slow' and we log in and they have 300 KBytes of transmit going on constantly that they are unaware is even happening. That's out-competing their other traffic and they feel ''slow'' when it's really just 'occupied''.

So, with ePMP - how can I look at an Access Point and see a real-time. constantly updated list of all associated stations and their throughput? I know we can't do it with our Canopy 900 gear (PMP100) and I don't see how to do that on the Cambium ePMP1000 either.

Under Monitor--Performance you can see total packets & kbits per SM in a list. This is cumulative but increasing every few seconds. A quick look at this will give you an idea of any SM where there is background apps pulling data. Representing this in a rate fashion is something we can look at.

Daniel_Sullivan · June 18, 2015, 2:27am

At Monitor -> Performance on the AP, there is a summary of all devices throughput both DL and UL in Kbits. Below this there are Subscriber Module Statistics that show DL and UL throughput in Kbits for each associated SM. You can see throughput by observing the increasing totals themselves or also by selecting Reset Stats from this same screen and this will start all stats back at 0 so you can see how quickly the throughput increases.

Cambium_Sri · June 18, 2015, 2:41am

Hi ninedd, The statistics mentioned by Dan and Sakid are available through SNMP as well. So once you load the ePMP MIB into Cacti (which I believe you are using), you can pull these stats and graph them.

Thanks,

Sriram

Cambium_RayS · June 18, 2015, 3:01am

thank you everyone for participating with your questions. Feel free to continue to add questions about ePMP troubleshooting, and we will get to them. Thank you for your business and sharing your ePMP network stories with us.

Thanks also to our ePMP development team. We appreciate your efforts.

ninedd · June 18, 2015, 6:17am

Under Monitor--Performance you can see total packets & kbits per SM in a list. This is cumulative but increasing every few seconds. A quick look at this will give you an idea of any SM where there is background apps pulling data. Representing this in a rate fashion is something we can look at.

Hi. Thank you, that's a good start. For us, having this displayed as a 'per second' rate is essential. And seeing all clients on the AP in one list is essential too. I'd love to see the Kbit/s (or KByte/s) number in the Monitor -> Wireless list, although adding it to the Monitor->Performance is fine too. As long as we can see all cleints and their real time RX/TX speeds, that's fine.

Only seeing an increasing kbits total data counters and trying to do the math to figure out a rate, while I'm on the phone with a client.... I can't see that being a long term solution. Having the display show a Kbit or Kbyte / second rate is essential with all the cloud/torrent issues we deal with.

ninedd · June 18, 2015, 6:25am

@Cambium_Sri wrote:
Hi ninedd, The statistics mentioned by Dan and Sakid are available through SNMP as well. So once you load the ePMP MIB into Cacti (which I believe you are using), you can pull these stats and graph them.

Hi. Yes, I think we have the CACTI graphing thing figured out now. But, I'm asking about seeing the throughput data in real time, right while we are talking to a client on the phone. Graphing is cool, but that's waiting 5 or 10 minutes and then seeing 1 or 2 pixels on the graph.

With StarOS or MikroTik, we can see instantly/constantly the throughput that each client is getting at that instant. If the Cambium AP list is updating only every 5 seconds instead, that's fine. But we will absolutely need to be able to see this info or we are dead in the water with ePMP. We use this every day, on nearly every single client call.

Sakid_Ahmed · June 18, 2015, 6:50pm

@ninedd wrote:

@Cambium_Sri wrote:

Hi ninedd, The statistics mentioned by Dan and Sakid are available through SNMP as well. So once you load the ePMP MIB into Cacti (which I believe you are using), you can pull these stats and graph them.

Hi. Yes, I think we have the CACTI graphing thing figured out now. But, I'm asking about seeing the throughput data in real time, right while we are talking to a client on the phone. Graphing is cool, but that's waiting 5 or 10 minutes and then seeing 1 or 2 pixels on the graph.

With StarOS or MikroTik, we can see instantly/constantly the throughput that each client is getting at that instant. If the Cambium AP list is updating only every 5 seconds instead, that's fine. But we will absolutely need to be able to see this info or we are dead in the water with ePMP. We use this every day, on nearly every single client call.

Ninedd,

Have you tried our free network management tool CNS. It has the ability to show you the tput graph you are looking for per subscriber along with many other features. While the colors maybe a bit strong in the image below it serves the function you are looking for. I am hesitant to do a real time tput calculation on the radio if the CNS solution suffices

Guilherme · June 18, 2015, 7:13pm

CNS polling times are and will always be too far apart to be a usefull "Instant tput" tool.

We tend to use mikrotik's "interface>> traffic " graph for troubleshooting, so we can see the traffic bursts.

It updates maybe 2-3 times a second, whereas CNSS updates every 5 minutes.

The main-screen tput graph is updated every 5(2 if you log-in and change it) seconds, and is very nice, even if a bit slow(when i'm looking for real time burst traffic, that is).

ninedd · June 19, 2015, 5:52am

Ninedd, Have you tried our free network management tool CNS. It has the ability to show you the tput graph you are looking for per subscriber along with many other features. While the colors maybe a bit strong in the image below it serves the function you are looking for. I am hesitant to do a real time tput calculation on the radio if the CNS solution suffices

Hello. First, thank you for taking the time to listen and to try to help. I do appreciate it. :)

Second, yes, I've looked at CNS - it's great at some things, but I don't see anything in there that shows a list of all the customer's on an access point and a real-time updating screen of all their info, including throughput. In the AP there is MONITOR - WIRELESS which does show a list of SM's and it does show some good data, but it doesn't show a real time list of throughput for every station.

So, with the StarOS that we use now, we see this list when we look at an AP. YES, this is an old-fashioned SSH/Text display - BUT it's also very very useful. It's updated in real time (once per second) and every number on here changes. It's showing us every station and their RX/TX throughputs in KBYTES/second. We can instantly see what speed every client is getting.

So, when client xx:xx:xx:xx:AF:D7 calls and says that his internet is ''slower than dialup'' and we can look and see tha t he is getting 259KBytes at that instant... we know that his internet isn't really slow - it's actually just occupied and probably a computer downloading Updates, or by a Torrent or Cloud Storage program that he's unaware of, or someone next door stealing his WiFi - and seeing that instant throughput reading is very, very, very useful info. Yes, graphing is also very important, and seeing longer term graphs is very useful as well - but it's totally different than seeing a real-time list of every client and their kbps or kBps throughput updated every second or two or five.

And - it's important to see all the clients at once. Sometimes client who has become a -77 for some reason, when he's getting his 250 KBytes, that may be screwing up someone else. I know - Cambium's Scheduler helps prevent that problem and that is one of many reasons that we're switching away from StarOS, but for many diagnostic situations, there's no substitute for seeing real-time throughput for all associated stations.

Another scenario - a client calls and they are trying to configure their email program. Every time they click ''send and recieve'', they get no mail. However, when they are doing that, we see a burst of data and instantly know that they ARE in fact picking up the mail... and that it's likely a filter in their Virus or SPAM or in their Email program that's filtering or blocking or something. Being able to see a real time throughput helps point us where to look in an email situation.

So - yes, graphing with CACTI is great. The CNS program is great. But waiting 5 minutes to see one data point.... that's not real time throughput reports.

Right now, in the ePMP's Access Point list - it already has MONITOR - PERFORMANCE which already does have a Subscriber Module Statistics area. And, that list already does show incrementing data totals for 'Total Uplink' and 'Total Downlink'. If that list could subtract the previous value from the current value to get the difference (the delta) since the last display interval, and then divide that by the display interval - that would be all I'm asking for.

So, two new fields in that list that show the RX and TX values of:
Calculated by (Current Total Data - Previous Interval's Total Data) / Webpage Auto Update Interval Time.

I think that's all we'd need.

Ideally, it'd also be nice to be able to change 'units' for the data, the same as we can for miles/kilometers. It'd be good to be able to set the units, and then see all the data counters and settings represetned by bits, or bytes or whatever units the admin wants to see them in. However - for now - we'd need to be able to see the throughput speed of all the assocated stations in a list - either (or both) the Wireless Performance list, and/or the Monitor Wireless list.

Thank you.