ePMP Release 4.4.3 is now available

Dmitry,

   I since rolling the 3000L back to 4.4.1 I have re-upgraded it to 4.4.3 (9 days ago) and even swapped out a N client with a AC (F300-16) (3 days ago) and no AP reboots or client drops.  So it appears that all of the problems we were having were due to the SA being on.  

1 Like

@brubble1 wrote:

Dmitry,

So it appears that all of the problems we were having were due to the SA being on.  


This is good news - good to hear. :) 

I had suggested to Cambium that the SA should 'almost always' be a timed SA. So, when we go to turn it on in a customer's SM or in an AP, the default way to turn it on should maybe require a time - something like 'Activate SA [for x minutes]' where it needs a value put in.  If someone wants to put in 1440 minutes and run it for a full day - or if they want to put in 43200 and run it for a full month, that's cool. But there should always be some automated method to have it shut off after some time period.

Longer term - there may be a good reason that the SA needs to be always running (it may be needed for Dynamic Frequency features) but at least in the meantime, having some method of not accidently leaving it on, even after we log out of the radio, would be helpful.

1 Like

@brubble1 wrote:

Dmitry,

   So it appears that all of the problems we were having were due to the SA being on.  


Thanks for the heads up and sorry that we caused you trouble with that.

Hi ninedd,

Thanks for the idea. I was thinking about similar implementation as a workaround, but as you mentioned we want to run spectrum analyzer in the background all time for the DCS feature and other future coming features.

Dmitry

1 Like

@brubble1 wrote:

" cannot stay in customers router,"

We are just now starting to roll out F300 to customer installs/repair so I'm curious, what does that mean ? 


We deploy Mikrotik routers at our customers, so we can log into the router and further assist when they are having an issue. 

Our connection didnt hold and kept dropping


@Dmitry Moiseev wrote:

@brubble1 wrote:

Dmitry,

   So it appears that all of the problems we were having were due to the SA being on.  


Thanks for the heads up and sorry that we caused you trouble with that.


OK well after 4.4.3 being back on the AP for about two weeks now we have had some problems. This morning I get a call from a customer on that AP saying they had no Internet. They reported "I've been randomly losing internet for about two weeks but it's usually only for a minute or two but this morning it's been down for 20 minutes now".

So try to log into their radio and can't reach it.  I log into the AP and it shows it's only been up for just over a day (so it rebooted a day'ish ago) I log into the BH Radio and it has been up for weeks , so it wasn't a power issue the 3000L rebooted.  I looked at the cacti logs and just the one mystery reboot about 1.5 days previous, but a day before that the logs also show a rash of dropped client connections...

Anyway back to the AP,  all 9 clients are connected but I can't access any of them. The Monitor > Throughput Chart shows that not a single bit is passing , not a single bit, 0 in 0 out.. well that seems impossible since I'm remoted into the radio and bits would have to pass in/out in order for the radio to show me no bits are passing in/out.

So I reboot the AP, only it doesn't really reboot it just logs me out.  At this point I download a support file from it (seems to take a really long time) and I try to reboot several more times but it just logs me out each time. So I try to SSH in but no response.   


I drive to the site and power cycle everything  ( the site consists of:

    F200 Backhaul <-> Power supply <-> Power Supply <-> 3000L 

 nothing else, not even a surge protector.)

It appears everything comes back up, everyone has internet , everyone is happy.

15 minutes later I get an alert that AP is down.   I can't reach it but I can reach the F200 that it's connected to and the ethernet port is flapping (1000green, 1000red , no connection).   I can access the AP as long as the BH is 1000green, I try to grab a support file but can't keep connection long enough (seems to take longer than normal to generate support file).   I did manage to make it to the firmware page and start 4.4.1 firmware upload and after a bit I manage to log in, see the firmware is waiting for a reboot and I reboot the radio (and it works this time).

The AP came back up 4.4.1 , that was about 3 hours ago and so far it looks like everything is good.

The only customer I talked to during the outage (several called the office to report it was down) was the one with the F300-16 (the one and only AC radio on this 3000L the other 8 are N). So I don't know if they are the only one having the "internet randomly dropping for a minute or two".   I do know that a week ago we put up our first 3000 AP with the 3000 Sector and Beam Forming panel (I know it doesn't work right now) and conneted my home to it with a F300-16 and immediatly noticed that my internet was dropping for just a minute or so a couple of times a night.   

I'm not even going to mess with trying to track the problem down I'm just going to roll everything back to 4.4.1 and see if the weirdness goes away.


If the internet drops again before I roll it back I'll grab a support file off my F300 and 3000AP in case support wants them.

Edit: Update (Monday 1/13/20) - So the Micropop has been on 4.4.1 since Saturday morning. I talked to the customer with the F300-16 a few minutes ago and she says she hasn't noticed it dropping since I fixed it Saturday.  While the radio shows it has been up since Sat (1 day, 23 hours) the graphs show it rebooted this morning at 9am...   But it took almost two weeks for it to go completely wonky so no idea if 4.4.1 is any better than 4.4.3 or not.

Weirdness with the 3000 AP we just put up a week ago. Tower / AP is LOS and 500ft from my house so I put myself on a F300-16.  I was the only client the first day and twice I lost, what I thought at the time, access to the internet.

I updated all the AP's and clients (all 2000 / N) to 4.4.3  so I could move some of the N clients to the new 3000AP.  I moved 9 over, so the AP had 1AC (myself) and 9 N's.  The randomly losing internet for 45 secs to a minute or so got worse, a lot worse.

I figured out that I wasn't just losing internet , I was actually unable to reach the F300-16 I was connected to.

PC <> Dumb Switch <> F300-16 in bridge mode <> 3000AP  <> Mikrotik Switch <> Fiber   

So my F300-16 would stop responding for about a minute. When I could reach it again I would instantly be able also reach the 3000AP but it would be 15 or 20 more seconds before I would be able to reach the switch the 3000AP was connected to (or anything past it). 

When I downgraded the AP to 4.4.1  all the N clients went back to their respective N AP's.  For 5 hours I never lost connectivity , so I started putting N clients back on and before I could get all 9 back on the 3000AP I started losing connectivity (same thing, could not reach my own F300-16).   

So I removed the N clients again and didn't lose connectivity for several hours, so I put the AP back on 4.4.3 and no issues for about 3 hours , then it lost connectivity, but then for the rest of the evening it was fine. 

When I got up this morning and looked at my graphs , I had lost connectivity 30'ish times over night  and when I looked at the AP one of the N clients had reconnected to the 3000AP over night.

So, regardless of 4.4.1 or 4.4.3 I my F300-16 client stops responding  (according to the session time/count it does not lose connection to the AP) to access via the LAN several times an hour if there is even a single N client connected to the AP  OR  if running 4.4.3 with no other clients on the AP it still happens but only every few hours.

Also odd that even when the F300-16 starts responding again it's still 15 or 20 seconds before I can reach beyond the 3000AP.  The F300-16 is bridged and all IP's are static and in the same /16 management subnet


PC 10.10.156.1 <> dumb switch <> F300-16 10.10.156.0 <> 3000AP 10.10.1.53 <> Mikrotik 10.10.1.251

As far as I know this is the same issue the customer with the only AC on the 3000L Micropop was having but their problem went away when I reverted the 3000L to 4.4.1 ( 1 AC client and 8 N clients).

I don't know if the N clients are having any issues when connected to the 3000AP , no one complained in the 5 days the were on it.   I'm going to swap out my F300-16 for a F300-25 today and see if it also has problems.

I have 1000 and 2000 AP's running 4.4.1 , and two of them have a couple of F300-16 clients in the mix and as far as I know there are no issues (or at least no one is complaining).

I don't have sectors with mixed AC and N clients, but I am having odd issues with 4.4.3.  I recently upgraded four ePMP1000 2.4 GHz sectors on  10 MHz channels with all ePMP1000 clients (force200 and connectorized SM's) from 3.5.6 to 4.4.3.  I immediately noticed a reduction and instability in the downlink MCS of the clients along with a 1.5x to 2x increase in retransmission rates.  The graphs below show a few clients on one of the sectors, with the upgrade happening around 3:25 AM:

On an ePMP1000 5 GHz sector with a 40 MHz channel that was upgraded, there does not appear to be any significant drop or instabitliy in MCS.  Retransmission may be slightly higher, but it is not clearly noticable.

On a 5 GHz force190 PTP link using DFS frequencies and 10 MHz channel, upgrading from 3.5.6 to 4.4.3 caused the link to drop the initial connection and switch to an alternate frequency.  This was reproducable on every reboot, it would establish a link then drop it within a minute and switch frequencies.  This had never happened before on that link.  I also noticed the same degredation in downlink performance on the PTP link as on the sector I upgraded.  I reverted the PTP link back to 3.5.6 and the MCS values went back up, the retransmission went back down, and it held on to its first assigned DFS frequency (it did to switch).

In all these cases, on the uplink there may be a minor degredation, but not clearly as noticeable as on the downlink.  I am suspecting there is some issue witih this firmware and 10 MHz channels, but I really don't know.

Is this degredation in performance expected on ePMP1000 with 4.4.3?  Is this a bug?  I'm downgrading my sectors back to 3.5.6 as it has been very stable and reliable for me.  I am confused as to why CnMaestro lists 4.4.3 as the "recommended" firmware for an ePMP1000 AP with all ePMP1000 SM's if there is a performance penelty.

1 Like

Not loving 4.4.1 or 4.4.3 right now. Started upgrading all of our 5Ghz 1000/2000 AP's and CPE's to 4.4.1 and about the same time we started seeing a big increase in people complaining about speed tests and buffering Netflix. So I upgraded to 4.4.3 on everything but it didn't fix anything.  

  The odd thing is the speed tests coming back 8Mbps (most of the customers are on 30Mbps rates but we even tried setting them much higher). We test the radio link and it comes back as it should, it's as common during off peak as on peak, the backhauls are fiber and look / test fine, no ethernet errors, nothing that explains these recurring 8Mbps speed tests.    It's customers on 1000, 2000, and 3000 AP's , some with all N CPE's connected some with a few AC clients. It affects N clients and AC clients..  Not sure what is going on but we really never had much  "MUH NETFLIX IS'UH BUFFER'EN ! " complaints until just recently.

I'm leaning towards a problem with QoS but haven't had time to really test my theory yet. When I have time and a couple of test subjects I'll shut off QoS on an AP and see what happens.


@cleonello wrote:

I don't have sectors with mixed AC and N clients, but I am having odd issues with 4.4.3.  I recently upgraded four ePMP1000 2.4 GHz sectors on  10 MHz channels with all ePMP1000 clients (force200 and connectorized SM's) from 3.5.6 to 4.4.3.  I immediately noticed a reduction and instability in the downlink MCS of the clients along with a 1.5x to 2x increase in retransmission rates. 


Since it only seems to be an issue with your 10Mhz channels, I would open a ticket with Cambium. We have 3 towers using 4.4.3 for APs and SMs (all SMs on said APs are N, no AC) with no issues, and even saw an improvement in throughput. These are 20Mhz wide channels. We have a force 200 40Mhz PTP link up running 4.4.3. Hits peaks of 130Mbps or so. Been up 64 days 16 hours and we have seen no problem with it either. 

1 Like

@brubble1 wrote:

Not loving 4.4.1 or 4.4.3 right now. Started upgrading all of our 5Ghz 1000/2000 AP's and CPE's to 4.4.1 and about the same time we started seeing a big increase in people complaining about speed tests and buffering Netflix. So I upgraded to 4.4.3 on everything but it didn't fix anything.  

  The odd thing is the speed tests coming back 8Mbps (most of the customers are on 30Mbps rates but we even tried setting them much higher). We test the radio link and it comes back as it should, it's as common during off peak as on peak, the backhauls are fiber and look / test fine, no ethernet errors, nothing that explains these recurring 8Mbps speed tests.    It's customers on 1000, 2000, and 3000 AP's , some with all N CPE's connected some with a few AC clients. It affects N clients and AC clients..  Not sure what is going on but we really never had much  "MUH NETFLIX IS'UH BUFFER'EN ! " complaints until just recently.

I'm leaning towards a problem with QoS but haven't had time to really test my theory yet. When I have time and a couple of test subjects I'll shut off QoS on an AP and see what happens.


Strange. We have seen an overall improvement in throughput with the 4.4.x software.....2.4Ghz and 5Ghz. For us at least, this is only true for APs where -all- SMs are N devices. All APs are 20mhz (2.4Ghz as well). On the AP's where we added some AC SMs, overall throughput declined. We use the MIR/QoS system built into the AP.

Well we didn't see any improvement going from 3.5.6 to 4.4.# on any of the 1k or 2k  AP's but nothing appears to have gotten worse when I look at the graphs/data,  just a large uptick in people complaining about buffering and pixelating and the odd 8Mbps speed test so many of them seem to have in common.  I've had other issues to deal with though so I have not really dug into this problem yet and it may turn out that none of them have related problems.

Normally I would not have rolled out a new software without a lot more testing but I really really wanted to stop buying new N client radios and just start rolling out AC client so the upgrade to 4.4 had to be done.

I was looking at the list of customers complaining about buffering and pixelating recently and it may or may not be relevant but none of them are one of the 200'ish 2.4Ghz customers we have (we don't install new 2.4 anymore these are just poor souls that we couldn't make 5Ghz work on when we moved from Ubiquiti to ePMP). All of our 2.4 ePMP/Elevate AP's and CPE's are still 3.4.1 ... so... still gathering evidence.

I'm curious. The new 4.4.3 software updated the GPS on the radios and while all show the GPS firmware is up to date they are all different - 1k APs report 1.51_2838 while 2k's show 5.1_8174, 3k and 3kL has 5.1.1_8513   I wonder if I'm possibly having a self interferrenc issue with slightly out of sync APs.


@cleonello wrote:

Is this degredation in performance expected on ePMP1000 with 4.4.3?  Is this a bug?  I'm downgrading my sectors back to 3.5.6 as it has been very stable and reliable for me.  I am confused as to why CnMaestro lists 4.4.3 as the "recommended" firmware for an ePMP1000 AP with all ePMP1000 SM's if there is a performance penelty.


Hi,

It is not expected by any means. It sounds like a defect, but we don't see the described behaviour on the majority of installations. It will be awesome if you can open a support ticket and work with us to dig to the bottom of the issue.

Thanks,

Dmitry

Brubble,

I recently put up a ptp link using an epmp1000 lite (the most perfect use for these guys) and we were getting strange behavior from it and with all radios that were able to hear the signal. I thought I was loosing my mind as well as patience with this link and its effects on the rest of the close network. It turned out that the gps did not get updated and the signal was different enough to cause the colocation settings, though set correctly for the intended use, to be wrong. This is with 3.5.6 firmware.
The funny part that I still do not understand is how the gps firmware is used during sync over power from cmm3(packetflux gigabit injector)? As soon as the gps matched the rest of the system, everything just settled back to normal.
There is different gps revisions based on which hardware you are referencing, but if all AP gps’ are upgraded from the 4.4.3 sync firmware file then those versions are compatible.
Just a thought for ya.

Community in general,

We are looking at putting up a couple 3000L’s (do not need MU-MIMO, just more bandwidth and wanting to stop buying N based hardware), so far I am being turned off of the current prospects due to firmware/hardware interactions that are being reported.
So far I am not willing to drop $10k on an upgrade that reminds me of a particular competitor hardware brand.

1 Like

@brubble1 wrote:

I was looking at the list of customers complaining about buffering and pixelating recently and it may or may not be relevant but none of them are one of the 200'ish 2.4Ghz customers we have (we don't install new 2.4 anymore these are just poor souls that we couldn't make 5Ghz work on when we moved from Ubiquiti to ePMP). All of our 2.4 ePMP/Elevate AP's and CPE's are still 3.4.1 ... so... still gathering evidence.

I'm curious. The new 4.4.3 software updated the GPS on the radios and while all show the GPS firmware is up to date they are all different - 1k APs report 1.51_2838 while 2k's show 5.1_8174, 3k and 3kL has 5.1.1_8513   I wonder if I'm possibly having a self interferrenc issue with slightly out of sync APs.


We have about a dozen e1k 2.4GHz GPS AP's out in the field... some doing sync between themselves, and some sync'ing with PMP450 2.4GHz. As long as your GPS firmware is up to date, the version difference doesn't matter... they all provide the same 1PPS GPS sync timing. Same thing if you're getting sync via satellite on the radio or CMM3/4/5, Packetflux, etc... all provide 1PPS timing. You should be able to look at your logs and see if you're regularly losing sync.

One thing I can't speak of is elevated clients and their performance, as I have none on my network. I don't think Cambium intended for long term use of elevated clients... you should make strides to get these swapped out as soon as possible so you can get rid of any 1x1 radios along with being able to enable short GI and get a bit of a performance boost. We've measured short GI gains in the field between 3.5.6 and various 4.x versions and it does make a difference.

Lastly, there were session drop issues with earlier 4.x firmware... but we've done some pretty extensive testing between 3.5.6, 4.4.1, and 4.4.3... 4.4.3 is slightly better then 3.5.6 and a whole lot better then 4.x. Further optimizations are expected for the upcoming 4.5 release.

1 Like

@Douglas Generous wrote:

We are looking at putting up a couple 3000L's (do not need MU-MIMO, just more bandwidth and wanting to stop buying N based hardware), so far I am being turned off of the current prospects due to firmware/hardware interactions that are being reported.
So far I am not willing to drop $10k on an upgrade that reminds me of a particular competitor hardware brand.

I'd encourage you to try the 3000L's... if you're simply using them for PtP applications, I think you'll be fine. It's a significant performance boost for not a lot of money over the original e1k's. There's also a number of features (like active background SA) that the e1k's will never support.

I'm not sure where you're getting the $10k figure... maybe that's to upgrade many radios at once? Anyway, why not just try upgrading one link and seeing how it goes? One thing to keep in mind... the e3kL's don't support sync over ethernet, so if you rely on sync, make sure you do a bit of testing to see if the internal or external GPS antenna is able to lock onto the most satellites.

1 Like

@Douglas Generous wrote:
I am being turned off of the current prospects due to firmware/hardware interactions that are being reported.
So far I am not willing to drop $10k on an upgrade that reminds me of a particular competitor hardware brand.

For what it's worth - only people experiencing problems typically make forum posts. :)  We have had ePMP3000L in the air for about as long as they've been available, and they've been really quite good for us. I had a LONG time with zero GPS issues at all... and then just recently I've had a GPS issue on one of our AP's.  That may or may not actually be a firmware/hardware issue... maybe I've got water somewhere - maybe I've got one of the other antennas on the south side of the tower blocking the GPS receiver.  Anyway - my point is... for every bad/problem report you're reading, there may be 10 or 100 people who are not reporting anything at all, because things are working well for them.  That was my situation until about a week ago.

1 Like

@Dmitry Moiseev wrote:

It is not expected by any means. It sounds like a defect, but we don't see the described behaviour on the majority of installations. It will be awesome if you can open a support ticket and work with us to dig to the bottom of the issue.


Thanks Dmitry, I'll do that.  I am curious what you mean by "we don't see the described behaviour on the majority of installations."  Does this mean you have seen this behavour on some installations? 

Every tower in our network has a Packetflux sync injector and the GPS antenna per AP/ptp-master. We dont rely on the sync over wire with the epmp but it has made huge differences in stability when the puck just cant keep satellites attached or someone tries to shoot a new radio link through one of our towers causing the gps pucks to be desensitized. We place the packetflux gps unit much lower on the tower so it usually works when the pucks dont. Sync over wire is also a really helpful tool when things are going wrong. I have had two epmps supposedly in sync but they were not in frame sync as one was not the same. Turn off one radio, all is good. Turn off the other and all is good. Turn on both and all hell breaks loose. Sync to cable and all is good. Upgraded to 3.5.6 with new gps firmware and all was fixed, had to wait for over a month for 3.5.6 to be released so we just ran on the cable sync.

The 10k figure is our average coverage zone spending. Each zone is set to 16km (10mile) and when you figure 3 - 4 clients per $1000 it doesnt take many clients to get the equipment costs up.

I have a 3000L and a force300-19 on the way for testing now. Just hasnt arrived yet.



I have a 3000L and a force300-19 on the way for testing now. Just hasnt arrived yet.


You probably already know but if in case you don't, if you are using Packetflux with 48v power 3000L can not do 48v, it kills them.