Parallel updates suggestion

I noticed something:

Say I have 10 APs, with 2 clients on each, all onboarded.

I want to update the firmware, so I set a job with all 10 APs and all of their clients selected, select to update SMs then APs, and set parallelization to 20 units at a time.

The job apparently proceeds to update two SMs off one AP, wait for completion, then update that AP, wait for completion, get two SMs off another AP, wait, then that AP, wait, rinse and repeat.

What I would hope from it would be to update 20 SMs simultaneously, then after all SMs off a given AP are finished update the AP.

Overall time difference can be huge, as short as two complete update cycles,  (SMs then APs) as opposed to twenty complete update cycles the way it behaves now for the constructed example above.

In addition, I'd offer the observation (based on my previous perl-scripted network-wide ePMP updates) that it's a waste of resources to scale for 20 simultaneous updates, then wait for them to complete.  It's more sensible IMHO to wait until each (up to simultaneous limit) has finished receiving the firmware file and started to flash, then move on.  Check back on progress, but the primary limitation on parallelizing updates is a firmware transfer bottleneck, so once the transfer is finished let another unit use the now-idle server resources.

j

edited to clarify, I'm running 1.2.2-r3 on premises.

1 Like

cnMaestro updates the ePMP/PMP  device images based on sector wise. In a sector first it will update the AP or SM based on the selection of the option either AP or SM first and then the SM or AP next. The parallel count of 20 is also for the sector itself. There are customers with more than 20 SMs in a sector itself and all of them can be updated in parallel 20 at a time. Once a particular sector is updated it will update the next sector of AP and SMs.

For wifi devices it is 20 devices parallely at a time.

If your requirement is different please post your requirement in the ideas section so that we can check the feasibility of implementation  for the future releases.


@KR wrote:

cnMaestro updates the ePMP/PMP  device images based on sector wise. In a sector first it will update the AP or SM based on the selection of the option either AP or SM first and then the SM or AP next. The parallel count of 20 is also for the sector itself. There are customers with more than 20 SMs in a sector itself and all of them can be updated in parallel 20 at a time. Once a particular sector is updated it will update the next sector of AP and SMs.


I'm aware of how it behaves now, as mentioned above.  Understand there are also customers who do NOT have 20+ SMs in each sector.


@KR wrote:

If your requirement is different please post your requirement in the ideas section so that we can check the feasibility of implementation  for the future releases.


It's not a question of my "requirement", it's an observation that the overall update job could be running and completing in a fraction of the time it currently eats, give the two suggestions I offered.

I've managed to update firmware on over 300 ePMP radios in under 30 minutes using my old perl script - the same update takes hours via cnMaestro.  (currently ~450 clients on ~60 APs - so in general cnMaestro would take 120 times the duration of a single update, assuming no sector over 20 clients)

j

Agreed.  We run a "low-site" network (aka micro-pop) and upgrading firmware on SMs with cnMaestro isn't feasible for us.  Having to expand every AP to then tell it to update every SM on the AP (after already clicking the "Select All Devices" checkbox) would take longer than it takes for our scrpt to just do the entire upgrade.

Just wanted to add that it seems silly that the parallelization doesn't apply to APs - cnMaestro only updates one AP at a time.  I literally updated 40+ APs last night while waiting on cnMaestro to update TWO.  (After updating 250 SMs in 10 minutes)

#!/usr/bin/perl

$ip=shift;
$community=shift||“private”;

print “Preparing to upgrade ePMP1000 SM at $ip.\n”;

my $v=/usr/bin/snmpget -v2c -c$community $ip cambiumCurrentSWInfo.0 -Ov;
chomp $v;
if ($v!=“3.1”)
{
print “Running firmware $v, updating to 3.1\n”;
my $r2=/usr/bin/snmpset -v2c -c$community $ip .1.3.6.1.4.1.17713.21.4.7.0 s http://10.200.10.7/fw/ePMP/ePMP-NonGPS_Synced-v3.1.tar.gz;
print $r2;
my $r2=/usr/bin/snmpset -v2c -c$community $ip .1.3.6.1.4.1.17713.21.4.4.0 i 1; # apply changes
print $r2;
sleep 3;
}

Ugly little hybrid script, I know: I couldn’t be bothered to utilize the Perl SNMP support in such a quick’n’dirty effort, but I have more complicated versions that do multiple updates if needed (to 2.4.3 then 3.1 for newly-arrived stock running ancient firmware, for example) and push default config files, and wanted Perl’s string handling rather than fight BASH.  From SSH login I run a one-liner to pull IPs from MySQL and update them all:

for ip in $(mysql -sss -p canmon -u root -e 'select ip from sm where devtype like "epmp%" and expect=1 and softver!="3.1";');do epmp-update.pl $ip rwcomm; done

And less than 20 minutes later all reachable SMs on the network have been updated to 3.1 firmware.

Then I run:

for ip in $(mysql -sss -p canmon -u root -e 'select ip from ap where devtype like "epmp%" and expect=1 and version!="3.1";');do epmpapupdate.pl $ip rwcomm;done

and all APs are done.  (same script, different firmware file referenced)

This was all from 2am to 2:30am - cnMaestro would have been still working at it when customers awoke this morning.

j

APs will not be updated parallely as if there is any error or mistake (Wrong image selected to update or image is not proper) cnMaestro wants to restrict it to the first sector or AP only and the user can manually stop the job or cnMaestro can stop the job autoimatically if the option Stop if any critical error occured is selected. 

Again it is sector based update and only one AP will be updated at a time and if any error occurs only that sector will be effected and other sectors will be safe .

There is an option to select 5/10/15/20 SMs to be updated in parallel and based on the number of SMs in each sector user can select a choice of his own for faster upgrade

Regarding the UI implementation, where a user needs to expand each AP in order to select the SMS, we are currently revising that so all devices can be selected.

I understand your comments about the speed of the upgrades and we will investigate these suggestions.

2 Likes

Taking the desire to avoid widespread rollout of a broken update into account, what I would like to see is:

Easy selection of APs and ALL their client devices with 1-2 clicks, ideally more simultaneous updates possible.  (currently N=5/10/15/20 - I'd really like to be acle to select higher, unless you guys can speed things up to where it doesn't wait for each unit to finish flashing and rebooting before moving on)

Build a temporary list of all SMs in sector order.

Update ONE SM (first on the temporary list), ensure it went smoothly.  Update up to N more SMs, regardless of which sector they're on - once firmware file has been downloaded for one, let another start downloading and just monitor in the background for the earlier one to finish with a successful flash/reboot.

In a separate thread, once all SMs on the first sector are finished, update that AP, ensure it went smoothly.  Then update all other APs as soon as their clients are all completed.

As a guesstimate for our network, that would end up taking about 1-1.5 hours, versus many hours the way it behaves now.  (and versus 20-40 mins doing it more aggressively and "without a net" with my own scripts)

j