It seems every time I program a firmware upgrade, something goes wrong. Using cnMaestro on-premise, I set it to upgrade a single v3000 from 1.2 to 1.2.1. This client is one of 6 attached to a v5000 and all were running 1.2. I sent the upgrade command. cnMaestro pushed the firmware to that radio only (as requested) and then every radio on that DN - including the DN - rebooted. Not what was requested. The firmware was upgraded in the one I requested but they all rebooted. When the DN came back online, it had turned off ignition for a different client radio. Took us a while to discover that was why one client never came back online.
I have yet to have a smooth upgrade and I feel like I should stop using cnMaestro to handle firmware for cnWave gear.
Can you please advise what version of cnMaestro On-Premises your currently using?
Can you kindly get below info, which would help me to comment more about this behavior
(1) Topology (tech dump of E2E controller would help)
(2) Upgrade path of the network , especially the CN causing the issue
(3) were there such issue in prior upgrades
(4) Field diags of involved devices
(4a) The CN which underwent upgrade
(4b) Another CN for which ignition was turned off
(4c) unintended CN which rebooted ← I would like to double confirm.
(4d) DN to which 6 CNs were linked up
3.1.0r1 is the cnMaestro on-premises. I am grabbing tech dumps from the radios. What is the best way to get those to you?
Not sure what you mean by upgrade path of the network. No issues on this network segment prior to upgrade.
Thanks for confirming “No issues on this network segment prior to upgrade”.
By upgrade path, I want to know the initial version of cnWave software during installation and further software versions to which the network was upgraded
For example: 1.0 → 1.1 → 1.2 → 1.2.1
Kindly upload related files via onedrive and share the link
(If above doesn’t go well, then share it to my email id prasanna.tm@cambiumnetworks.com)
Replied directly to your email with a shared .zip file containing the requested files.
Great Thanks.
I have successfully downloaded shared files.
Shall keep you posted on my findings shortly
Hi Chadwick,
Sorry for the delay, as I had to make some internal follow-up and confirm things
In summary,
DN_V5000_884ec7 configured as Onboard E2E controller experienced reboot due to firmware crash causing all LINKs to go DOWN. We are working to get a fix that caused this reboot.
Furthermore, CN_V3000_VisitCntrNode for which ignition status was seen disabled is separate issue and is being looked upon.
As tried in Cambium lab, we were not able to repro above issues.
But the logs shared by you is good enough to take things further.
Can I request AU wireless to repeat the sequence during a maintenance period and share observation ?
As a side note - When using onboard E2E controller, its recommended to upgrade the whole devices together instead of upgrading the leaf node first. I would be documenting this suggestion for customers shortly.
I will attempt to upgrade a single radio again and see what happens and report back.
I went to upgrade a point to point link from 1.2 to 1.2.1 using cnMaestro. As soon as I clicked the button to syncronize the new firmware version, both radios rebooted. It did not do the upgrade, it just rebooted them. This is the same behavior we saw with the last upgrade on the v5000 node - sync the new version and it reboots both ends.
I have tech support files from this test as well. This makes me very hesitant to upgrade anything else.
Hi Chadwick,
Thanks for sharing field diagnostic files of related nodes.
Shall keep you posted on my analysis.
Hi Chadwick,
Thanks for sharing more info via email.
Can you kindly select 1.2.1 and then click Sync image
Above suggestion is with reference to below notes from “Software update” section (page 282), “60 GHz cnWave Network Configuration” chapter of cnMaestro user guide, which is applicable to your usecase.
“Onboard E2E controller will support only one synced image. If user needs to sync another image,
select the image from Versions drop down and click Sync Image.”
Summary on my analysis from field diags shared by you:
(a) Remote V3000 (mnt-ridge) did not experience reboot
(b) PoP V3000 experienced reboot due to “dirty reset”
Kindly keep this thread posted on how it goes.
What you are asking me to try is exactly what I have done twice and each time I do it, it crashes the POP DN. When I selected 1.2.1 image and clicked “sync image”, it synced the image to both radios and then immediately rebooted the POP DN. I never asked it upgrade and it didn’t upgrade, it just synced the image.
I have repeated this exact procedure twice now and both times, it syncs the image and then reboots the POP DN. The first time was a multi-point environment and it rebooted the V5000. The second time was in a PtP link and it rebooted the V3000 POP DN.
Reboot of related nodes is due to assert 0x2738 [inferences from recently shared field diagnostic files]
Version 1.2.2 has fix for 0x2738 which potentially would resolve issues during your future upgrades
