2 AP problem - i can't upgrade or reboot

Giuseppe · January 31, 2017, 12:39am

first case :

ePMP 1000 gps sync with 3.2 version of firmware

- the dashboard is locked to the old frequency and old numbers of wireless client

- i can't reboot the AP from the button in the webserver page

- i can't reboot the AP also from ssh access

Oct 1 02:28:18 Diff1 admin[217.*.*.*]: reboot: Started

___________________________________________________________________________

Second Case:

ePMP 1000 no gps sync with 2.6 version of firmware

- i can't upgrade/reboot the AP in the syslog i found this

ePMP1000_c5688b>show syslog
Sep 3 07:02:44 ePMP1000_c5688b kernel: Out of memory: kill process 1659 (uwsgi) score 64 or a child
Sep 3 07:02:44 ePMP1000_c5688b kernel: Killed process 1673 (uwsgi)
Sep 3 07:02:45 ePMP1000_c5688b kernel: Out of memory: kill process 14883 (uwsgi) score 87 or a child
Sep 3 07:02:45 ePMP1000_c5688b kernel: Killed process 14883 (uwsgi)
Sep 3 07:02:46 ePMP1000_c5688b kernel: Out of memory: kill process 1659 (uwsgi) score 46 or a child
Sep 3 07:02:46 ePMP1000_c5688b kernel: Killed process 1674 (uwsgi)
Sep 3 07:02:47 ePMP1000_c5688b kernel: Out of memory: kill process 14886 (uwsgi) score 87 or a child
Sep 3 07:02:47 ePMP1000_c5688b kernel: Killed process 14886 (uwsgi)
Sep 3 07:02:48 ePMP1000_c5688b kernel: Out of memory: kill process 14887 (uwsgi) score 87 or a child
Sep 3 07:02:48 ePMP1000_c5688b kernel: Killed process 14887 (uwsgi)
Sep 3 07:02:48 ePMP1000_c5688b kernel: Out of memory: kill process 14888 (sh) score 32 or a child
Sep 3 07:02:48 ePMP1000_c5688b kernel: Killed process 14889 (sh)
Sep 3 07:02:49 ePMP1000_c5688b kernel: Out of memory: kill process 1659 (uwsgi) score 29 or a child
Sep 3 07:02:49 ePMP1000_c5688b kernel: Killed process 1659 (uwsgi)
Sep 3 07:02:50 ePMP1000_c5688b kernel: Out of memory: kill process 14898 (sh) score 49 or a child
Sep 3 07:02:50 ePMP1000_c5688b kernel: Killed process 14899 (ifconfig)
Sep 3 07:02:51 ePMP1000_c5688b kernel: Out of memory: kill process 14897 (sh) score 32 or a child
Sep 3 07:02:51 ePMP1000_c5688b kernel: Killed process 14898 (sh)
Sep 3 07:02:51 ePMP1000_c5688b kernel: Out of memory: kill process 1662 (nginx) score 28 or a child
Sep 3 07:02:51 ePMP1000_c5688b kernel: Killed process 1664 (nginx)

Please help me to resolve this issue , the ap are at 150km from me

Eric_Ozrelic · January 31, 2017, 12:42am

I think this issue is addressed in the firmware release that came out today. Check it HERE.

Cambium_Sri · January 31, 2017, 12:46am

Hi Guiseppe,

For the out of memory radio, if you are able to access the radio, ~~can you please reboot it~~ any chance you can power cycle it and then try the upgrade? I realize its 150 km away. This is an old issue where the user space gets filled up and only a reboot clears the cache to provide enough space to download the new firmware file.

As for the 3.2 issue, Eric is right. Are you using cnMaestro by any chance. You should be able to reboot the radio from cnMaestro.

Thanks,

Sriram

Giuseppe · February 2, 2017, 2:59pm

with cnmaestro i'have rebooted the AP but i still see this in the log

Sep  1 07:18:02 ePMP1000_c5688b kernel: Out of memory: kill process 1659 (uwsgi) score 95 or a child
Sep  1 07:18:02 ePMP1000_c5688b kernel: Killed process 1685 (uwsgi)
Sep  1 07:26:02 ePMP1000_c5688b kernel: Out of memory: kill process 8304 (uwsgi) score 97 or a child
Sep  1 07:26:02 ePMP1000_c5688b kernel: Killed process 8304 (uwsgi)
Sep  1 07:27:22 ePMP1000_c5688b kernel: Out of memory: kill process 9601 (uwsgi) score 113 or a child
Sep  1 07:27:22 ePMP1000_c5688b kernel: Killed process 10111 (sh)
Sep  1 07:27:44 ePMP1000_c5688b kernel: Out of memory: kill process 9601 (uwsgi) score 113 or a child
Sep  1 07:27:44 ePMP1000_c5688b kernel: Killed process 10510 (sh)
Sep  1 07:27:46 ePMP1000_c5688b kernel: Out of memory: kill process 9601 (uwsgi) score 102 or a child
Sep  1 07:27:46 ePMP1000_c5688b kernel: Killed process 9601 (uwsgi)

Giuseppe · February 3, 2017, 1:46pm

and anyway i can't still upgrade the AP also from cnmaestro

Chris-T · February 5, 2017, 7:36am

Ok try this, strange yes but may work.

Had to do this a few times and this seems to work for me.

Goto Monitor and Reset Stats

Once done goto cnMaestro and select reboot. (this generally does not reboot it, but seems to be a step that is required, tried skipping this a few times and didnt work without this step) Odd huh.

Now go directly back to the device and hit the reboot button on the Web UI.

Device should now reboot.

Now update.

Seems to work for me everytime inb this order.

Giuseppe · February 5, 2017, 6:18pm

after the reboot of AP the syslog show this , so now the AP don't connect the cnmaestro (AP is 2.6.1 )

Sep 1 00:01:13 ePMP1000_c5688b kernel: Killed process 1662 (nginx)
Sep 1 00:01:16 ePMP1000_c5688b kernel: Out of memory: kill process 1660 (nginx) score 85 or a child
Sep 1 00:01:16 ePMP1000_c5688b kernel: Killed process 2503 (nginx)
Sep 1 00:02:01 ePMP1000_c5688b DEVICE-AGENT[1815]: EINPROGRESS in connect()
Sep 1 00:02:01 ePMP1000_c5688b DEVICE-AGENT[1815]: Got 'X509_verify_cert failed' for hostname '172.16.14.3' and certificate:
Sep 1 00:02:01 ePMP1000_c5688b DEVICE-AGENT[1815]: server's cert didn't look good 29
Sep 1 00:02:01 ePMP1000_c5688b DEVICE-AGENT[1815]: Not able to find cnMaestro, Try Discovering again
Sep 1 00:02:01 ePMP1000_c5688b DEVICE-AGENT[1815]: Discovery of cnMaestro Failed
Sep 1 00:02:01 ePMP1000_c5688b DEVICE-AGENT[1815]: Unable to discover cnMaestro

Chris-T · February 6, 2017, 1:15am

So after the reboot you still cant update?

Douglas_Generous · February 6, 2017, 10:21pm

This will be a stupid idea, but I had a similar issue with bothe 2.6.1 and 3.2 with cnMeastro.

Disable cnMeastro in the AP and then ensure cnMeastro is offline (i use VM's for this so I just shutdown the VM)

now go to the AP in a fresh browser and clear the stats and log info. pull a page refresh by deleting the browser cache and refreshing the page (dont know why but the AP's required this of me) and know you should be able to reboot the APs and on reboot they will take an update. Please ensure you step through the correct versions to get to the current 3.2.2 on your units. If this doesnt work then a long drive to the units is basically the only way to fix this. take a spare set of APs as its faster generally to swap out than to fight with them.

Giuseppe · February 7, 2017, 2:21am

there is the solution.....

so,

- 1st step

change the name of SSID and security wpa password

- 2nd step

reboot

- 3th

upgrade the AP

_______________________________

i think maybe the problem is cpu or ram at 100% or similar ?

Douglas_Generous · February 8, 2017, 2:54am

My problem was cnMeastro interfering with the local update. As soon as i stepped through removing its control i had no issues.

I have noticed that during an update the epmp radios dont report the correct values for practically any OID that is polled during the upload and update process. I have sofar just ignored that since writing firmware to nvram/flash while running exclusively from ram is a heavy task and all things return to normal after the write is done and its waiting to reboot.