PMP450 Subscriber units freezing and requiring power cycle at customers premise on 15.1.5

Hi,

Just wondering if anyone has had issues with customers subscriber units freezing with no data traffic (subscriber however is still in session with the AP).

The subscriber is showing in session from the AP, however the subscriber has no traffic and is unable to be reached through the AP or via SNMP.

The only way to recover is to power off the subscriber and then power on, after which the susbscriber unit then begins working fine again.

At first there was only one or two - but we are up to alot more now and seems to be more regular across the network. There seems to be no common timing and happens on random units across the network - it has happened on PMP450 and PMP450b units. It occurs with both 450i and 450m APs. All running 15.1.5.

Is anybody else seeing these issues on 15.1.5?

1 Like

I have not heard of this yet, but have asked engineering to look into it.  They may want to retrieve some diagnostics from affected radios if possible.

Any chance it could be this? (From 15.1.5 release notes).  450b/450i SM CPY-14170: Occasionally traffic will stop passing on a VC in the downlink to 450b or 450i SM. The link may autorecover after 5 minutes by dropping the session. Otherwise the session will need to be manually dropped at the AP. This will be fixed in a future release.

Not sure.  What might tell us more is the logs from a radio that is experiencing the problem.

If you want, you can send them to me directly: matt @ cambiumnetworks <dot> com 

It is also a good idea to open a support ticket so that this issue gets properly logged and tracked.

1 Like

Hi,

Is there a way to retrieve the logs after a hard power off /on from the subscriber unit? We have checked logs on the subscriber units after (cannot access when issue is occurring) but there seems to be no history in the web GUI . Happy to send any logs from AP as required. Just let us know what you need.

Regards,

I recently bumped another thread, same issue has been occurring for us since moving from 14.2 to 15.1.5.  Radios are plain old 450 models.

Dropping the session fixes the customer and we can still get logs if I recall.  It's still sporadic, with no rhyme or reason though.

Edit:  To drop the session from the AP, go to Tools->Sessions, find the SM in the drop down list and then hit the drop session button.  Essentially, it breaks the RF link (which is still active) and the forced reconnect restores the logical link at the same time.

Edit again:  Sorry, NOT 15.1.5, 15.1.3 in our case.

2 Likes

Ok - thank you for that. We had one freeze just now and managed to get it back on by dropping the session as you suggested.

We did see these in the subscriber logs which seemed strange -

01/01/2016 : 00:00:00 UTC : :LOCK_NET_HEAP failed error 4
01/01/2016 : 00:00:00 UTC : :sm_v() OSSemPost failed, err 0, semaphore id 0, task ROOT, priority 46

And -

01/01/2016 : 00:00:00 UTC : :Intentional RF Shutdown.
01/01/2016 : 00:00:28 UTC : :Poked the value 00000000 to address ff201a10
01/01/2016 : 00:00:28 UTC : :Poked the value 00000000 to address ff201a4c
01/01/2016 : 00:00:28 UTC : :Poked the value 00001fc0 to address ff201a58
01/01/2016 : 00:00:28 UTC : :Poked the value 80000000 to address ff200008
01/01/2016 : 00:00:29 UTC : :Poked the value 3c030000 to address ff201064
01/01/2016 : 00:00:34 UTC : :Poked the value 00000080 to address ff201a10
01/01/2016 : 00:00:34 UTC : :Poked the value 00000000 to address ff201a4c
01/01/2016 : 00:00:34 UTC : :Poked the value 00001fc0 to address ff201a58
01/01/2016 : 00:00:34 UTC : :Poked the value 80000000 to address ff200008
01/01/2016 : 00:00:35 UTC : :Poked the value 3c030000 to address ff201064

etc, etc, etc ....

? No idea.... ?


@arobo wrote:

Ok - thank you for that. We had one freeze just now and managed to get it back on by dropping the session as you suggested.

We did see these in the subscriber logs which seemed strange -

01/01/2016 : 00:00:00 UTC : :LOCK_NET_HEAP failed error 4
01/01/2016 : 00:00:00 UTC : :sm_v() OSSemPost failed, err 0, semaphore id 0, task ROOT, priority 46

And -

01/01/2016 : 00:00:00 UTC : :Intentional RF Shutdown.
01/01/2016 : 00:00:28 UTC : :Poked the value 00000000 to address ff201a10
01/01/2016 : 00:00:28 UTC : :Poked the value 00000000 to address ff201a4c
01/01/2016 : 00:00:28 UTC : :Poked the value 00001fc0 to address ff201a58
01/01/2016 : 00:00:28 UTC : :Poked the value 80000000 to address ff200008
01/01/2016 : 00:00:29 UTC : :Poked the value 3c030000 to address ff201064
01/01/2016 : 00:00:34 UTC : :Poked the value 00000080 to address ff201a10
01/01/2016 : 00:00:34 UTC : :Poked the value 00000000 to address ff201a4c
01/01/2016 : 00:00:34 UTC : :Poked the value 00001fc0 to address ff201a58
01/01/2016 : 00:00:34 UTC : :Poked the value 80000000 to address ff200008
01/01/2016 : 00:00:35 UTC : :Poked the value 3c030000 to address ff201064

etc, etc, etc ....

? No idea.... ?


arobo,

Would you grab a CNUT capture of the radio with these logs for me to analyze? Then email it to me at charlie dot galik at cambiumnetworks.com? 

Yes, as mentioned above, we are aware of the internal issue number CPY-14170 that can cause stuck session. We were not aware it was affecting 450 units however. This issue is fixed and in the next 15.2 beta, 15.2 (Build BETA-4) that should be available soon.

The "Poked the value" is just the initial radio calibration so is completely normal at the beginning of the log. I think I understand the "ROOT task semaphore issue as well, but want to confirm. This is fixed in next 15.2 beta, 15.2 (Build BETA-4). Neither of these issue should cause stuck sessions. 

Charlie

2 Likes

Charlie,

I have sent you a CNUT capture file.

Please let us know if you find anything and solutions to fix the issues.

Thanks.

Just following up if any feedback on this issue? We did send through the required CNUT capture files as requested.

We will try the new beta and see how this goes ..

Thanks!

Any progress on this?  I've had to create a support ticket, just seeing way too many of these lately.  I'm tempted to roll back to 14.2 though I'd lose cnMaestro support.

I've also seen this happen twice now with two different subrscribers on our 450b High gain SM's. We use 450i AP's. I'm unable to access the web interface of the SM until I go to the AP and then drop the session, and then I'm instantly able to access the SM. Unforunately, we still have legacy 430 SM's so we're debating on whether to upgrade to 15.2 or not. 


@jw1 wrote:

I've also seen this happen twice now with two different subrscribers on our 450b High gain SM's. We use 450i AP's. I'm unable to access the web interface of the SM until I go to the AP and then drop the session, and then I'm instantly able to access the SM. Unforunately, we still have legacy 430 SM's so we're debating on whether to upgrade to 15.2 or not. 


Yes, this was a bug in 15.1.5 and fixed in 15.2.

15.2 does support 430 SM's as well (but DES no longer supported as discussed on other threads).