CBRS Failures this morning

This morning we have been seeing a number of these 504 errors on APs across our network. Federated is our SAS and cnMaestro cloud is how we are interfacing. There have been no reachability issues between cnMaestro and the gear, so I can’t help but think it’s on the CN->SAS interface side.

Anyone else seeing the same?

I can’t open support tickets again via the support portal, so haven’t opened one yet, but that is an unrelated issue…


08/31/2021 : 10:12:46 PST : Transmit time expired hence transitioning to granted state 
08/31/2021 : 10:13:05 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:13:46 PST : Processing Transmit expiry timeout 
08/31/2021 : 10:14:12 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:15:13 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:15:59 PST : Enabling transmission on radio 
08/31/2021 : 10:19:34 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:19:40 PST : Heartbeat request connection timed out, state = CBSD_HEARTBEAT_TX 
08/31/2021 : 10:22:31 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:22:36 PST : Heartbeat request connection timed out, state = CBSD_HEARTBEAT_TX 
08/31/2021 : 10:27:47 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:27:53 PST : Heartbeat request connection timed out, state = CBSD_HEARTBEAT_TX 
08/31/2021 : 10:28:23 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:28:42 PST : Transmit time expired hence transitioning to granted state 
08/31/2021 : 10:28:43 PST : Enabling transmission on radio 
08/31/2021 : 10:36:00 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:36:05 PST : Heartbeat request connection timed out, state = CBSD_HEARTBEAT_TX 
08/31/2021 : 10:41:19 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:41:24 PST : Heartbeat request connection timed out, state = CBSD_HEARTBEAT_TX 
08/31/2021 : 10:51:19 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:51:24 PST : Heartbeat request connection timed out, state = CBSD_HEARTBEAT_TX 
08/31/2021 : 10:51:55 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:52:14 PST : Transmit time expired hence transitioning to granted state 
08/31/2021 : 10:52:30 PST : SAS Error HTTP 504: Request to SAS timed out 
08/31/2021 : 10:53:14 PST : Processing Transmit expiry timeout 
08/31/2021 : 10:53:25 PST : Enabling transmission on radio

We are getting the same errors.

Same… looks like things started to return to normal around noon-ish PST for us.

We are actually still getting the 504 errors.

Scratch that… we’re still having some AP’s yo yo still… and now Ubiquiti is reporting problems with their UNMS and AWS services having issues.

Great… looks like it’s just Oregon still having AWS issues:

Things seemed to resolve for us around 2PM Pacific in Northern CA.

This is a prime example of why none of our commercial customers are on CBRS gear at this point. It’s no fault of Cambium’s, but there are still too many single potential points of failure in the CBRS ecosystem. It really isn’t acceptable that an AWS outage causes complete connectivity failures for our customers.

I’m really hopeful that these are just growing pains and things will keep improving over time.