Is anyone else having an issue with the SAS I’m getting a 504 Error on all APs
We had the same issue for all of our APs, using Federated, then they all came back up while we were on the phone with Cambium support checking on the outage.
Yeah, it was a Federated SAS outage. Logs showed Cambium Domain Proxy responding with 504 Bad Gateway.
1 Like
Saw the same error, will know what it means for next time!
Did your APs and SMs come back automatically, or were they stuck in Grant_Suspended state? How long did you wait, or what manual steps did you take to get them back online?
I think we need a recommended procedure from Cambium for getting APs and SMs out of the Grant_Suspended state. I know we were in chaos here, reduced to trying random s**t. Relinquish grant, deregister, reboot, from AP sessions page or from cnMaestro, even had to get one customer to power cycle their SM. It was not a good situation. I can’t imagine if we had 10,000 CBRS customers.
Is it possible that either cnM Cloud or Federated SAS was overloaded after the outage, processing thousands of CBSDs renewing their grants all at once?
I remember Federated supposedly did an order of magnitude increase in their computing power after a previous outage, maybe cnMaestro Cloud got overloaded? Things certainly seemed sluggish and flakey for awhile.
All but one came back on its own. The big problem is Federated does not immediately authorize new devices so you can easily get stuck in grant suspended until CPAS. This means that this isn’t exactly as viable for commercial services as everyone has been lead to believe.
It will get better with time for most areas but if you are near an active naval base by the coast you are going to have a lot of strange DPAs and I’m not sure how much better this can be made in my experience the full lower 100Mhz will be suspended rather than the in-use channels only.
I saw that the outage was supposedly 15 minutes. How long after that did your APs and SMs come back up? I spent 2-3 hours trying to get everybody back up.
CPAS has never been an issue here. We’re in north central Illinois, in the middle of corn fields. No coasts, no naval bases.
it took about 10 minutes after the outage for most of it to come back up then until CPAs for the SMs that went into grant suspended.
So some of your SMs had to wait overnight?
Once in IAP Pending your stuck until CPAS
Did you see the status as IAP Pending in cnMaestro or the device itself? Or you had to infer that?
I’ve never seen that as a status.
So you’re saying at least one of your SMs was down until the next morning?
Yes more than one.
This is what’s shown when you mouse over the little i next to the grant suspended msg in cnMaestro with a new CBSD or Deregister and re-attempt to get the grant if your transmit expiry has been processed
Thanks. I don’t think I saw that, and everything’s back up now so I can’t check. But next time I know what to look for.
It was bad enough taking calls from the “Zoom moms” (what I call the moms with K-12 kids doing remote classes) for a couple hours. I can’t imagine having it continue until the next morning, through an entire evening with no Netflix. Oh, the humanity!
1 Like
We had around 18-22 min outage on our entire CBRS deployment caused by this 504 outage.
Does anyone have a good way to diagnose where an issue is?
Is there a quick status /connectivity page on federated / cambium somewhere?
Right now it’s an all-hands-on-deck situation along with questions from all sides on where the outage is originating.
None of the reports on the Federated SAS webpage (https://spectrum.federatedwireless.com/login) seemed to be quick enough with a 3 hour granularity.
2 Likes