[ROOT CAUSE IDENTIFIED] cnPilot E410 / E-Series – Device Agent 3.47 malformed HTTP headers cause HTTP 400 with Sophos WAF – Firmware 4.2.3.3-r9/r10

Products Enterprise Wi-Fi Networks
1

[ROOT CAUSE IDENTIFIED] cnPilot E410 / E-Series – Device Agent 3.47 sends malformed HTTP headers, causing HTTP 400 behind Sophos WAF

Products: Cambium cnPilot E410 / E-Series, cnMaestro On-Premises
Firmware tested: 4.2.3.3-r9 and 4.2.3.3-r10
Device Agent: 3.47
Reverse proxy: Sophos XGS Firewall, SFOS 22.0.2 MR-2
Status: Root cause identified and independently reproduced. Direct connectivity works. Official firmware correction pending.

Overview

We would like to share the results of an extensive investigation into a cnPilot E410 onboarding failure when connecting to cnMaestro On-Premises through a Sophos WAF reverse proxy.

After testing certificates, DNS, firewall policies, TLS, WebSocket connections, firmware updates, and ultimately capturing the original HTTP request generated by the access point, we identified a reproducible HTTP/1.1 formatting defect in the Cambium Device Agent.

The Device Agent terminates its HTTP discovery request headers with \r\n\n instead of the expected \r\n\r\n.

The Sophos reverse proxy rejects this request with HTTP 400 Bad Request, while the cnMaestro backend accepts it when the AP connects directly.

We independently reproduced the behavior by sending two otherwise identical HTTPS requests to Sophos and changing only one byte.

This post documents the investigation, findings, reproduction procedure, and available workarounds.

1. Environment

Our deployment consists of approximately 100 Cambium E-Series access points being migrated to cnMaestro On-Premises. Some APs are located at remote sites and must connect through a public HTTPS endpoint.

Access point

Component Value
Model Cambium cnPilot E410
Firmware originally installed 4.2.3.3-r9
Firmware after upgrade 4.2.3.3-r10
Device Agent 3.47 on both releases
Device type reported by agent Sage
Connection protocol HTTPS and WebSocket
cnMaestro discovery endpoint /cns-onboarding/device
WebSocket endpoint /device

Controller and firewall

Component Value
Controller cnMaestro On-Premises 6.3.0-r1
Platform Linux virtual machine on Microsoft Hyper-V
Controller HTTPS port TCP 443
Firewall Sophos XGS2100
Firewall firmware SFOS 22.0.2 MR-2 Build 546
Publication method Sophos WAF / reverse proxy
Reverse proxy protocol HTTPS
WebSocket passthrough Enabled

The intended connection path is:

Remote / Internal cnPilot E410
          |
          | HTTPS TCP 443
          v
Public cnMaestro hostname
          |
          v
Sophos XGS WAF reverse proxy
          |
          | HTTPS TCP 443
          v
cnMaestro On-Premises

During initial troubleshooting, Cloudflare was also part of the public HTTPS path. It was subsequently disabled (DNS Only) to isolate the Sophos reverse proxy.

2. Initial symptoms

The E410 successfully resolved the cnMaestro hostname, established a TCP connection, and completed TLS negotiation.

However, discovery immediately failed.

The Device Agent reported:

SSL_connect successful

Server certificate is verified and it is valid

SSL write 282 bytes

Received 432 bytes

HTTP content-length is 266

Unexpected response:
JSON error line:1 text: '[' or '{' expected near '<'

Not able to find cnMaestro, Try Discovering again

Discovery of cnMaestro Failed

The JSON error appeared because the Device Agent received an HTML error page instead of the expected discovery response.

The corresponding Sophos log showed:

srcip="<AP_INTERNAL_IP>"
method="GET"
statuscode="400"
url="/cns-onboarding/device"
server="cnmaestro.example.com:443"
ctype="text/html"
uagent="-"
ruleid="-"

An important observation was that the request was rejected without normal association with the Sophos WAF publication rule.

There was also no corresponding successful forwarded request in the cnMaestro Nginx access logs.

At this point, the evidence suggested that the HTTP 400 was generated by the intermediate reverse proxy before the backend application processed the request.

3. First isolation test: bypassing the WAF

We configured the E410 to connect directly to the internal IP address of the cnMaestro server:

https://<CNMAESTRO_INTERNAL_IP>

The result was completely different.

The discovery endpoint returned:

Found http 302 status code
Found Location Header
HTTP content-length is 26

The Device Agent then discovered the WebSocket endpoint:

https://<CNMAESTRO_INTERNAL_IP>:443/device

The WebSocket handshake succeeded:

LWS_CALLBACK_CLIENT_ESTABLISHED

Connected to cnMaestro (0)

send ping after 17 seconds

PING_DATA: len=29

The E410 subsequently appeared Online in cnMaestro when connected through the direct internal path.

This demonstrated that the AP and controller could communicate successfully without the HTTP reverse proxy.

It also showed that the problem was not simply a cnMaestro onboarding or AP credential failure.

4. Sophos WAF troubleshooting

We checked the relevant reverse proxy configuration:

  • HTTPS publication on port 443

  • Correct protected server address

  • Root path /

  • WebSocket passthrough enabled

  • Original Host header forwarding enabled

  • Correct public hostname and TLS listener

We also tested the publication without the configured WAF inspection policy and IPS protection.

The HTTP 400 persisted.

This was an important finding because it showed that the rejection did not depend on an ordinary ModSecurity/OWASP signature.

The request was being rejected at a lower HTTP protocol processing layer.

Comparing the AP with curl

We sent requests to the same Sophos public endpoint using Windows curl.

They returned HTTP 302.

We then tested various HTTP request differences.

Test Result
curl with hostname-based SNI HTTP 302
curl using IP address and explicit Host header HTTP 302
Host containing :443 HTTP 302
Missing User-Agent and Accept headers HTTP 302
HTTP/1.0 request HTTP 302
Actual E410 discovery request HTTP 400

This ruled out several initial hypotheses.

In particular, the Host: hostname:443 format and SNI were not sufficient to reproduce the failure.

We needed to examine the actual bytes sent by the Cambium firmware.

5. Capturing the original HTTP request from the E410

We configured a temporary HTTPS diagnostic listener on a Linux test port using OpenSSL.

This listener was used only with the laboratory AP.

Example:

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout /tmp/cambium-debug.key \
  -out /tmp/cambium-debug.crt \
  -days 1 \
  -subj "/CN=cnmaestro.example.com"

Then:

openssl s_server \
  -accept <SERVER_INTERNAL_IP>:8443 \
  -cert /tmp/cambium-debug.crt \
  -key /tmp/cambium-debug.key \
  -quiet -naccept 1

The AP was temporarily configured to use:

https://<SERVER_INTERNAL_IP>:8443

A secondary issue during this test

The cnMaestro Linux server had a default DROP policy in its local iptables INPUT chain.

Although OpenSSL was listening on TCP 8443, the connection initially timed out.

A packet capture showed repeated incoming SYN packets without SYN-ACK responses.

We temporarily allowed TCP 8443 specifically from the test AP, which resolved this local connectivity issue.

This was unrelated to the Sophos HTTP 400; it was only necessary to collect the diagnostic request.

Captured Device Agent request

The actual request appeared as follows, with sensitive values removed:

GET /cns-onboarding/device?mac=<REDACTED>&type=Sage&serialNo=<REDACTED> HTTP/1.1
cookie:deviceId=<REDACTED>
HOST:<SERVER_INTERNAL_IP>:8443
User-Agent:cnDA/1.0 (Sage/4.2.3.3-r9; DA/3.47)

At first glance, the request looked acceptable.

The unusual capitalization of HOST and the absence of spaces after header colons were not sufficient evidence of an HTTP violation.

The actual problem was in the invisible line-ending characters.

6. The key discovery: malformed HTTP header termination

We inspected the decrypted request at the byte level.

The analysis produced:

CRLF (0D 0A): 4

LF without CR: 1

Final bytes: b'3.47)\r\n\n'

First line ends with CRLF: True

The Device Agent ended its request like this:

User-Agent:cnDA/1.0 (...)\r\n\n

The expected HTTP/1.1 format is:

User-Agent:cnDA/1.0 (...)\r\n\r\n

Hexadecimal comparison

Observed in the E410 request:

... 33 2E 34 37 29 0D 0A 0A
                   ^^^^^^^^
                   CR LF LF

Expected:

... 33 2E 34 37 29 0D 0A 0D 0A
                   ^^^^^^^^^^^
                   CR LF CR LF

The difference is one missing 0D byte before the final 0A.

This makes the message’s header-section termination inconsistent with the standard HTTP/1.1 CRLF format.

HTTP implementations may be tolerant of LF-only line endings, but they are not required to accept them.

This could explain why the cnMaestro backend accepted the request while the Sophos reverse proxy rejected it.

7. Reproducing the Sophos HTTP 400 by changing one byte

This was the decisive test.

Using a Windows PowerShell script with a raw TLS socket, we sent two otherwise identical HTTP requests to the same Sophos WAF endpoint.

The first request ended with the correct HTTP header terminator:

\r\n\r\n

The second used the terminator observed in the Cambium Device Agent:

\r\n\n

Results

HTTP CORRECT: HTTP/1.1 302 Found

FORMATO E410: HTTP/1.1 400 Bad Request

Translated:

VALID HTTP: HTTP/1.1 302 Found

E410 FORMAT: HTTP/1.1 400 Bad Request

Changing only the header-section terminator was sufficient to reproduce the rejection.

This demonstrated that the malformed termination was not merely an unusual characteristic of the request: it directly triggered the observed HTTP 400 behavior in our Sophos reverse proxy.

Reproduction approach

The essential request construction was:

$path = "/cns-onboarding/device?mac=<TEST_MAC>&type=Sage&serialNo=<TEST_SERIAL>"

$req = "GET $path HTTP/1.1`r`n" +
       "HOST:${hostName}:443`r`n" +
       "User-Agent:cnDA/1.0 (Sage/4.2.3.3-r9; DA/3.47)`r`n" +
       $terminator

We performed the two tests with:

$terminator = "`r`n"  # Valid

and:

$terminator = "`n"    # E410 behavior

Both requests used the same destination IP, hostname, HTTPS/TLS connection method, endpoint, and headers.

The only intended change was the final line-ending sequence.

8. Firmware upgrade did not resolve the issue

We initially tested firmware:

4.2.3.3-r9
Device Agent 3.47

We then upgraded the laboratory E410 to:

4.2.3.3-r10
Device Agent 3.47

The Device Agent version remained unchanged.

After the upgrade, the public discovery attempt still produced:

sw_vers [4.2.3.3-r10]

user_agent [cnDA/1.0 (Sage/4.2.3.3-r10; DA/3.47)]

SSL_connect successful

SSL write 283 bytes

Received 432 bytes

HTTP content-length is 266

Unexpected response:
JSON error line:1 text: '[' or '{' expected near '<'

Discovery of cnMaestro Failed

Therefore, upgrading from r9 to r10 did not resolve the observed Sophos discovery failure.

We captured the malformed HTTP bytes directly on r9. On r10, we confirmed the continued failure and unchanged Device Agent version.

9. Additional firmware analysis

We downloaded the original E410 firmware package:

E410_E430_E510_E600_E700-4.2.3.3-r10.img

The file was not a conventional mountable disk image.

Its outer container started with:

-----BEGIN PKCS7-----

We extracted the embedded content and identified a Cambium firmware manifest.

Further examination revealed a Linux filesystem archive and the Device Agent executable:

/usr/sbin/device-agent

The executable is an ARM 32-bit ELF binary.

Static analysis identified HTTP request formatting strings consistent with the malformed header termination observed in the AP’s traffic.

We also identified a firmware update verification mechanism based on PKCS#7 signatures.

No modified firmware was installed on the AP.

An experimental binary-level correction was investigated separately, but it has not been validated on actual hardware and is not an official or recommended production fix.

Because the firmware update mechanism verifies digital signatures, modifying and repackaging the original firmware is not a practical supported upgrade method.

We strongly recommend an official Cambium firmware correction instead.

10. Temporary workaround

For internal testing, the successful workaround was straightforward:

Allow the E410 to reach cnMaestro directly over HTTPS, without an intermediate HTTP reverse proxy.

In our environment:

E410
  |
  | HTTPS TCP 443
  v
cnMaestro On-Premises

Result:

HTTP discovery: Successful
WebSocket: Established
Device status: Online

For remote APs, the following architecture is technically appropriate to evaluate:

Remote E410
  |
  | HTTPS TCP 443
  v
Public IP / Firewall
  |
  | Layer-4 TCP forwarding / DNAT
  | No HTTP parsing
  | No TLS termination
  v
cnMaestro On-Premises TCP 443

This avoids HTTP parsing by the intermediary while preserving the original HTTPS session between the access point and cnMaestro.

Important: We have verified direct internal connectivity. A dedicated public Layer-4 DNAT configuration remains a proposed workaround and still requires pilot validation in our environment, especially for long-lived WebSocket sessions.

For security, this architecture should use an appropriate public certificate, restricted management access, necessary firewall controls, and a dedicated public IP or equivalent isolated publication.

We do not recommend globally disabling strict HTTP validation on an internet-facing reverse proxy.

11. Separate observation: Cloudflare and WebSocket

During our earlier tests, Cloudflare was enabled in front of Sophos.

In that configuration, initial HTTP discovery sometimes succeeded, but the subsequent WebSocket connection to:

/device

was not stable.

The cnMaestro Nginx logs showed HTTP 101 switching-protocol responses, while the sessions ended prematurely.

The E410 eventually logged messages such as:

lws_connect_timeout_cb: Unable to connect websocket

This is a separate interoperability symptom.

We have not established that the malformed HTTP discovery terminator is the cause of the Cloudflare/WebSocket issue.

We disabled Cloudflare proxying to isolate and reproduce the Sophos HTTP 400 described in this post.

12. Common troubleshooting approaches that did not fix the issue

We spent considerable time investigating possible causes before capturing the original HTTP bytes.

The following did not fix the Sophos HTTP 400:

  • Replacing or investigating TLS certificates

  • Changing DNS resolution while retaining the Sophos WAF path

  • Disabling WAF inspection policies

  • Disabling IPS protection for the test publication

  • Modifying OWASP exclusions

  • Changing SNI behavior in synthetic client tests

  • Removing User-Agent and Accept headers in synthetic requests

  • Using HTTP/1.0 in synthetic requests

  • Sending the hostname with an explicit :443 suffix

  • Upgrading the E410 from 4.2.3.3-r9 to 4.2.3.3-r10

We also observed some unknown-msn and invalid-msn registration events during earlier onboarding experiments. These were not evidence of the cause of the HTTP 400 at the Sophos ingress layer.

The decisive step was inspecting and reproducing the exact HTTP header terminator sent by the Device Agent.

13. Request to Cambium Networks engineering

We have opened a Cambium technical support case and provided the investigation results.

The initial automated support response acknowledged that the observed header termination is malformed and that the evidence warrants engineering validation. Human engineering confirmation and an official fix are still pending.

We would appreciate clarification on the following:

  1. Is the malformed HTTP discovery header terminator a known issue in Device Agent 3.47?

  2. Does it affect other cnPilot E-Series devices using the same Device Agent implementation?

  3. Is there an officially supported firmware containing an updated discovery HTTP request builder?

  4. Could Cambium release a hotfix for firmware 4.2.3.3-r10 or a later supported E-Series release?

  5. Is there a supported method to validate a corrected Device Agent on laboratory hardware without modifying the signed firmware package?

The expected code-level correction appears to be small: ensure the HTTP discovery request header section ends with CRLF CRLF rather than CRLF LF.

However, the exact implementation change should be reviewed and released by Cambium engineering.

14. Final findings

The investigation produced the following conclusions:

Finding Status
TCP connectivity to the public Sophos endpoint Confirmed
TLS handshake with Sophos Confirmed
Sophos HTTP 400 during AP discovery Confirmed
Successful direct HTTP discovery to cnMaestro Confirmed
Successful direct WebSocket connection Confirmed
E410 Online through direct internal path Confirmed
Malformed \r\n\n header termination in captured r9 request Confirmed
Same malformed termination independently reproduces Sophos HTTP 400 Confirmed
Firmware r10 resolves the discovery failure No
Device Agent upgraded in r10 No, remains 3.47
Official Cambium firmware fix Pending
Public Layer-4 DNAT workaround Proposed, not yet validated

Key takeaway

If a cnPilot E-Series AP establishes TLS successfully but fails cnMaestro discovery with HTTP 400 behind a strict HTTP reverse proxy, inspect the original request’s header-section terminator before spending time changing certificates, DNS, or WAF policies.

In our E410 / Device Agent 3.47 environment, the request ended with:

0D 0A 0A

instead of:

0D 0A 0D 0A

We demonstrated that this single-byte difference changes the Sophos response from HTTP 302 to HTTP 400.

We hope these findings save other Cambium administrators time troubleshooting similar deployments.

If anyone has experienced the same issue with E410, E430, E510, E600, or E700 devices, or has received an official Device Agent hotfix, we would appreciate your feedback.

IT Infrastructure / Security Team
Maza Tarraf – Brazil

1 Like