Epmp hacked? unexplainable traffic/malware?

[Ryo_Haines]

We have something spreading through our EPMP network and can’t seem to narrow down what it is.
We have UBNT devices as well that do not seem to be affected at this time.

Somehow the traffic seems to be redirecting on the client devices on the other side of the router, all traffic is being redirected. They disconnect from the network, cellular network works fine.
Any website they browse to redirects them to 4 or 5 different pages, for example dussbn.lightexercisbroad.xyz.
We have been resetting the radios and it seems to fix it for a day or so and then it happens again.
We’ve upgraded firmware, changed passwords, and the radio seems to still be redirecting traffic.
We’ve confirmed there are different routers on the client side, so wouldn’t think it’s the client router.
Common variable seems to be EPMP radios.
Has anyone seen this or have any explanation as to why or how this is happening?
Thoughts?

Thanks in advance,

Ryo

[ninedd]

Sounds like this - which Cambium hasn’t commented on yet that I can see

Malware/Virus on Our Cambium system

I work at small WISP. A mix of Cambium and Ubnt. Our Cambium customers(force 180s) have gotten hit with Malware (5 billionth search) After eliminating everything device, I found that reset/reconfig of radio, would fix problem. It seemed to work, for a couple of days. Now I’m starting to get calls about problems coming back. Any suggestions

[brubble1]

Nice… we have about 200 F180’s on our network…

[brubble1]

Please tell me they are all F180’s like the guy ninedd linked !

Are you using the default SNMP strings ?

[Ryo_Haines]

Nope, seems to be 1000’s or 180’s.
Well figured out why the radios are doing it, just havent figured out how.
They are updating the client side dns setting.
Changing the password doesn’t seem to help, so something is changing the radios another way?

What about the snmp string?
I thought that was read only, can something change a setting through default settings without a password?

Thanks,
Ryo

[Ryo_Haines]

Going through our radios there seems to be a lot that have been hit with whatever this is.
Does anyone know what variable or macros are available to change the ‘DHCP DNS IP address’ setting through cnmaestro via a template?
I can start changing them one by one, but would much rather apply a template because from the looks of it will just happen again.
Thanks in advance for the assistance,
Ryo

[brubble1]

Nope, seems to be 1000’s or 180’s

Wonderful… just wonderful…

What about the snmp string?

In the radio Configuration > System > Simple Network Management Protocol

There is a Read-Only string and a Read-Write string.

I can start changing them one by one

Seems to me that if the radios get re-hacked after being defaulted then going through and changing the DNS settings won’t fix will it ? Won’t the hack just change it back like it does when you default/reconfigure ?

Edit: So I went looking for DNS traffic on our network to/from those two IP addresses and did not find them , and this made me happy. But then I started monitoring all DNS and filtered out all that I recognized (google, hurricane, cogent, opendns etc… etc…). Wasn’t a lot of them that looked suspect so I started hunting down each radio and logging in to see if the DNS had been changed.

I found one on our network An old force 180 that was installed in 2015 and I don’t have any records of us ever replacing it so hopefully at some point over the years some setting got defaulted by a software upgrade or maybe it escaped a config change over the years… A guy can hope can’t he ?!?

I contacted the customer, he said for the last week or so their Apple devices had not been able to access any secure websites and while their TV’s/Rokues worked their windows / android devices were bringing up Spam sites when they would try to bring up web pages.

I blocked their radio at the AP network and I’m going out to replace it with a 300-16 right now and hope the 300-16 isn’t vulnerable.

The DNS settings on this radio was
194.9.70.168
45.129.97.29

Just so it’s searchable I’m listing the DNS addresses you screen capped.

31.172.78.104
87.236.146.187

Not that it matters but all 4 IP addresses tracert/whois to Russia and Ukraine.

[ninedd]

It’d be good to get some comment from Cambium - this is a pretty critical situation.
I’m not sure who to tag exactly?
@Fedor @Chinmay_K @Simon_King

[brubble1]

Ok I have the compromised radio on the bench. I saved the config on it and compared it to an automated backup config pulled off of it a few weeks ago.

There were 6 changes to the config. The customer had called into heldesk a few days ago so it is possible that they changed the UPNP and IPV6 setting… I don’t know why they would (especially the IPv6) but it is possible. Also firmware was upgraded from 4.6.1 to 4.6.2 after the backup was made.

Hacked radio on Left backup config on the Right

“networkBridgeDNSIPAddrPrimary”: “194.9.70.168”,
“networkBridgeDNSIPAddrSecondary”: “45.129.97.29”,

“networkLanDNSIPAddrPrimary”: “194.9.70.168”,
“networkLanDNSIPAddrSecondary”: “45.129.97.29”,

“networkUPNP”: “1”,

“systemConfigIPv6Support”: “1”,

Edit: Oh and radio does not use default passwords for anything, snmp read and write are not default, uses a separate management interface that can only be access via VPN from the wireless interface side but can be accessed from any IP on the LAN side… so I’m thinking/hoping at this point the hack comes from something on the customer’s side of the radio. Seems to be somewhat supported by the fact the it has been on this customer’s radio for about a week but does not appear to have spread to any other radios on the network.

I replaced the 180 with a 300-16 that I will be watching very closely.

Also blocking those 4 addresses.

[brubble1]

I’m not an expert on hacking but it seems to me that in order to add very specific settings to the config that were not initially in the config (the bridge DNS setting) this has to be specifically designed for ePMP.

Also, I would think that it’s some kind of backdoor/injection type changes because… why wouldn’t they change the passwords to prevent the ISP from easily just changing the DNS settings back and possibly blocking further access ? At least I would think that whatever process they are using (we do not use default SNMP strings and I checked this radio’s settings to make sure it wasn’t) is one that can not change the passwords or this would be farrrrr worse for anyone with this spread across their network.

[brubble1]

See if you can log into your radios with
Username: installer
Password: installer

The Installer account on this compromised radio is enabled and using the default installer Password! We didn’t start using / enabling the installer account until a couple of years ago so if this is how they are getting access then WOOOHOOO for me because this is not enabled on the vast majority of our radios and shouldn’t be the default password on the ones it is enabled on.

Hopefully this is the only radio on our network that slipped through with an enabled/default Installer account.

This would explain why the password isn’t being changed, installer can change most of the settings in the radio but enabling / disabling accounts, changing account passwords or even changing SNMP strings are not among the things an installer account can change.

[ninedd]

@brubble1

See if you can log into your radios with
Username: installer
Password: installer

So, perhaps we all ‘just’ need to push a template to everything via cnMaestro to either change the installer password, or to disable the installer account then? I mean, it should already be – but incase there are virgin ones out there, that probably would mitigate it?

Here’s a ‘how to’ to create an push a template to change passwords via cnMaestro:

Automating ePMP Password Changes using cnMaestro

Some time ago, I posted a method for Automating PMP450 Password Changes with cnMaestro.  There were a few requests asking how to do the same thing with ePMP.  This post will describe the process to automate ePMP password changes. Create The Config String: The first step is to create a configuration string cnMaestro will push to the radios.  Since ePMP devices don’t require complete configurations, we can simply copy the cambiumSysAccountsTable from a device configuration file.  You can get the…

[brubble1]

I have no idea why the installer account would be enabled on any of our older radios since we never used the installer account… but I went back and was checking all the oldest customer radios and I was surprised to find that about 1 in 5 have the installer account enabled and using the default password on every one of the…

So at some point either the installer account was on by default or we were using a template that unknowingly had that account enabled. All of the radios installed in the last 2 years have the installer account enabled but the password is not default on them.

I just recently set up cnM on premises and was about to go looking for the exact info you just provided. Thanks !

Edit: Clarify , we never used the installer account on the old N radios and never intentionally enabled it. A couple of years ago we did start using it but we had moved to AC by then and we did not use the default password. The few radios I found on our network with the installer/installer un/pw were very old N radios so at some point way back in the day we must have had a template that had it enabled and we didn’t know it.

[ninedd]

@brubble1

was about to go looking for the exact info you just provided. Thanks !

On my first test, that example from Cambium seems to have reset the radio back to some default info (such as it’s device name), and caused a reboot in the process. SO, that might not be the best instructions (or maybe I did it incorrectly?)

[Ryo_Haines]

Wow, good catch brubble1.
The installer account is enabled and default password…
LOL, well that’s just kinda terrible, I’m surprised it took someone this long to hack it.
Dang…

So I don’t have to manually go in and disable every installer account on 100’s of radios anyone know how to do that with cnmaestro?
I guess I need to modify the client DNS settings and disable the installer account.
Any idea?

I saw the template the change the password, I guess I should at least do that like yesterday.
I’m surprised no one else has seen this, unless everyone already knew to change that password or disable that account. I guess I missed that memo…
Thanks for the feedback,
Ryo

[Eric_Ozrelic]

To create a new template that you can push out to devices onboarded into cnMaestro, first create a template:

Press the “Add Template” button on the right hand side, here’s an example:

You’ll want to use the following JSON script to disable all accounts other then the admin account:

{
	"device_props": {
		"cambiumSysAccountsTable": [{
				"cambiumSysAccountsName": "installer",
				"cambiumSysAccountsHash": "!$1$yHIXrXnB$tr\/Na771wSO4\/dv8yCvmU\/",
				"cambiumSysAccountsUID": "2000",
				"cambiumSysAccountsGID": "100",
				"cambiumSysAccountsDir": "\/tmp",
				"cambiumSysAccountsShell": "\/bin\/false"
			},
			{
				"cambiumSysAccountsName": "home",
				"cambiumSysAccountsHash": "!$1$\/Bv9wvkq$5lBZqqsoAJj4uYOhb7pOS0",
				"cambiumSysAccountsUID": "3000",
				"cambiumSysAccountsGID": "100",
				"cambiumSysAccountsDir": "\/tmp",
				"cambiumSysAccountsShell": "\/bin\/false"
			},
			{
				"cambiumSysAccountsName": "readonly",
				"cambiumSysAccountsHash": "!$1$zn4\/PlCl$1qP9PGmPChDwkDAgp5Qgw0",
				"cambiumSysAccountsUID": "4000",
				"cambiumSysAccountsGID": "100",
				"cambiumSysAccountsDir": "\/tmp",
				"cambiumSysAccountsShell": "\/bin\/false"
			},
			{
				"cambiumSysAccountsName": "dashboard",
				"cambiumSysAccountsHash": "$1$Y9oz6eRg$ysokBk9VZCGo4AN6WQgVF0",
				"cambiumSysAccountsUID": "5000",
				"cambiumSysAccountsGID": "100",
				"cambiumSysAccountsDir": "\/tmp",
				"cambiumSysAccountsShell": "\/bin\/false"
			}
		]
	}
}

After you’ve made the template, you can then push it out to ePMP radios (both AP and SM) that are onboard and communicating with cnMaestro. Click on ‘Monitor and Manage’, then select ‘System’ network entry, then click the ‘Configuration’ tab near the top middle of the console. Set the device type to ePMP, and then select the newly created template. Then select the ePMP radios you’d like to push the change out to:

Scroll to the bottom then you can apply the changes now or schedule them to be applied later:

This has been tested with e1k/e3k radios on 4.6.2… but it should work with all of them. UPDATE! For some reason it seems as though it can cause some AP’s to reboot. I have not had this script reboot any SM’s yet.

[Ryo_Haines]

Well I think I’ve figured it out.
I’ve applied it to a few devices and I think it’s doing what it’s supposed to be doing.
Here is the just:

{
  "device_props": {
    "networkBridgeDNSIPAddrPrimary": "8.8.8.8",
    "networkBridgeDNSIPAddrSecondary": "208.67.222.222",
    "networkLanDNSIPAddrPrimary": "8.8.8.8",
    "networkLanDNSIPAddrSecondary": "208.67.222.222",
    "cambiumSysAccountsTable": [
      {
        "cambiumSysAccountsName":    "admin",
        "cambiumSysAccountsHash":    "insert-hash-here",
        "cambiumSysAccountsUID":     "1000",
        "cambiumSysAccountsGID":     "4",
        "cambiumSysAccountsDir":     "\/tmp",
        "cambiumSysAccountsShell":   "\/usr\/bin\/clish"
      },
      {
        "cambiumSysAccountsName":    "installer",
        "cambiumSysAccountsHash":    "insert-hash-here",
        "cambiumSysAccountsUID":     "2000",
        "cambiumSysAccountsGID":     "100",
        "cambiumSysAccountsDir":     "\/tmp",
        "cambiumSysAccountsShell":   "\/bin\/false"
      },
      {
        "cambiumSysAccountsName":    "home",
        "cambiumSysAccountsHash":    "insert-hash-here",
        "cambiumSysAccountsUID":     "3000",
        "cambiumSysAccountsGID":     "100",
        "cambiumSysAccountsDir":     "\/tmp",
        "cambiumSysAccountsShell":   "\/bin\/false"
      },
      {
        "cambiumSysAccountsName":    "readonly",
        "cambiumSysAccountsHash":    "insert-hash-here",
        "cambiumSysAccountsUID":     "4000",
        "cambiumSysAccountsGID":     "100",
        "cambiumSysAccountsDir":     "\/tmp",
        "cambiumSysAccountsShell":   "\/bin\/false"
      }
    ]
  }
}

Looks like they also changed the network bridge DNS, so I updated that as well.
I tried to figure out how to disable all of the accounts except the admin, but don’t see anything that changes when I toggle the setting in the config. That would be my preference, but hopefully just changing the password for now will block the issue at hand.
Fingers crossed, hopefully this works.
Thanks,
Ryo

[Ryo_Haines]

Oh, just saw your post Eric…
If that works for disabling the accounts I’ll try that also.
Thanks,
Ryo

[iBound]

So this is not actually a hack, but rather a default installer password allowing the settings on the SM to be changed? As basic rule I disable everything except admin login on my SM’s. If using cnPilot R-Series then change pass for useradmin and user to prevent users from being able to see network config on our side.

Re the template, I’d love to be able to just push an manual config change to devices. Setting up a template and sync’ing things is too much of a commitment for me now

[aka (Andrii)]

NOTICE
Cambium Networks acknowledge persistence of the issue with unauthorized configuration changes. Primarily DNS settings are affected but other settings can be affected too.

Issue can be observed with all firmware versions.

Presumably the attack is performed via default user passwords and snmp communities.
Workaround: change all default passwords and snmp communities.

We are grateful for all your feedback! Investigation is in progress.