I am looking for some advice from someone more technically inclined. I have a couple of buildings connected using ePMP Integrated radios since the past 5 years. For the most part they ran without issues but lately I am facing a peculiar problem.
At 6pm everyday both the radios go offline and are not accessible via the internet or even via the LAN. They will then come back online the next day around 10am. I went through a lot of topics here on the community and discovered that my Firmware version was very old: 2.5.1 and so I upgraded both today to the latest firmware.
I also onboarded them onto the CN Maestro Cloud portal. But it appears to have made no difference. They are setup as EPTP Master and Slave and have a clear line of sight from the roofs of both buildings which are hardly 200 meters apart so no problems in that sense. They have been assigned Static IP addresses of 192.168.1.6 and 192.168.1.7 and the subnet is 255.255.255.0 and the gateway is set as 192.168.1.1.
What could be the problem? Am I missing something? Is some setting somewhere making them cycle off and on at a particular time everyday? Please help. any tips will be much appreciated.
When you say you canât access them via the LAN , is this both radios ? Meaning you go onsite, to both buildings and can not access either radio via the lan ? If you are accessing them remotely then how is âInternetâ different than âLANâ in this case ?
When you are able to get into the radios again, what is the âSystem Uptimeâ ? If you are not able to get into the radio until around 10am and the system uptime is only a few minutes then it would seem to indicate that the radio has been without power during this time (so someone turning off the power when they leave in the evening and turning it back on in the morning ? A timer that turns power on/off ?).
However if they system uptime shows the radio was powered on the entire time, then something else is going on and will need to know more about exactly how you are accessing the radio via the LAN vs Internet.
Yes it is both radios that become inaccessible. Even when on the LAN, you cannot connect to them either by 192.168.1.6 or 7 that is their assigned static IP addresses and also via the fallback IP address.
Iâve actually left a PC running at the AP site connected via ethernet to the router. I connect to this PC via Teamviewer and try and access them through the LAN that way.
I will check on the uptime tomorrow once they come back online or by physically visiting the site. I know that there is power because I was there yesterday and a few days prior to that when they went down and they were both showing lights on the radios. Iâll attach a pic that I took.
Iâll be happy to share more info or perhaps some screenshots tomorrow once theyâre back online. Iâm truly baffled by this behaviour as it tends to happen at exactly the same times every dayâŚ
Is that a picture of the radio you canât access via LAN and is it taken when the link is down ?
It shows Eth1 link so that likely kills the idea that the ethernet/router is being shut off / disabled.
It shows wireless link a full bars so that seems to remove noise or some kind of interference taking the link out during those times (and if it is the radio you are physically connected to the wireless link status wouldnât matter anyway.)
Possible that some light or other device is turning off/on on a schedule and the Cat cable runs near and it is killing the ethernet connection without actually taking the ethernet link down ? Can you plug into the radio with a temp cable, run a different route, and see if it works ?
What kind of router (or is it a switch ) is that radio plugged into ? Iâm not aware of the cambium radios having any scripting or other function that would allow you to do anything based on time of day so that would seem to suggest possibly something behind the radio, like the router it is plugged into, does ? For example we recently installed a free outdoor WiFi access for a local school and the Mikrotik router enables the wireless interface every morning at 7am and disables it every night at 7pm so wonder if there is a script or something running on the router your radio is plugged into that is set to enable/disable/change something on a schedule.
I think the next thing would require being onsite when the link is down and plugging your PC directly into the radio with no router between the PC and radio⌠if you still canât access the radio via the LAN then power cycle it, if you still canât reach via the LAN then change the power supply, if you still canât access the radio via the LAN then how hard would it be to connect directly to it with a different CAT cable ?
Brubble1, yes that is a pic of the SM whilst it was down. If I connect directly to the radio by using an Ethernet cable through the power supply I can connect to it. Will try it again today. I know that the Internet connection doesnât go down and that power is not lost, since there is a PC connected to the same power source on the AP side that remains accessible via TeamViewer and there is a CCTV system on the SM side that records without interruption. The AP is connected via Ethernet to a Huawei HG8145V5 router on which Iâve opened 800, 8000 and 554 ports to access the CCTV cameras via a DDNS (noip) service. There appears to be no other setting specified on the router and certainly nothing thatâs changed recently. I also checked if the ISPâs dynamic IP address change was affecting things but the IP address has not changed during this downtime.When you check the Ethernet port status on the router during this downtime, itâs shows the link to the AP as âDownâ but the other Ethernet port to the PC as âUPâ.j
Now by looking via cnMaestro I can see that the radios are cycling off for exactly 16 hours everyday. Just before 6pm and coming back just before 10am! Attached the screenshot of the cnMaestro showing this:
Iâve just added this setting to restart the ethernet under the Watchdog section, will this help at 6pm today? Or should I try changing it to device reboot?
OK so you said you CAN access the radio when connected directly to it via its power supply ?
So make sure I understand the process here -
(1) Link goes down
(2) While link is down you go onsite and plug into the router the radio is plugged into.
(3) You can access the router but not access the radio through the router.
(4) You unplug the radio and your laptop from the router and plug radio into laptop.
(5) You can now access the radio
(6) If you plug the radio back into the router (after being able to access it via the laptop) does the link start working ( in other words does just unplugging the radio from the router and plugging it back in fix it ?)
Number (6) there matters because if you can access the radio when plugged into it and then can not access it when you plug it back into the router then whatever is going on , itâs most likely the router.
However, if once you unplug the radio from the router and plug it into your laptop the link works and then it also works when you plug it back into the router⌠then itâs starting to look like a problem with the Ethernet on the radio. I say that because in the previous picture you show the radio lights showing the radio has an ethernet link when the link is down and this most recent post you say the router shows there is not an Ethernet link. So if both are true then when the link is down the radio believes it has a link when the router says it does not.
That could still be either the router or the radio but we have had several occasions where the the ethernet quits working on ePMP radios yet the radios show they have a link even if the ethernet cable is not plugged into anything other than the power supply.
So curious to see if the watchdog resetting the interface fixes it and if it doesnât , possibly set it to reboot the radio instead and see if that does anything ?
Still though, it could be the router, need to know if the link starts working after step 6 if you plug the Ethernet back into the router.
Still though , ethernet failing / working on a schedule does not really fit the problem being the radio. If it was the RF failing then you could blame temps changing at night/morning or reflection off crops vs time of day/night or lots of things but Ethernet really isnât normally subject to these kinds of things.
It appears that there is an interference via the AC mains power , I would suggest to remove the PSU and connect the radios/router with a set of batteries for 24 hours in order to isolate the system from the mains. Because the problem occurs at the same time, some kind of external equipment must be starting working i.e. air conditioning, neon lights , motors.
Just as an update, on Wednesday, I received a call back from the Cambium support team. They advised that its not possible to go directly from the 2.5.1 version of firmware to 4.5.6 directly. Instead the radios need to go to 3.5.6 first. So long story short, he connected remotely to the PC kept on the AP side at the site and downgraded the firmware to 3.5.6. Then he spent time adjusting the power settings to the radios to bring the RSSI as close to or under 60 as possible. Then he advised me to observe the behaviour over the next day and if there was no change, to upgrade it back to 4.5.6 and send them the tech support file as the older firmware does not have this feature. Well basically, there was no change, the radios still go offline for more than 16 hours each day and since the past two days the SM has not come online at all. I havenât had a chance to visit the site yet but I guess it will need restarting. Watchdog on the AP to restart ethernet did not make a difference and so did activating it on the SM and setting it to restarting the wireless. So once Ive restarted the SM once again physically, I will be able to know moreâŚ
Ok so I have lost track of when itâs the SM or the AP and whatâs plugged into what.
So the picture was of the SM when the link was down, but the router saying there was no Ethernet link was connected to the AP ? Is this correct ? If so , disregard most of what I said because I completely misunderstood the problem.
When you say you canât access the SM are you at the end with the SM or are you at the end with the AP ?
Make sure I understand the topology here :
Internet <> Huawei Router <> AP<> SM <> Switch or Router <> Stuff (no internet connection at this end right ?)
When it goes down you say you can not access either radio even via LAN. When you say that do you mean you go to each end ? That is, go to the building with the AP , physically plug into the AP and are unable to access the AP ? Then you go to the building with the SM, physically plug into it and are unable to access the SM ?
This seems to be a cable thing. Something is getting cold and not working then next day it warms back up and works.
Try new cables. (Not guaranteed to be the answer as there may be more going on). Despite what accounting will say, cables do wear out and need replacing.
Is this a critical link that needs to stay up 24/7? If so, are you using a UPS on both ends?
You must step through the firmwares in the right order or you can get odd issues. Try factory defaulting the radios with power cycle (not just rebooting) after defaulting then configure the link again. This will clear the old config out so the new one will be fresh on the new firmware.
