Repeated crashes / outages

I have two Iotawatts with 27 CTs in total, each device connected with the 5V usb psu and the 9v sense. For the first ~2 years, they worked great, but have recently started crashing. By crashing I mean, they stop responding to network activity. They are still recording data during these states (can see it in the graph). They do not respond to pings or tcp/80 until they are power cycled, at which point they continue to work for a few minutes / hours (at random) until it again stops responding. I’ve verified in my wifi AP that the devices are still maintaining wifi association and have sufficient signal strength.

I thought maybe it was bad SD cards, so I replaced both of them, but the problem continues. What should I try next?

Running Firmware version: 02_08_03

Thanks!

Logs from the last few reboots:

** Restart **

SD initialized.
5/21/24 21:38:47z Real Time Clock is running. Unix time 1716327527 
5/21/24 21:38:47z Reset Reason: Power-fail restart.
5/21/24 21:38:47z ESP8266 ID: 1122977, RTC PCF8523 (68)
5/21/24 21:38:47z IoTaWatt 5.0, Firmware version 02_08_03
5/21/24 21:38:47z SPIFFS mounted.
5/21/24 17:38:47 Local time zone: -5:00, using DST/BST when in effect.
5/21/24 17:38:47 device name: IotaWatt
5/21/24 17:38:50 Connecting with WiFiManager.
5/21/24 17:38:54 HTTP server started
5/21/24 17:38:55 WiFi connected. SSID=xxxx, IP=192.168.0.41, channel=11, RSSI -76db
5/21/24 17:38:55 timeSync: service started.
5/21/24 17:38:55 statService: started.
5/21/24 17:38:55 Updater: service started. Auto-update class is NONE
5/21/24 17:38:55 dataLog: service started.
5/21/24 17:38:57 dataLog: Last log entry 05/21/24 17:38:35
5/21/24 17:39:00 historyLog: service started.
5/21/24 17:39:00 historyLog: Last log entry 05/21/24 17:38:00
5/21/24 17:39:00 grid: Started
5/21/24 17:39:01 grid: Last log entry 05/21/24 17:38:35

** Restart **

SD initialized.
5/22/24 02:14:18z Real Time Clock is running. Unix time 1716344058 
5/22/24 02:14:18z Reset Reason: Power-fail restart.
5/22/24 02:14:18z ESP8266 ID: 1122977, RTC PCF8523 (68)
5/22/24 02:14:18z IoTaWatt 5.0, Firmware version 02_08_03
5/22/24 02:14:18z SPIFFS mounted.
5/21/24 22:14:18 Local time zone: -5:00, using DST/BST when in effect.
5/21/24 22:14:18 device name: IotaWatt
5/21/24 22:14:21 Connecting with WiFiManager.
5/21/24 22:14:26 HTTP server started
5/21/24 22:14:26 WiFi connected. SSID=xxxx, IP=192.168.0.41, channel=11, RSSI -79db
5/21/24 22:14:26 timeSync: service started.
5/21/24 22:14:26 statService: started.
5/21/24 22:14:26 Updater: service started. Auto-update class is NONE
5/21/24 22:14:26 dataLog: service started.
5/21/24 22:14:28 dataLog: Last log entry 05/21/24 22:13:35
5/21/24 22:14:31 historyLog: service started.
5/21/24 22:14:31 historyLog: Last log entry 05/21/24 22:13:00
5/21/24 22:14:31 grid: Started
5/21/24 22:14:32 grid: Last log entry 05/21/24 22:13:30

** Restart **

SD initialized.
5/22/24 02:15:21z Real Time Clock is running. Unix time 1716344121 
5/22/24 02:15:21z Reset Reason: Power-fail restart.
5/22/24 02:15:21z ESP8266 ID: 1122977, RTC PCF8523 (68)
5/22/24 02:15:21z IoTaWatt 5.0, Firmware version 02_08_03
5/22/24 02:15:21z SPIFFS mounted.
5/21/24 22:15:21 Local time zone: -5:00, using DST/BST when in effect.
5/21/24 22:15:21 device name: IotaWatt
5/21/24 22:15:24 Connecting with WiFiManager.
5/21/24 22:15:28 HTTP server started
5/21/24 22:15:29 WiFi connected. SSID=xxxx, IP=192.168.0.41, channel=11, RSSI -79db
5/21/24 22:15:29 timeSync: service started.
5/21/24 22:15:29 statService: started.
5/21/24 22:15:29 Updater: service started. Auto-update class is NONE
5/21/24 22:15:29 dataLog: service started.
5/21/24 22:15:31 dataLog: Last log entry 05/21/24 22:15:10
5/21/24 22:15:34 historyLog: service started.
5/21/24 22:15:34 historyLog: Last log entry 05/21/24 22:15:00
5/21/24 22:15:34 grid: Started
5/21/24 22:15:35 grid: Last log entry 05/21/24 22:15:10

** Restart **

SD initialized.
5/28/24 16:15:55z Real Time Clock is running. Unix time 1716912955 
5/28/24 16:15:55z Reset Reason: Power-fail restart.
5/28/24 16:15:55z ESP8266 ID: 1122977, RTC PCF8523 (68)
5/28/24 16:15:55z IoTaWatt 5.0, Firmware version 02_08_03
5/28/24 16:15:55z SPIFFS mounted.
5/28/24 12:15:55 Local time zone: -5:00, using DST/BST when in effect.
5/28/24 12:15:55 device name: IotaWatt
5/28/24 12:15:58 HTTP server started
5/28/24 12:15:58 WiFi connected. SSID=xxxx, IP=192.168.0.41, channel=11, RSSI -86db
5/28/24 12:15:58 timeSync: service started.
5/28/24 12:15:58 statService: started.
5/28/24 12:15:58 Updater: service started. Auto-update class is NONE
5/28/24 12:15:58 dataLog: service started.
5/28/24 12:16:00 dataLog: Last log entry 05/28/24 12:15:45
5/28/24 12:16:03 historyLog: service started.
5/28/24 12:16:03 historyLog: Last log entry 05/28/24 12:15:00
5/28/24 12:16:03 grid: Started
5/28/24 12:16:04 grid: Last log entry 05/28/24 12:15:40

The log shows very poor WiFi RSSI which could cause your symptoms. I don’t see any crashes in the log, just a couple of daily restarts, and the latest power-cycle restart after a week of running OK.

In today’s environment, WiFi can be problematic for no apparent reason. Consider that there are loads of non WiFi “wireless” devices that use the band, as well as other Wifi networks from neighboring sites. So although your IoTaWatt did not change and you didn’t change anything, the wireless environment may have changed.

I would recommend the following:

Try to relocate your IoTaWatt or AP to improve the RSSI to below -70.

Try different channels on the WiFi. The best are 1,6 and 11. You are on 11, so maybe 1 or 6 would work better.

Thanks for pointing that out! I was able to move the two iotawatt units a bit and got them both to -69 to -72 RSSI. Can’t do any better without a lot of work to install another AP. Hopefully this will be sufficient but if not I’ll put another AP in.

Fortunately there’s zero wifi interference here, I live about half a mile from my neighbors, and all wifi scanner apps I’ve tried show only my own access points.

1 Like

OK, it happened again. This time rssi was at -65 and while the system was replying to pings, and would accept connections to tcp/80, it did not respond to http requests and would quickly close the connection. I’ve been running smokeping against it and the network connection has zero packet loss and decent pings (15ms average). Just like last time, a power cycle brings it back up immediately.

Any ideas? Thanks!

Nope. If you want to send me the unit I’ll see if it hold up on my Ubiquity WiFi.

I’m using Unifi WiFi here, specifically a U6-LR running 6.6.55.15189. I think it’s really strange the issue is happening almost identically (although at different times) across 2 Iotawatt devices.

Could it be that some database on the SD card is corrupt? I have data in there going back 2 years almost, though this is only 2gb and I have the devices using 32gb cards now.

Anything is possible, but on both units?

Somewhere in this forum a user had similar problems using Ubiquiti and as I recall it was resolved with an AP firmware update. I have several different AP models models but have had no issues.

That firmware version caused lots of issues with my two IoTaWatt devices, each connected to different UniFi APs.

I switched the UniFi APs to use 6.6.67 and the problems have gone away. This is an Early Access version of firmware. It has been running great for me for a few weeks now.

1 Like