IotaWatt becomes unreachable after days/weeks

Noerr · April 27, 2023, 2:55pm

My IotaWatt was quite reliable - out of site, out-of-mind - for about 1.5 years. Starting around March the device has stopped uploading (HomeAssistant and influxdB2), eventually I notice and then find I cannot reach it by HTTP interface either.

I reset the device by cycling power off/on and find that everything restores / reconnects immediately and that the IotaWatt has valid power/energy data covering the duration of the connectivity outage. So it was alive and recording, but not doing desired network things.

I have not changed the location of my WiFi access points (Aruba AP22’s WiFi 6) or the IotaWatt. I have not moved walls or furniture.

Over the course of four months, I see just one WiFi disconnected. log message with reconnect ~2 minutes later.

Why is my IotaWatt now unstable?

I do notice a firmware update logged just before this problem began. All restarts following that firmware update are unannounced - i.e. nothing is logged for a long duration until the restart sequence is logged - the restart sequence triggered by my power off-on cycle.

BTW, Since I have multiple WiFi access points serving the same network, it would be helpful if the WiFi connected log entry included the access point’s MAC address.

My latest log, year-to-date follows:

1/05/23 06:17:21z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 07:17:27z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 08:17:33z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 09:17:39z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 10:17:46z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 11:17:52z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 12:17:59z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 13:18:05z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 14:18:11z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 15:18:17z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 16:18:23z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 17:18:29z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 18:18:35z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 19:18:41z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 20:18:47z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 21:18:53z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 22:18:59z Updater: update tables from 0.0.6 to 0.0.7
1/05/23 23:19:05z Updater: update tables from 0.0.6 to 0.0.7
1/06/23 00:19:11z Updater: update tables from 0.0.6 to 0.0.7
1/06/23 01:19:17z Updater: update tables from 0.0.6 to 0.0.7
1/18/23 02:11:42z timeSync: Six week routine restart.

** Restart **

SD initialized.
1/18/23 02:11:44z Real Time Clock is running. Unix time 1674007904 
1/18/23 02:11:44z Reset reason: Software/System restart
1/18/23 02:11:44z Trace:  1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:1[2], 1:2[3], 9:0[3], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[2], 1:6[2], 1:6[3], 1:5[20], 1:6[4], 20:0, 20:1
1/18/23 02:11:44z ESP8266 ID: 8528520, RTC PCF8523 (68)
1/18/23 02:11:44z IoTaWatt 5.0, Firmware version 02_07_05
1/18/23 02:11:44z SPIFFS mounted.
1/18/23 02:11:44z Local time zone: +0:00
1/18/23 02:11:44z device name: IotaWatt
1/18/23 02:11:44z HTTP server started
1/18/23 02:11:44z influxDB_v2: Starting, interval:5, url:http://10.0.1.143:8086
1/18/23 02:11:44z timeSync: service started.
1/18/23 02:11:44z statService: started.
1/18/23 02:11:44z dataLog: service started.
1/18/23 02:11:47z dataLog: Last log entry 01/18/23 02:11:40
1/18/23 02:11:49z historyLog: service started.
1/18/23 02:11:50z historyLog: Last log entry 01/18/23 02:11:00
1/18/23 02:11:54z WiFi connected. SSID=New6, IP=10.0.1.177, channel=1, RSSI -67db
1/18/23 02:11:54z MDNS responder started for hostname IotaWatt
1/18/23 02:11:54z LLMNR responder started for hostname IotaWatt
1/18/23 02:11:54z Updater: service started. Auto-update class is MINOR
1/18/23 02:11:54z influxDB_v2: Resume posting 01/18/23 02:11:30
1/18/23 02:11:56z Updater: Auto-update is current for class MINOR.
1/25/23 06:29:25z Updater: Invalid response from server. HTTPcode: -4

** Restart **

SD initialized.
2/25/23 23:50:52z Real Time Clock is running. Unix time 1677369052 
2/25/23 23:50:52z Reset Reason: Power-fail restart.
2/25/23 23:50:52z ESP8266 ID: 8528520, RTC PCF8523 (68)
2/25/23 23:50:52z IoTaWatt 5.0, Firmware version 02_07_05
2/25/23 23:50:52z SPIFFS mounted.
2/25/23 23:50:53z Local time zone: +0:00
2/25/23 23:50:53z device name: IotaWatt
2/25/23 23:50:56z Connecting with WiFiManager.
2/25/23 23:51:02z HTTP server started
2/25/23 23:51:02z influxDB_v2: Starting, interval:5, url:http://10.0.1.143:8086
2/25/23 23:51:02z WiFi connected. SSID=New6, IP=10.0.1.177, channel=1, RSSI -67db
2/25/23 23:51:02z MDNS responder started for hostname IotaWatt
2/25/23 23:51:02z LLMNR responder started for hostname IotaWatt
2/25/23 23:51:02z timeSync: service started.
2/25/23 23:51:02z statService: started.
2/25/23 23:51:02z Updater: service started. Auto-update class is MINOR
2/25/23 23:51:02z dataLog: service started.
2/25/23 23:51:05z dataLog: Last log entry 02/25/23 23:41:35
2/25/23 23:51:07z historyLog: service started.
2/25/23 23:51:08z historyLog: Last log entry 02/25/23 23:41:00
2/25/23 23:51:08z influxDB_v2: Resume posting 02/25/23 23:41:20
2/25/23 23:51:08z Updater: Auto-update is current for class MINOR.
3/03/23 21:05:19z Updater: Update from 02_07_05 to 02_08_02
3/03/23 21:05:19z Updater: download 02_08_02
3/03/23 21:05:26z Updater: Release downloaded 7443ms, size 964192
3/03/23 21:05:38z Updater: signature verified
3/03/23 21:05:47z Updater: firmware upgraded to version 02_08_02
3/03/23 21:05:47z Updater: Firmware updated, restarting.

** Restart **

SD initialized.
3/03/23 21:05:58z Real Time Clock is running. Unix time 1677877558 
3/03/23 21:05:58z Reset reason: Software/System restart
3/03/23 21:05:58z Trace:  1:5[31], 1:6[4], 31:0, 31:1, 31:2[3], 29:60, 29:60, 29:61, 31:1, 1:6[6], 1:1[13], 1:2[14], 9:0[14], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[3], 1:5[5], 1:6[4], 5:0, 5:7
3/03/23 21:05:58z ESP8266 ID: 8528520, RTC PCF8523 (68)
3/03/23 21:05:58z IoTaWatt 5.0, Firmware version 02_08_02
3/03/23 21:05:58z Updater: Installing update files for version 02_08_02
3/03/23 21:05:58z Updater: Installing EDIT.HTM
3/03/23 21:05:58z Updater: Installing FAVICON.ICO
3/03/23 21:05:59z Updater: Installing GRAPH.HTM
3/03/23 21:05:59z Updater: Installing GRAPH.JS
3/03/23 21:05:59z Updater: Installing GRAPH2.HTM
3/03/23 21:05:59z Updater: Installing GRAPH2.JS
3/03/23 21:06:00z Updater: Installing INDEX.HTM
3/03/23 21:06:01z Updater: Installing TABLES.TXT
3/03/23 21:06:02z Updater: Installing CNFSTYLE.CSS
3/03/23 21:06:02z Updater: Installation complete.
3/03/23 21:06:02z SPIFFS mounted.
3/03/23 21:06:02z Local time zone: +0:00
3/03/23 21:06:02z device name: IotaWatt
3/03/23 21:06:02z HTTP server started
3/03/23 21:06:02z influxDB_v2: Starting, interval:5, url:http://10.0.1.143:8086
3/03/23 21:06:02z timeSync: service started.
3/03/23 21:06:02z statService: started.
3/03/23 21:06:03z dataLog: service started.
3/03/23 21:06:06z dataLog: Last log entry 03/03/23 21:05:25
3/03/23 21:06:08z historyLog: service started.
3/03/23 21:06:08z historyLog: Last log entry 03/03/23 21:05:00
3/03/23 21:06:12z Updater: service started. Auto-update class is MINOR
3/03/23 21:06:12z WiFi connected. SSID=New6, IP=10.0.1.177, channel=1, RSSI -66db
3/03/23 21:06:12z influxDB_v2: Resume posting 03/03/23 21:04:50
3/03/23 21:06:13z Updater: Auto-update is current for class MINOR.

** Restart **

SD initialized.
3/18/23 05:46:46z Real Time Clock is running. Unix time 1679118406 
3/18/23 05:46:46z Reset Reason: Power-fail restart.
3/18/23 05:46:46z ESP8266 ID: 8528520, RTC PCF8523 (68)
3/18/23 05:46:46z IoTaWatt 5.0, Firmware version 02_08_02
3/18/23 05:46:46z SPIFFS mounted.
3/18/23 05:46:47z Local time zone: +0:00
3/18/23 05:46:47z device name: IotaWatt
3/18/23 05:46:50z Connecting with WiFiManager.
3/18/23 05:46:55z HTTP server started
3/18/23 05:46:56z influxDB_v2: Starting, interval:5, url:http://10.0.1.143:8086
3/18/23 05:46:56z WiFi connected. SSID=New6, IP=10.0.1.177, channel=1, RSSI -68db
3/18/23 05:46:56z timeSync: service started.
3/18/23 05:46:56z statService: started.
3/18/23 05:46:56z Updater: service started. Auto-update class is MINOR
3/18/23 05:46:56z dataLog: service started.
3/18/23 05:46:59z dataLog: Last log entry 03/18/23 05:46:35
3/18/23 05:46:59z influxDB_v2: Resume posting 03/18/23 05:46:20
3/18/23 05:47:00z Updater: Auto-update is current for class MINOR.
3/18/23 05:47:01z historyLog: service started.
3/18/23 05:47:01z historyLog: Last log entry 03/18/23 05:46:00
3/26/23 10:02:39z WiFi disconnected.
3/26/23 10:05:00z WiFi connected. SSID=New6, IP=10.0.1.177, channel=1, RSSI -64db

** Restart **

SD initialized.
3/31/23 19:08:14z Real Time Clock is running. Unix time 1680289694 
3/31/23 19:08:14z Reset Reason: Power-fail restart.
3/31/23 19:08:14z ESP8266 ID: 8528520, RTC PCF8523 (68)
3/31/23 19:08:14z IoTaWatt 5.0, Firmware version 02_08_02
3/31/23 19:08:14z SPIFFS mounted.
3/31/23 19:08:15z Local time zone: +0:00
3/31/23 19:08:15z device name: IotaWatt
3/31/23 19:08:18z Connecting with WiFiManager.
3/31/23 19:08:21z HTTP server started
3/31/23 19:08:21z influxDB_v2: Starting, interval:5, url:http://10.0.1.143:8086
3/31/23 19:08:21z WiFi connected. SSID=New6, IP=10.0.1.177, channel=1, RSSI -66db
3/31/23 19:08:22z timeSync: service started.
3/31/23 19:08:22z statService: started.
3/31/23 19:08:22z Updater: service started. Auto-update class is MINOR
3/31/23 19:08:22z dataLog: service started.
3/31/23 19:08:25z dataLog: Last log entry 03/31/23 19:08:05
3/31/23 19:08:25z influxDB_v2: Resume posting 03/31/23 19:07:50
3/31/23 19:08:26z Updater: Auto-update is current for class MINOR.
3/31/23 19:08:27z historyLog: service started.
3/31/23 19:08:27z historyLog: Last log entry 03/31/23 19:08:00

** Restart **

SD initialized.
4/27/23 14:02:23z Real Time Clock is running. Unix time 1682604143 
4/27/23 14:02:23z Reset Reason: Power-fail restart.
4/27/23 14:02:23z ESP8266 ID: 8528520, RTC PCF8523 (68)
4/27/23 14:02:23z IoTaWatt 5.0, Firmware version 02_08_02
4/27/23 14:02:23z SPIFFS mounted.
4/27/23 14:02:24z Local time zone: +0:00
4/27/23 14:02:24z device name: IotaWatt
4/27/23 14:02:27z Connecting with WiFiManager.
4/27/23 14:02:30z HTTP server started
4/27/23 14:02:30z influxDB_v2: Starting, interval:5, url:http://10.0.1.143:8086
4/27/23 14:02:30z WiFi connected. SSID=New6, IP=10.0.1.177, channel=1, RSSI -71db
4/27/23 14:02:30z timeSync: service started.
4/27/23 14:02:30z statService: started.
4/27/23 14:02:30z Updater: service started. Auto-update class is MINOR
4/27/23 14:02:30z dataLog: service started.
4/27/23 14:02:33z dataLog: Last log entry 04/27/23 14:02:15
4/27/23 14:02:34z influxDB_v2: Resume posting 04/27/23 14:02:00
4/27/23 14:02:35z historyLog: service started.
4/27/23 14:02:35z historyLog: Last log entry 04/27/23 14:02:00
4/27/23 14:02:35z Updater: Auto-update is current for class MINOR.

overeasy · April 27, 2023, 9:39pm

I’m not saying there isn’t a problem, but the message log doesn’t match your description.

I see that. The update was on March 3. There were 3 power fail resets after that on March 18, March 31 and today April 27.

The unit appears to have been running OK. at each restart. The datalog was current and influxDB was up to date.

The problem with home assistant may be related to recent Home Assistant updates. I don’t know the current status, but they did introduce breaking changes to the integration recently. Inability to access the IoTaWatt from your browser may be a problem with network configuration. I know you say nothing has changed, but clearly something has. Browsers and operating systems are updated automatically all the time.

Have you tried accessing the IoTaWatt using its IP address 10.0.1.177 rather than iotawatt.local? If that works, you might check your integration in HASS and input the IP address rather than iotawatt.local as the unit identifier.

Noerr · April 27, 2023, 10:17pm

but the message log doesn’t match your description.
In what way? I’m not following you.

Maybe I could have been more explicit but each of the power fail resets was me turning on and off the power to the IotaWatt. Right before power-off, I was observing no HomeAssistant data feed AND no access through http://iotawatt.local. Then shortly after power-on I see new data in HomeAssistant and http://iotawatt.local serves up the expected web page.

I don’t know why HomeAssistant would fail at the same time as the IotaWatt web interface.

I can make a mental note to check a few things next time this occurs:

is the HTTP interface accessible through the IP address instead of mDNS.
does my WiFi access point think the IotaWatt is still connected
Is the IotaWatt still live-uploading to influxDB

(regarding 3, I realize I didn’t check for this before, so I may have been wrong when I said HomeAssistant and influxDB2)

oh, so we can look at the timestamps in this statement indicating influxDB_v2 was still up-to-date with posts right up until the power-off.

4/27/23 14:02:34z influxDB_v2: Resume posting 04/27/23 14:02:00

overeasy · April 27, 2023, 11:17pm

It does. Influx is using it to upload. Also, the datalog is uneventful until the restart. You would see problems with checking the time and checking for updates if it wasn’t connected.

Understand that the IoTaWatt doesn’t upload to HomeAssistant. That would be a push protocol and homeassistant would not allow IoTaWatt to do that like it does with influx, PVoutput and Emoncms. The home assistant integration is a pull protocol where HASS periodically issues standard query requests to the IoTaWatt. That’s why I believe the HASS issue and your browser access issue are the same problem.

I believe it is may be related to the way your network is handling the zeroconfig protocol, either mDNS(Apple) or LLMNR(Microsoft). If so, using the IP instead of IoTaWatt.local should solve both your browser and HASS issue.

As a push protocol, the Influx uploader is not using a zeroconfig protocol.

Noerr · April 28, 2023, 2:46am

Good info, thanks! I’ll poke into this direction next time the HomeAssistant integration drops off.

Noerr · May 1, 2023, 8:41pm

I notice the iotawatt integration in HomeAssistant stopped updating data on Saturday and web page access through iotawatt.local is down too.

As you predicted, I can access the web interface through its up address http://10.0.1.177/

I’ll have to research more about mDNS to understand what’s going on and what is expected. I have looked at mDNS traffic through WireShark for another project a year ago.

overeasy · May 1, 2023, 9:21pm

It’s probably not that complicated. This usually happens when the WiFi network is inhibiting multicast. That’s common in public networks so that you can’t see other users or access their resources, but many home routers have that as an option, particularly on guest networks. Try to access using a windows machine. If you have the same problem, it’s probably inhibited multicast because windows uses a different protocol LLMNR that’s the same but different.

BTW:/ you can specify the IP address in Home Assistant instead of iotawatt.local.

Noerr · May 2, 2023, 3:07am

I do not see any other indications of mDNS issues or other multicast issues on my home LAN. It seems to me the most likely failure point is that the mDNS responder running on the IotaWatt device is failing after the timeframe of days/weeks.

Using WireShark, I’m observing mDNS “Standard query” packets querying for iotawatt.local. They go unanswered. I cycled power on the Iotawatt device. Within a few seconds, “Standard query response” packets are seen with the 10.0.1.177 IP address as payload. HTTP and HomeAssistant integration resume normal operations.

Importantly, I did not reset routers or wifi, or anything else with the network.

Considering the workaround to change my Home Assistant configuration to IP address, I’m definitely open to it. After some digging and digging, it seems to do so requires removing the integration and re-adding, which would unravel some down-things in place. I’ll hold on that for a bit.

overeasy · May 2, 2023, 3:30am

Have you tried to access with a windows hosted browser?

Noerr · May 21, 2023, 9:57pm

I don’t have a Windows machine around. I thought one might visit my house if I waited a bit… but not yet.

I did test with linux this cycle though:

$ avahi-resolve --name iotawatt.local
Failed to resolve host name 'iotawatt.local': Timeout reached

# I used the web interface "Restart" button here ( http://10.0.1.177 ), then:

$ avahi-resolve --name iotawatt.local
IotaWatt.local	10.0.1.177

I feel better doing the web restart button than flipping the breaker,… now that I know about the IP address access.