Iotawatt restarting every 38 seconds

[Tom16]

Hi all,

Had my Iota watt for a few years with no probs uploading to emoncms.org (and sometimes a local InfluxDB). I changed this week to a local EmonPi for emoncms and the Iotawatt had been uploading data to the new device ok.

Today it stopped and on investigation the Iotawatt was flashing the LED sequence indicating there was a problem with a log file and scanning was taking place. Once it came back online I noticed it was rebooting frequently. It’s possible this may have been happening before I noticed it doing its log file scan. Here is the latest log:

1/10/23 20:43:48z WiFi connected. SSID=Router57657374, IP=193. xx .1.95, channel=3, RSSI -56db
1/10/23 20:43:48z MDNS responder started for hostname IotaWatt
1/10/23 20:43:49z LLMNR responder started for hostname IotaWatt
1/10/23 20:43:49z emoncms: Start posting at 01/10/23 20:43:10
1/10/23 20:43:50z Updater: Auto-update is current for class MAJOR.
1/10/23 20:44:20z influxDB_v1: stopped, Last post 01/01/70 00:00:00

** Restart **

SD initialized.
1/10/23 20:44:21z Real Time Clock is running. Unix time 1673383461
1/10/23 20:44:21z Reset reason: Exception
1/10/23 20:44:21z Trace: 8:4, 8:6, 8:8, 8:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[6], 31:90, 21:110, 21:110, 31:91, 31:9, 31:1, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[4], 3:91, 15:0[32]
1/10/23 20:44:21z ESP8266 ID: 192587, RTC PCF8523 (68)
1/10/23 20:44:21z IoTaWatt 4.x, Firmware version 02_07_05
1/10/23 20:44:21z SPIFFS mounted.
1/10/23 20:44:21z Local time zone: +0:00, using DST/BST when in effect.
1/10/23 20:44:21z device name: IotaWatt
1/10/23 20:44:21z HTTP server started
1/10/23 20:44:21z emoncms: Starting, interval:10, url:ht tp://193. xx .1.94/emoncms
1/10/23 20:44:21z influxDB_v1: Starting, interval:5, url:ht tp://193. xx .1.100:8086
1/10/23 20:44:21z timeSync: service started.
1/10/23 20:44:22z statService: started.
1/10/23 20:44:22z dataLog: service started.
1/10/23 20:44:22z dataLog: Last log entry 01/10/23 20:44:20
1/10/23 20:44:22z historyLog: service started.
1/10/23 20:44:22z historyLog: Last log entry 01/10/23 20:44:00
1/10/23 20:44:28z WiFi connected. SSID=Router57657374, IP=193. xx .1.95, channel=3, RSSI -57db
1/10/23 20:44:28z MDNS responder started for hostname IotaWatt
1/10/23 20:44:28z LLMNR responder started for hostname IotaWatt
1/10/23 20:44:28z Updater: service started. Auto-update class is MAJOR
1/10/23 20:44:28z emoncms: Start posting at 01/10/23 20:43:50
1/10/23 20:44:58z influxDB_v1: stopped, Last post 01/01/70 00:00:00

** Restart **

SD initialized.
1/10/23 20:44:59z Real Time Clock is running. Unix time 1673383499
1/10/23 20:44:59z Reset reason: Exception
1/10/23 20:44:59z Trace: 8:4, 8:6, 8:8, 8:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[6], 31:90, 21:110, 21:110, 31:91, 31:9, 31:1, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[4], 3:91, 15:0[32]
1/10/23 20:44:59z ESP8266 ID: 192587, RTC PCF8523 (68)
1/10/23 20:44:59z IoTaWatt 4.x, Firmware version 02_07_05
1/10/23 20:44:59z SPIFFS mounted.
1/10/23 20:44:59z Local time zone: +0:00, using DST/BST when in effect.
1/10/23 20:44:59z device name: IotaWatt
1/10/23 20:44:59z HTTP server started
1/10/23 20:44:59z emoncms: Starting, interval:10, url:ht tp://193. xx .1.94/emoncms
1/10/23 20:44:59z influxDB_v1: Starting, interval:5, url:ht tp://193. xx .1.100:8086
1/10/23 20:44:59z timeSync: service started.
1/10/23 20:44:59z statService: started.
1/10/23 20:45:00z dataLog: service started.
1/10/23 20:45:00z dataLog: Last log entry 01/10/23 20:44:55
1/10/23 20:45:00z historyLog: service started.
1/10/23 20:45:00z historyLog: Last log entry 01/10/23 20:44:00
1/10/23 20:45:06z WiFi connected. SSID=Router57657374, IP=193. xx .1.95, channel=3, RSSI -57db
1/10/23 20:45:06z MDNS responder started for hostname IotaWatt
1/10/23 20:45:06z LLMNR responder started for hostname IotaWatt
1/10/23 20:45:06z Updater: service started. Auto-update class is MAJOR
1/10/23 20:45:06z emoncms: Start posting at 01/10/23 20:44:30
1/10/23 20:45:07z Updater: Auto-update is current for class MAJOR.

** Restart **

SD initialized.
1/10/23 20:45:08z Real Time Clock is running. Unix time 1673383508
1/10/23 20:45:08z Reset reason: Exception
1/10/23 20:45:08z Trace: 15:14[15], 15:15, 15:16, 32:95, 31:126, 31:1, 1:6[6], 1:1, 1:2[1], 9:0[1], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[4], 3:91, 15:0[32]
1/10/23 20:45:08z ESP8266 ID: 192587, RTC PCF8523 (68)
1/10/23 20:45:08z IoTaWatt 4.x, Firmware version 02_07_05
1/10/23 20:45:08z SPIFFS mounted.
1/10/23 20:45:08z Local time zone: +0:00, using DST/BST when in effect.
1/10/23 20:45:08z device name: IotaWatt
1/10/23 20:45:08z HTTP server started
1/10/23 20:45:08z emoncms: Starting, interval:10, url:ht tp://193. xx .1.94/emoncms
1/10/23 20:45:08z influxDB_v1: Starting, interval:5, url:ht tp://193. xx .1.100:8086
1/10/23 20:45:08z timeSync: service started.
1/10/23 20:45:08z statService: started.
1/10/23 20:45:09z dataLog: service started.
1/10/23 20:45:09z dataLog: Last log entry 01/10/23 20:45:05
1/10/23 20:45:13z historyLog: service started.
1/10/23 20:45:13z historyLog: Last log entry 01/10/23 20:45:00
1/10/23 20:45:14z WiFi connected. SSID=Router57657374, IP=193. xx .1.95, channel=3, RSSI -57db
1/10/23 20:45:14z MDNS responder started for hostname IotaWatt
1/10/23 20:45:14z LLMNR responder started for hostname IotaWatt
1/10/23 20:45:14z Updater: service started. Auto-update class is MAJOR
1/10/23 20:45:14z emoncms: Start posting at 01/10/23 20:45:10
1/10/23 20:45:44z influxDB_v1: stopped, Last post 01/01/70 00:00:00

** Restart **

SD initialized.
1/10/23 20:45:45z Real Time Clock is running. Unix time 1673383545
1/10/23 20:45:45z Reset reason: Exception
1/10/23 20:45:45z Trace: 31:2[6], 31:90, 21:110, 21:110, 31:91, 31:9, 31:1, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[2], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[8], 31:5, 31:1, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[4], 3:91, 15:0[32]
1/10/23 20:45:45z ESP8266 ID: 192587, RTC PCF8523 (68)
1/10/23 20:45:45z IoTaWatt 4.x, Firmware version 02_07_05
1/10/23 20:45:45z SPIFFS mounted.
1/10/23 20:45:45z Local time zone: +0:00, using DST/BST when in effect.
1/10/23 20:45:45z device name: IotaWatt
1/10/23 20:45:45z HTTP server started
1/10/23 20:45:45z emoncms: Starting, interval:10, url:ht tp://193. xx .1.94/emoncms
1/10/23 20:45:45z influxDB_v1: Starting, interval:5, url:ht tp://193. xx .1.100:8086
1/10/23 20:45:45z timeSync: service started.
1/10/23 20:45:45z statService: started.
1/10/23 20:45:46z dataLog: service started.
1/10/23 20:45:46z dataLog: Last log entry 01/10/23 20:45:40
1/10/23 20:45:50z historyLog: service started.
1/10/23 20:45:50z historyLog: Last log entry 01/10/23 20:45:00
1/10/23 20:45:51z WiFi connected. SSID=Router57657374, IP=193. xx .1.95, channel=3, RSSI -57db
1/10/23 20:45:51z MDNS responder started for hostname IotaWatt
1/10/23 20:45:51z LLMNR responder started for hostname IotaWatt
1/10/23 20:45:51z Updater: service started. Auto-update class is MAJOR
1/10/23 20:45:51z emoncms: Start posting at 01/10/23 20:45:20
1/10/23 20:46:21z influxDB_v1: stopped, Last post 01/01/70 00:00:00

** Restart **

SD initialized.
1/10/23 20:46:22z Real Time Clock is running. Unix time 1673383582
1/10/23 20:46:22z Reset reason: Exception
1/10/23 20:46:22z Trace: 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[6], 31:90, 21:110, 21:110, 31:91, 31:9, 31:1, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[4], 3:91, 15:0[32]
1/10/23 20:46:22z ESP8266 ID: 192587, RTC PCF8523 (68)
1/10/23 20:46:22z IoTaWatt 4.x, Firmware version 02_07_05
1/10/23 20:46:22z SPIFFS mounted.
1/10/23 20:46:22z Local time zone: +0:00, using DST/BST when in effect.
1/10/23 20:46:22z device name: IotaWatt
1/10/23 20:46:22z HTTP server started
1/10/23 20:46:22z emoncms: Starting, interval:10, url:ht tp://193. xx .1.94/emoncms
1/10/23 20:46:22z influxDB_v1: Starting, interval:5, url:ht tp://193. xx .1.100:8086
1/10/23 20:46:22z timeSync: service started.
1/10/23 20:46:22z statService: started.
1/10/23 20:46:23z dataLog: service started.
1/10/23 20:46:23z dataLog: Last log entry 01/10/23 20:46:20
1/10/23 20:46:28z historyLog: service started.
1/10/23 20:46:28z historyLog: Last log entry 01/10/23 20:46:00
1/10/23 20:46:28z Updater: service started. Auto-update class is MAJOR
1/10/23 20:46:28z WiFi connected. SSID=Router57657374, IP=193. xx .1.95, channel=3, RSSI -57db
1/10/23 20:46:28z MDNS responder started for hostname IotaWatt
1/10/23 20:46:29z LLMNR responder started for hostname IotaWatt
1/10/23 20:46:29z emoncms: Start posting at 01/10/23 20:45:50
1/10/23 20:46:30z Updater: Auto-update is current for class MAJOR.
1/10/23 20:47:00z influxDB_v1: stopped, Last post 01/01/70 00:00:00

** Restart **

SD initialized.
1/10/23 20:47:02z Real Time Clock is running. Unix time 1673383622
1/10/23 20:47:02z Reset reason: Exception
1/10/23 20:47:02z Trace: 8:4, 8:6, 8:8, 8:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[6], 31:90, 21:110, 21:110, 31:91, 31:9, 31:1, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[4], 3:91, 15:0[32]
1/10/23 20:47:02z ESP8266 ID: 192587, RTC PCF8523 (68)
1/10/23 20:47:02z IoTaWatt 4.x, Firmware version 02_07_05
1/10/23 20:47:02z SPIFFS mounted.
1/10/23 20:47:02z Local time zone: +0:00, using DST/BST when in effect.
1/10/23 20:47:02z device name: IotaWatt
1/10/23 20:47:02z HTTP server started
1/10/23 20:47:02z emoncms: Starting, interval:10, url:ht tp://193. xx .1.94/emoncms
1/10/23 20:47:02z influxDB_v1: Starting, interval:5, url:ht tp://193. xx .1.100:8086
1/10/23 20:47:02z timeSync: service started.
1/10/23 20:47:02z statService: started.
1/10/23 20:47:03z dataLog: service started.
1/10/23 20:47:03z dataLog: Last log entry 01/10/23 20:47:00
1/10/23 20:47:07z historyLog: service started.
1/10/23 20:47:07z historyLog: Last log entry 01/10/23 20:47:00
1/10/23 20:47:08z WiFi connected. SSID=Router57657374, IP=193. xx .1.95, channel=3, RSSI -58db
1/10/23 20:47:08z MDNS responder started for hostname IotaWatt
1/10/23 20:47:08z LLMNR responder started for hostname IotaWatt
1/10/23 20:47:08z Updater: service started. Auto-update class is MAJOR
1/10/23 20:47:08z emoncms: Start posting at 01/10/23 20:46:30

(URLs masked as forum wouldn’t let me post them)

Just realised if I stop the emoncms upload the Iotawatt doesn’t keep restarting, but whilst it is running it successfully posts to my EmonPi for the 30-odd seconds.

Any ideas folks?

[overeasy]

Can you post the Emoncms setup please?

And your status display with all tabs expanded please?

[Tom16]

Hi.

Emoncms uploader config:

Status page:

Thanks.

[overeasy]

I don’t see anything with the setup that should cause the restarts. What I do see, and what you described, is that there was a problem detected with the current log and it was deleted. Given that your unit has been in service for 4 1/2 years, it’s quite possible that you SD card is failing. I would advise replacing it.

Remove/replace SDcard

Sometimes it becomes necessary to remove the microSD card. You may need to replace it or look at the message log if the device isn’t working, or maybe you want to back up the datalogs. For whatever reason, it’s best to be careful as the type of holder and position makes it easy to damage during removal. Power down the IoTaWatt by removing the USB power plug and AC reference plug. If mounted to a wall, remove the mounting screws. Remove the four Philips head screws in the back. Remo…

Building a new SDcard

It can become necessary to build a new SDcard. This post describes how to do that. IoTaWatt uses a microSD card, formatted to FAT32. New units use a 8GB level 10 card. Any card 4GB-32GB level 4 or 10 will work fine. Obtain a new microSD card or reformat an old card to FAT32. Download the SD files from github SD directory If using windows, this link will downlod the files into a .zip file. Copy the files to the root of the SDcard. If you were able to salvage the config.txt file from the ol…

When you copy the contents of the old card (if you can) omit the current log /iotawatt/iotalog.log, and retain the history log /iotawatt/histlog.log. After restart with the new card, start the Emoncms uploader. If the restarts continue, delete both logs using the browser URL

http://iotawatt.local/command?deletelog=both

[Tom16]

Ok, thanks. Should have some time on Friday to fiddle with this.

You mention about copying from the current SD card if I can - should I just copy everything from this on to a fresh SD card (except the current log) or do I still need to download the files from github first?

Thanks again for your help.

[overeasy]

If you can copy the root files from the old card, that will be fine. It is also not a problem to preload the Github files then copy your old card over it replacing duplicates.

[Tom16]

Just had some time to do this.

I removed the SD card from the iotawatt and was able to copy everything off it on to my PC with no issues. I then copied everything (except the current log file) on to a brand new 8GB SD card and put the iotawatt back together.

It all started up ok and my config was all there, but unfortunately 36 seconds after the emoncms upload was started, the iotowatt restarted again. I left if for a couple of minutes to check this behaviour continued, it did. I then stopped the emoncms process, ran the delete both logs command and the unit restarted and showed that the logs were now empty. Unfortunately the same restart behaviour continued once the emoncms upload started.

I’ve checked the logs on my emoncms instance and there is nothing untoward. Again, data was posted to emoncms for each 30 odd seconds period the iotawatt was up.

The iotawatt was stable since I last posted on here with the emoncms upload not running, and I could view recent data in the iotawatt graphing tool. I was also able to download the config and current log via the ‘File manager and editor’ before I changed the SD card.

Does this mean there’s an issue with the firmware? Are there any other settings I should check?

Thanks in advance.

[overeasy]

This is a sticky problem that has me puzzled.

You state that the new local instance of Emoncms worked OK, then apparently just stopped. There was a datalog corruption maybe at the same time and the current log was deleted and recreated. It’s hard to tell the significance of those things without a better timeline.

Can you download (or copy from the SD) your entire message log /iotawatt/iotamsgs.txt and post it to me as an uploaded file in a PM please?

Two other things as well: Your Emoncms setup does not show a userid. Is that how it is configured or have you removed it from the post for security reasons. It determines whether the communications are encrypted or not and is right in the middle of those exceptions.

The failure appears to happen during processing of the response to the Emoncms post. If you can run the IoTaWatt from a computer USB cable with a serial monitor active (115,200 baud) the response from Emoncms should be logged. Another method would be to use wireshark or other capture tool.

Thanks.

[Tom16]

@overeasy

You state that the new local instance of Emoncms worked OK, then apparently just stopped.

On looking at the logs, it turns out as soon as I changed the config to upload to my local emonPi, the restarts started. I didn’t notice this before as I was uploading historic data so the inputs in emoncms weren’t going to say they were current anyway and my attention was on setting up the feeds etc. in emoncms.

@overeasy

There was a datalog corruption maybe at the same time and the current log was deleted and recreated. It’s hard to tell the significance of those things without a better timeline.

I noticed the data log being scanned over 24 hours after I changed the config. Perhaps the constant restarts led to this.

@overeasy

Can you download (or copy from the SD) your entire message log /iotawatt/iotamsgs.txt and post it to me as an uploaded file in a PM please?

PM sent.

@overeasy

Two other things as well: Your Emoncms setup does not show a userid. Is that how it is configured or have you removed it from the post for security reasons. It determines whether the communications are encrypted or not and is right in the middle of those exceptions.

I seemed to recall that one shouldn’t enter a userid for a local instance of emoncms - but I can’t find where I thought I read that now. I tried last night to enter the userid (1) but I got an invalid response message (but no restart!)

@overeasy

The failure appears to happen during processing of the response to the Emoncms post.

Indeed - some data is uploaded to emoncms then nothing happens for 30 odd seconds before the restart of the IoTaWatt and then the process repeats.

@overeasy

If you can run the IoTaWatt from a computer USB cable with a serial monitor active (115,200 baud) the response from Emoncms should be logged. Another method would be to use wireshark or other capture tool.

I’ve never used either wireshark or a serial monitor. I’ve installed wireshark but I can only appear to get to show traffic between my PC and the IoTaWatt or the emonPI and my PC, not between IoTaWatt and the emonPi. I will keep fiddling to see if I missed something.

Thanks for the help thus far.

[Tom16]

@Tom16

I’ve never used either wireshark or a serial monitor. I’ve installed wireshark but I can only appear to get to show traffic between my PC and the IoTaWatt or the emonPI and my PC, not between IoTaWatt and the emonPi. I will keep fiddling to see if I missed something.

Update:

Have adjusted a setting on my switch which allowed Wireshark to work. I’m not exactly sure what I’m looking for but the only obvious error message which was captured from the emonPi to the IoTaWatt was this PHP error:

<b>Deprecated</b>:  strlen(): Passing null to parameter #1 ($string) of type string is deprecated in <b>/var/www/emoncms/Modules/input/input_methods.php</b> on line <b>269</b><br />

This appears several times before a restart occurs. Could this be the cause?

Line 269 in input_methods.php is:

if (strlen($item[$i]))

Looking at the input sent from IoTaWatt shows a null value:

[460,"node1",null,206.19,1.56,0.34,0.63,208.08,-0.21]

Not sure what the null is as I only have 6 inputs in the emoncms upload config and there are 7 data items there, or if this is the null referred to in the error above.

Thanks.

[overeasy]

@Tom16

Not sure what the null is as I only have 6 inputs in the emoncms upload config and there are 7 data items there, or if this is the null referred to in the error above.

The null is a placeholder because the bulk upload protocol identifies the inputs by their position in the list. Your outputs begin with input 2 so a null is required in the input 1 position.

I think this may be one of the problems. The other is why the IoTaWatt chokes on the error message returned from Emoncms.

I just tested using a null item 1 to Emoncms.org and it works fine. There is something about your Emoncms that is rejecting it. What is the version of Emoncms that you have, and what is the version of PHP on your server? Are you using Apache?

[Tom16]

@overeasy

@Tom16

Not sure what the null is as I only have 6 inputs in the emoncms upload config and there are 7 data items there, or if this is the null referred to in the error above.

The null is a placeholder because the bulk upload protocol identifies the inputs by their position in the list. Your outputs begin with input 2 so a null is required in the input 1 position.

Ah of course. That’s my laziness coming back to bite me there. I just used the default upload template but removed the voltage input from position 1 and left grid where it was. If moving everything up one position sorts it, I will do that - or did you want to try to get IoToWatt to be able to handle such an error first?

@overeasy

I just tested using a null item 1 to Emoncms.org and it works fine. There is something about your Emoncms that is rejecting it.

Indeed, I had the inputs in the same positions when I was uploading to emoncms.org with no issues.

@overeasy

What is the version of Emoncms that you have, and what is the version of PHP on your server? Are you using Apache?

It’s 11.2.8 (seemingly newer than emoncms.org), PHP 8.1.12 and yes, Apache. It was all preconfigured by the Open Energy Monitor guys as the Pi unit was bought from their shop (emonBase 433Mhz (RFM69SPI) - Shop | OpenEnergyMonitor).

[overeasy]

So, can you give it a try with output 1 configured for voltage. If you don’t configure a process list, it doesn’t really do anything other than show in your inputs list.

If that eliminates the problem, I’ll contact Trystan and see if its intentional or an oversight.

[Tom16]

Configured input 1 as voltage and the upload worked without any restarts.

Problem solved (from my point of view anyway!)

Thanks for your assistance.