Iotawatt bootlooping

[mbbush]

I’ve been using my iotawatt for several years, and it generally works great. But this morning it stopped responding to any IP traffic, then started constantly rebooting itself. It looked like the LED was blinking red-red-red, but I’m not completely certain of that.

I power cycled it, and the LED started blinking green-green-green. That’s not one of the patterns on the troubleshooting site.

So I took it apart, and determined that:

• The SD card is readable.
• When I clone the entire contents of the SD card to a new SD card, it’s still flashing “green green green”
• Based on what I can now see when powering it up with the case off, it looks like the LED is giving one green flash, then it’s crashing at some point early in the boot process, rebooting, and giving another green flash on the next boot (since the blue LEDs on the microcontroller are similarly cycling.

Here’s what I found in the logs:

At exactly the time the problem started (as measured by the last data received by home assistant):

10/05/24 08:15:31 timeSync: Six week routine restart.^M
^M
** Restart **^M

SD initialized.^M
10/05/24 15:15:32z Real Time Clock is running. Unix time 1728141332 ^M
10/05/24 15:15:32z Reset reason: Software/System restart^M
10/05/24 15:15:32z Trace:  1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:1[11], 1:2[12], 9:0[12], 9:0, 9:1, 8:4, 8:6, 8:8, 9:3, 9:5, 9:9, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[2], 1:6[2], 1:6[3], 1:5[20], 1:6[4], 20:0, 20:1^M
10/05/24 15:15:32z ESP8266 ID: 522967, RTC M41T81 (68)^M
10/05/24 15:15:32z IoTaWatt 5.0, Firmware version 02_08_03^M
10/05/24 15:15:32z SPIFFS mounted.^M
^M
** Restart **^M

SD initialized.^M
10/05/24 15:15:33z Real Time Clock is running. Unix time 1728141333 ^M
10/05/24 15:15:33z Reset reason: Exception^M
10/05/24 15:15:33z Trace:  34:30, 34:30, 34:10[3], 34:30, 34:10[4], 34:30, 34:10[5], 34:30, 34:10[7], 34:30, 34:10[8], 34:30, 34:10[9], 34:30, 34:10[10], 34:30, 34:10[11], 34:30, 34:10[12], 34:30, 34:10[13], 34:30, 34:10[14], 34:30, 34:5, 34:5, 34:5, 34:6[3], 34:10[5], 34:30, 34:25[103], 34:28^M
10/05/24 15:15:33z ESP8266 ID: 522967, RTC M41T81 (68)^M
10/05/24 15:15:33z IoTaWatt 5.0, Firmware version 02_08_03^M
10/05/24 15:15:33z SPIFFS mounted.^M
^M

That last log block then repeats verbatim other than the timestamps (trace is identical) every 1-2 seconds, over 34,000 times (until I unplugged it).

There is one other different trace, which appears twice. I cannot correlate the two timestamps (16:01:17 UTC and 00:25:07 UTC the next day) to anything I did. I was out of the house at both times, and as far as I know the iotawatt was undisturbed.

SD initialized.^M
10/05/24 16:01:17z Real Time Clock is running. Unix time 1728144077 ^M
10/05/24 16:01:17z Reset reason: Exception^M
10/05/24 16:01:17z Trace:  11:30[2], 11:31, 11:31[1], 11:31[2], 34:5, 34:5, 34:5, 34:6[3], 34:25[45], 34:28, 34:30, 34:5, 34:5, 34:5, 34:6[3], 34:10[1], 34:30, 34:10[2], 34:5, 34:5, 34:5, 34:6[2], 34:25[43], 34:28, 34:5, 34:5, 34:5, 34:6[3], 34:25[117], 34:28[2], 34:30, 34:10[5]^M
10/05/24 16:01:17z ESP8266 ID: 522967, RTC M41T81 (68)^M
10/05/24 16:01:17z IoTaWatt 5.0, Firmware version 02_08_03^M
10/05/24 16:01:17z SPIFFS mounted.^M
^M
** Restart **^M

The only other notable thing I found was that the last modified times of the all the files on the SD card was either Dec 31, 1999, Dec 31 2097, or Jan 21, 2098. Obviously those are all wrong, but I don’t know if that’s expected, harmless wrongness, or perhaps a sign that there’s something wrong with the clock.

I have not yet tried flashing a clean SD card image. It sounds like there should be instructions/a download for that somewhere I can find, so I’m assuming that’s something I can do myself. I’ll post again with the results once I try that.

[overeasy]

The exception occurs while processing the configuration file (config.txt). My best guess, without actually seeing the config file, is that your SDcard is worn out or otherwise damaged and the config file is compromised. Building a new SDcard from scratch may resolve the problem.

You might consider copying the history file initially to see if it may be undamaged and thus retain your history. (/iotawatt/histlog.log).

IoTaWatt does not maintain accurate file date/times on the SDcard. It doesn’t always know the date and time when creating files during startup.

Remove/replace SDcard

Sometimes it becomes necessary to remove the microSD card. You may need to replace it or look at the message log if the device isn’t working, or maybe you want to back up the datalogs. For whatever reason, it’s best to be careful as the type of holder and position makes it easy to damage during removal. Power down the IoTaWatt by removing the USB power plug and AC reference plug. If mounted to a wall, remove the mounting screws. Remove the four Philips head screws in the back. Remo…

Building a new SDcard

It can become necessary to build a new SDcard. This post describes how to do that. IoTaWatt uses a microSD card, formatted to FAT32. New units use a 8GB level 10 card. Any card 4GB-32GB level 4 or 10 will work fine. Obtain a new microSD card or reformat an old card to FAT32. Download the SD files from github SD directory If using windows, this link will downlod the files into a .zip file. Copy the files to the root of the SDcard. If you were able to salvage the config.txt file from the ol…

[mbbush]

Thanks for the quick reply!

Here’s what I can see as far as the config.txt file, when reading the original SD card mounted on my computer:

• It is properly-formatted JSON (as reported by jq)
• The config-1.txt is also properly-formatted JSON, and the only difference corresponds to the last change to my configuration (about a month ago I added a new output).

This is my config file. The only change I made was to the value of the influxdb2 authtoken. The original token is the same length, with the same number of padding characters, and when I decoded it with url-safe base64, I get an inscrutable sequence of bytes.

config-redacted.txt (5.4 KB)

I’ll try the new SD card build instructions, and see what I can salvage.

[mbbush]

It’s definitely something to do with the config file.

With a clean SD card (following initial setup instrucitons), the iotawatt boots normally. When I copy my original config.txt file and make no other changes, it starts bootlooping again.

The last change I made to the config was adding the Solar_charge output, with this config

    {
      "name": "Solar_charge",
      "units": "Watts",
      "script": "@5-!grid"
    },

When I remove that output from the config file, it boots normally again. I’m pretty sure this is the culprit, although I don’t understand why.

I now have reproducible steps to cause the bootloop:

• Set up a new output as (car)-(grid.net) (car is an input, grid is another output
• The Iotawatt accepts the output configuration as valid, and writes it to disk.
• The next time it reboots it enters a bootloop.

Setting up the output as -1 * grid.net + car (which is mathematically equivalent) does not produce the bootloop. This is serialized to the config file as #-1*!grid+@5.

[overeasy]

I’ve tried those steps to reproduce but no joy. The original trace shows the exception while encoding the script, but that said, all of the scripts get encoded every time you make any update to the config, so if the change is accepted and only goes into the “bootloop” upon subsequent restart, there is more to it.

One thing that I notice is that you state that by referencing “grid” in an output, you say you are referencing another output. It doesn’t work like that. You are actually referencing the integrator by the same name. I can’t see where that makes a lot of difference with respect to this problem, but I would rather see it without the ambiguity i.e. use unique names for the output “grid” and the integrator “grid”. Maybe name the “grid” integrator “int_grid” or something like that.

Similarly you have both an output and an integrator named “usage”, so maybe “int_usage”?

Finally, usage should always be a monotonically increasing value, so you should not need an integrator for that. Integrators add complexity and overhead and their use should be minimized. The ESP8266 has limited resources.