Possible SD card issue? (Or user error)

yngndrw · June 4, 2023, 4:34pm

My IotaWatt has been working great for years, then I touch things and things start going wrong - Opps.

Anyway I think this time it’s bad timing, but I of course can’t rule out user error.

So as I say, my IotaWatt has been running great for years. Recently (About 6 months ago) we moved to a property with solar so I set everything up again and it continued to work well. I use the local graphs a lot and I also export to InfluxDB.

I recently re-did some of my collected metrics (Added Power Factor and Mains Current) so I cleared down InfluxDB and set off the exporter again - This has taken a few days to re-process and is still in progress as we speak but seems to be going fine.

I then remembered that I still have some EmonCMS credits from when I first purchased my IotaWatt so I thought I’d set that up again now that I can take advantage of their solar dashboard. When I did this it seemed to take a very long time before it started to export historical data, then suddenly out of nowhere the whole IotaWatt UI stopped responding. A restart didn’t help, but in the end I blocked emoncms.org in my DNS server and restarted the IotaWatt which forced the EmonCMS upload to fail and therefore allowed the UI to work again.

Upon checking the logs, I noticed the following: (The first is when it restarted with access to emoncms.org, the second is after emoncms.org was blocked)

** Restart **

SD initialized.
6/04/23 08:33:50z Real Time Clock is running. Unix time 1685867630 
6/04/23 08:33:50z Reset Reason: Power-fail restart.
6/04/23 08:33:50z ESP8266 ID: 6910659, RTC PCF8523 (68)
6/04/23 08:33:50z IoTaWatt 4.x, Firmware version 02_07_05
6/04/23 08:33:50z SPIFFS mounted.
6/04/23 08:33:50z Local time zone: +0:00
6/04/23 08:33:50z device name: IotaWatt
6/04/23 08:33:53z Connecting with WiFiManager.
6/04/23 08:33:57z HTTP server started
6/04/23 08:33:57z emoncms: Starting, interval:10, url:http://emoncms.org
6/04/23 08:33:57z influxDB_v2: Starting, interval:5, url:http://192.168.1.187:8086
6/04/23 08:33:57z WiFi connected. SSID=yngndrw, IP=192.168.1.130, channel=11, RSSI -61db
6/04/23 08:33:57z MDNS responder started for hostname IotaWatt
6/04/23 08:33:57z LLMNR responder started for hostname IotaWatt
6/04/23 08:33:57z timeSync: service started.
6/04/23 08:33:57z statService: started.
6/04/23 08:33:57z Updater: service started. Auto-update class is MAJOR
6/04/23 08:33:57z dataLog: service started.
6/04/23 08:33:59z dataLog: Last log entry 06/03/23 22:08:10
6/04/23 08:33:59z historyLog: service started.
6/04/23 08:33:59z IotaLog: file damaged /iotawatt/histlog.log

6/04/23 08:33:59z IotaLog: Creating diagnostic file.

** Restart **

SD initialized.
6/04/23 08:39:02z Real Time Clock is running. Unix time 1685867942 
6/04/23 08:39:02z Reset Reason: Power-fail restart.
6/04/23 08:39:02z ESP8266 ID: 6910659, RTC PCF8523 (68)
6/04/23 08:39:02z IoTaWatt 4.x, Firmware version 02_07_05
6/04/23 08:39:02z SPIFFS mounted.
6/04/23 08:39:02z Local time zone: +0:00
6/04/23 08:39:02z device name: IotaWatt
6/04/23 08:39:05z Connecting with WiFiManager.
6/04/23 08:39:08z HTTP server started
6/04/23 08:39:08z emoncms: Starting, interval:10, url:http://emoncms.org
6/04/23 08:39:08z influxDB_v2: Starting, interval:5, url:http://192.168.1.187:8086
6/04/23 08:39:08z WiFi connected. SSID=yngndrw, IP=192.168.1.130, channel=11, RSSI -62db
6/04/23 08:39:08z MDNS responder started for hostname IotaWatt
6/04/23 08:39:08z LLMNR responder started for hostname IotaWatt
6/04/23 08:39:08z timeSync: service started.
6/04/23 08:39:08z statService: started.
6/04/23 08:39:08z Updater: service started. Auto-update class is MAJOR
6/04/23 08:39:08z dataLog: service started.
6/04/23 08:39:10z dataLog: Last log entry 06/03/23 22:08:10
6/04/23 08:39:10z historyLog: service started.
6/04/23 08:39:10z historyLog: Last log entry 06/03/23 22:08:00
6/04/23 08:39:13z Annexe_And_Outbuildings: Started
6/04/23 08:39:13z Annexe_And_Outbuildings: Last log entry 06/03/23 22:08:10
6/04/23 08:39:13z Mains: Started
6/04/23 08:39:13z Mains: Last log entry 06/03/23 22:08:10
6/04/23 08:39:30z emoncms: Query failed -4
6/04/23 08:39:31z influxDB_v2: Resume posting 04/24/23 12:24:10
6/04/23 08:39:32z Updater: Auto-update is current for class MAJOR.
6/04/23 08:39:35z emoncms: stopped, Last post 01/01/70 00:00:00

(Notice the line: IotaLog: file damaged /iotawatt/histlog.log)

Here’s the logdiag.txt:

1534537140,0,1534621680,1409
1534779180,4034,1534779180,4034
1534621800,1411,1534840680,5059
-1,-1,-1,-1
-1,-1,-1,-1
1534840860,5062,1534840920,5063
-1,-1,-1,-1
1534841040,5065,1534841160,5067
-1,-1,-1,-1
-1,-1,-1,-1
-1,-1,-1,-1
1534841400,5071,1534841640,5075
-1,-1,-1,-1
1534841760,5077,1534841880,5079
-1,-1,-1,-1
1534842000,5081,1534842360,5087
-1,-1,-1,-1
-1,-1,-1,-1
1534842540,5090,1534842840,5095
-1,-1,-1,-1
-1,-1,-1,-1
-1,-1,-1,-1
1534843080,5099,1534947960,6847
1534947960,6847,1534947960,6847
1534947960,6847,1534947960,6847
1534948140,6850,1534948440,6855
1534948440,6855,1534948440,6855
1534948560,6857,1534948560,6857
1534948560,6857,1534948560,6857
1534948560,6857,1534948560,6857
1534948740,6860,1534948740,6860
1534948740,6860,1534948740,6860
1534948740,6860,1534948740,6860
1534948740,6860,1534948740,6860
1534948740,6860,1534948740,6860
1534948740,6860,1534948740,6860
1534949100,6866,1534949160,6867
1534949160,6867,1534949160,6867
1534949160,6867,1534949160,6867
1534949340,6870,1534949340,6870
1534949340,6870,1534949340,6870
1534949460,6872,1535063160,8767
1675562567,2101521413,1675562567,2101521413
-419619746,485709326,-419619746,485709326
939126646,-635383972,939126646,-635383972
1674953954,-1053202714,1674953954,-1053202714
2023554401,-1408503608,2023554401,-1408503608
978931501,412967780,978931501,412967780
-2000364006,-826998137,-2000364006,-826998137
-1466239020,-2066662334,-1466239020,-2066662334
-1992038488,-413979274,-1992038488,-413979274
1535132880,9929,1535136120,9983

I’m presuming that the reason the EmonCMS upload failed was because the history file is now corrupt and that I have a bad (Or failing) SD card.

Interestingly, there’s now another issue that has cropped up in regular use.

The way I have it setup is that I have a Mains and Annexe_And_Outbuildings input (Amongst others) - The Mains input is measured at the meter and the Annexe_And_Outbuildings input includes both the annexe usage as well as the solar inverter.

I have an integrator for both Mains and Annexe_And_Outbuildings as both of these can go negative, and I have the following output:

Generated	Watts	= Annexe_And_Outbuildings.neg x -1

If I look at the status page, the Generated value looks correct, however now when I look at the built-in graphs Generated is always 0 for both Watts and WattHours. Annexe_And_Outbuildings is correct for both Watts and WattHours on the graph.

Generated used to work just fine, but I’m wondering if the SD card has started to fail and is now affecting the integrators.

I should add that the plain inputs themselves are still recording without issue.

If this is likely to be the SD card and I need to purchase a replacement, please can you advise on a particular model. I don’t care about how much it costs, I’d rather get something which was more reliable even if it means a higher cost of replacement. Maybe a Sandisk Max Endurance 32GB?

overeasy · June 4, 2023, 5:57pm

The logdiag file indicates two kinds of damage:

You have a very old unit and the history log has been maintained for nearly five years. Unfortunately in those early years there were firmware issue related to how it handled power failures and otherwise sequenced access to the SDcard that caused errors in the file and see some of those types of errors (the -1,-1,-1,-1). When writing SDcards initially set all the bits to 1’s and then turn off the zero bits. -1 is binary 0xFFFFFFFF.

Farther along, there are just bogus blocks and I suspect large parts of the file just missing. This is probably just worn out SDcard. I believe the cards in that vintage machine were just standard 4GB or maybe 8GB Sandisk.

So yes, you should probably install a new SDcard. You may get away with copying everything from the old card to the new, but I would recommend leaving the histlog.log file behind. You say you moved to a new house anyway so that history is probably no longer useful anyway. Your current log spans about 18 months. It has a capacity of 12 months so you must have had the unit off for 6 months or so. In any event, the IoTaWatt will create a new histlog from the existing currlog. It will take a long time. You will be able to see the progress in the status display. I’d recommend you stop the influx upload for the few days it may take to rebuild the histlog.

The MAX Endurance appears looks fine. I’m currently using Sandisk Industrial SDHC 8GB class 10. They explicitely state that the card has active wear leveling. Not all cards have it and as I understand it not all manufacturers use the same technology. Be sure it’s formatted to FAT32 or do so with the SDcard Association formatter.

I do not recommend running long history uploads on multiple uploaders simultaneously. Finsh the influx upload until current, then do the Emoncms, but stop both while the history log is rebuilt.

Are you using Graph+? Show me.

As an aside, you can make Generated positive more efficiently with either:

Generated = Annexe_And_Outbuildings.neg abs or
Generated = 0 - Annexe_And_Outbuildings.neg

rather than doing signed multiplication.

yngndrw · June 4, 2023, 10:01pm

Thank you.

You’re right in that most of the history is of little use to me, I hope to be able to recover the 6 months of data from the new house though so I’ll try copying everything expect histlog.log as you suggest and I’ll stop the uploads during that process. (I think the InfluxDB upload will have finished by the time the new card arrives which is handy, although I’ll need to double check if all the data has gone across)

I’ve checked the Sandisk Max Endurance datasheet for the wear levelling and it appears that it’s really aimed at pure write endurance for video in scenarios such as CCTV or dash cams. It doesn’t mention wear levelling at all so I’ll give that one a miss.

I struggled to find a handy retailer (Amazon) for the Sandisk Industrial cards but I did find them on Mouser: https://www.mouser.co.uk/ProductDetail/467-SDSDQAF3-032G-XI

I did however find a Kingston Industrial card which also has wear levelling and is available on Amazon so I’ll go for that. It’s worth noting that write endurance (And hopefully the effectiveness of wear levelling) scales with storage and that on Amazon, the 64GB card (With SD card adapter) is cheaper than the 32GB card so I’ll pick up one of them for £35:

I don’t know if the IotaWatt will accept a 64GB volume but if not, hopefully the card will internally use the free space for wear levelling.

Yes I’m using Graph+. (I really like the built-in graphing btw)

The graphs for Watts and WattHours align as you’d expect but as you can see from the graph, there’s a tiny blip on the Generated line right after I restarted it with the blocked EmonCMS DNS entry but then it just stays on zero.

And thank you for the advice, I’ll update my calculations.

yngndrw · June 4, 2023, 10:09pm

Just realised there’s a few other graph values which are now wrong - Again all of these used to work before the issue.

Setup:

Graph+:

Mains has an integrator (Named the same as the input, but that used to work fine - Is that unwise?), Mains_Billed is meant to be the positive component of Mains (Import) and Battery_Required is meant to be the negative component of Mains (Export).

Yet on the graph Mains_Billed is showing a little import at the start when Mains was negative and Battery_Required shows nothing at all. It’s like both integrators are completely messed up.

Here’s the same graph over a longer period - You can see how they used to be fine, but the integrator seems to have gone crazy while it was off:

overeasy · June 4, 2023, 11:17pm

I do not recommend >32GB cards. It may work if formatted down to 32GB FAT32 but if you have trouble the first thing I will tell you to do is get a smaller card.

Can you plot the annex_and_outbuildings input for the same period?

yngndrw · June 4, 2023, 11:46pm

Ahh well I’ve already ordered it now so I’ll format it to 32GB and give it a go, I’ll let you know if it works or not and will order a smaller one if I have any issues with it.

Here’s the longer (3 days) period with the Annexe_And_Outbuildings input plotted in green:

I’ve also included a more focused view of the problematic area just after the restart where you can see that the Mains_Billed plot doesn’t react to the short import periods:

Finally here’s the normal (Working) period covering a couple of days in a little more detail for comparison:

overeasy · June 5, 2023, 12:41am

Not exactly sure how this is getting messed up, but I’m pretty sure it has to do with the integrators Mains and Annexe_And_Outbuildings having the same name as inputs and also one being longer than eight characters.

I’d suggest you:

Delete the integrators
Use the file manager to delete the /iotawatt/integrations directory
Recreate the integrators with unique name no longer than eight characters

yngndrw · June 5, 2023, 1:33am

Okay I’ve done that, I’ve also stopped the InfluxDB uploader and have deleted the data since the restart. I’m just waiting for the integrator logs to be re-generated and then I’ll have a look at the graph. (Right now it just shows zero for anything using an integrator.)

yngndrw · June 5, 2023, 9:32am

Okay so after re-generating the integrators with shorter / unique names, they appear to be working again:

Interestingly it hasn’t re-generated them for historical data where they had been working fine, but it has regenerated them for the period when they were zero. It’s almost as if the integration process wasn’t running at all for that period.

Is there a way to force the integration regeneration for the entire current log? (Probably one for when the new SD card arrives)

yngndrw · June 6, 2023, 10:25pm

Received my new SD card today but as you suspected it didn’t work even when formatted as 30GB. In hindsight it’s to be expected as it’s SDXC rather than SDHC.

I’ve ordered a 16GB card instead, same type - Will report back when it arrives:

yngndrw · June 8, 2023, 7:31pm

The 16GB card has arrived, it was already formatted as FAT32 so I copied over the files (With the deleted history log) but found that the powerlight was flickering green and occasionally turning red. Additionally the status page wasn’t showing anything even though the input configuation page was populated. Finally the message log was showing restarts due to an exception.

I then cleared down the entire iotawatt directory and tried again but with the same results.

** Restart **

SD initialized.
6/08/23 19:26:43z Real Time Clock is running. Unix time 1686252403 
6/08/23 19:26:43z Reset reason: Exception
6/08/23 19:26:43z Trace:  10:14, 10:14, 10:15, 10:16, 10:16[1], 34:30, 34:30, 33:0, 34:30, 10:16[1], 34:30, 34:30, 33:0, 34:30, 10:16[1], 10:16[1], 34:30, 34:30, 33:0, 34:30, 10:16[1], 10:16[1], 10:16[2], 10:17, 10:17, 31:110, 31:110, 10:18, 31:110, 31:110, 10:23, 10:17
6/08/23 19:26:43z ESP8266 ID: 6910659, RTC PCF8523 (68)
6/08/23 19:26:43z IoTaWatt 4.x, Firmware version 02_07_05
6/08/23 19:26:43z SPIFFS mounted.
6/08/23 19:26:43z Local time zone: +0:00
6/08/23 19:26:43z device name: IotaWatt
6/08/23 19:26:43z HTTP server started

** Restart **

SD initialized.
6/08/23 19:31:14z Real Time Clock is running. Unix time 1686252674 
6/08/23 19:31:14z Reset reason: Exception
6/08/23 19:31:14z Trace:  10:14, 10:14, 10:15, 10:16, 10:16[1], 34:30, 34:30, 33:0, 34:30, 10:16[1], 34:30, 34:30, 33:0, 34:30, 10:16[1], 10:16[1], 34:30, 34:30, 33:0, 34:30, 10:16[1], 10:16[1], 10:16[2], 10:17, 10:17, 31:110, 31:110, 10:18, 31:110, 31:110, 10:23, 10:17
6/08/23 19:31:14z ESP8266 ID: 6910659, RTC PCF8523 (68)
6/08/23 19:31:14z IoTaWatt 4.x, Firmware version 02_07_05
6/08/23 19:31:14z SPIFFS mounted.
6/08/23 19:31:14z Local time zone: +0:00
6/08/23 19:31:14z device name: IotaWatt
6/08/23 19:31:14z HTTP server started

** Restart **

SD initialized.
6/08/23 19:31:24z Real Time Clock is running. Unix time 1686252684 
6/08/23 19:31:24z Reset reason: Exception
6/08/23 19:31:24z Trace:  10:14, 10:14, 10:15, 10:16, 10:16[1], 34:30, 34:30, 33:0, 34:30, 10:16[1], 34:30, 34:30, 33:0, 34:30, 10:16[1], 10:16[1], 34:30, 34:30, 33:0, 34:30, 10:16[1], 10:16[1], 10:16[2], 10:17, 10:17, 31:110, 31:110, 10:18, 31:110, 31:110, 10:23, 10:17
6/08/23 19:31:24z ESP8266 ID: 6910659, RTC PCF8523 (68)
6/08/23 19:31:24z IoTaWatt 4.x, Firmware version 02_07_05
6/08/23 19:31:24z SPIFFS mounted.
6/08/23 19:31:24z Local time zone: +0:00
6/08/23 19:31:24z device name: IotaWatt
6/08/23 19:31:24z HTTP server started

This is using a 16GB SDHC card so it should be compatible. Have I somehow managed to break my IotaWatt?

overeasy · June 8, 2023, 7:38pm

There appears to be something amiss with your influxDB_V1 setup. If you upload the config.txt file in a PM to me I can take a look at it. The PM is private, so please don’t modify it in any way.

OOPS: Followed wrong path in trace. This has something to do with the status display request. One of the datalogs current, history or integrator(s).

Can you try booting it up without the app on “status”. (click setup->device->cancel) or just terminate the app until the device is solid green for a few minutes.