How to upgrade local storage

1

How many months of data can the default SD card store if I’m using all 14 inputs? I’d like to be able to get charts for at least 2 years so I can compare year over year. I setup an Emoncms server locally and am pushing the data to that but it really feels like overkill.

I’ve found the Graph+ support in IotaWatt to be better/simpler and provides everything I’ve been looking for. It also doesn’t require me to run a DB, Redis server, and web server, configure all the virtual feeds, create dashboards, etc and then debug a much more complex system when things inevitably don’t work.

If I take a 64 or 128 GB SD card and format it FAT32 then copy my current config/data over from the old card, will IotaWatt take advantage of the additional space? Is there a way to upgrade the local storage so I can get rid of my Emoncms server?

2

The short answer is that the default capacity of 10 years will easily fit on the supplied 8GB SDcard.

The current log, which holds 5 second data, needs about 8.8MB/day. It’s limited to about 485 day’s worth of data by the allowable file size in the FAT file system. Once full, it wraps and replaces the oldest data with new. As shipped, the size of the current log is set to 365 days, but that can be extended in the configuration file (not recommended).

The history log is identical to the current log, except that the data is 60 second intervals, and it is set to wrap at 3650 days (ten years). It could, in theory, hold about 16 years. Another key difference of the history log is that it is built from the current log, and does not have any holes. That means that access is direct and immediate. The current log can have holes, and so retrieval can be slowed by searching.

It doesn’t matter how many inputs you are using. The datalog records the state of all 15 channels in each entry.

Here are my home logs. I had my current log very small for awhile to exercise the wrap logic, but I reset it’s limit and it’s now growing to a year.

image

I’m happy to hear you like Graph+. The old graph program was a port from Emoncms, and while it works well with that database, it didn’t have the ability to access all of the metrics available as standard in IoTaWatt. Since Graph+, I find myself using it for everything as well.

But the underlying new query API is the real power. It paves the way for other tools that could, for instance, summarize kWh for all channels over any selected period, and compare to another period. Because of the way the datalog is structured, a query of kWh used each month of a year compared to the previous year would take less time than plotting your daily power.

Here’s my last three years. You can see how the minisplit factors into the winter usage.

image
image1320×1011 37.3 KB

2 Likes
3

Fantastic! I’m super excited to know the history log can hold 10 years of data - that is much more than I expected. I really don’t see any need for keeping my Emoncms install. That will be nice to shut down as I no longer need it. Now I’m wondering about backup/restore…

I looked at the file manager tool and see that you can upload files to the SD card but couldn’t find a way to backup the current data. Is that available and I’m just missing it or is it a feature request for a future release? :slight_smile:

Thank you so much for all the work you have put into this great product.

4

The only way to backup the datalogs is to shut down the IoTaWatt, remove the SDcard, and copy the files to another media. That may be something you would do annually, but not anything I would expect anyone to do frequently.

Early on in the project, a few folks lost their datalogs. People were quick to blame the SDcards as unreliable and failure prone, but I don’t see that as the case. Those failures were caused by problems in the sequencing of updates and being in an unreliable state on power failure. That has been remedied for some time now.

SDcards can only handle a certain number of writes per sector, but that number is in the tens of thousands per sector. Moreover, they all have a mapping layer that does wear leveling and error detection. With CRC, many errors can be corrected, and the mapping layer can take that sector out of service. If you think about the billions of cards in use just in cameras and phones, they have to have integrity. In fact, I would say that manually moving the card to back it up presents more risk of losing data through physical damage than through spontaneous failure. Each of the datalog blocks is written just twice per year.

All that said, backup would be a nice feature. Unfortunately, the IoTaWatt is busy 67% of the time sampling your power, and backing up 8GB/day would be a pretty serious undertaking. There are other issues with the datalog that should be addressed. Like being able to aggregate the logs from multiple IoTaWatt seamlessly.

The ESP32 has more potential to do that. First, it has SDMMC, which has four times the throughput for copying. Second, it has a two processors, so one can do the sampling, and the other can take it’s time to transfer large amounts of data. That works for backup, and also for large data queries.

In the meantime, cross your fingers or upload your data to a server.

2 Likes