Is it possible to make a backup of all the logs (more then 2 years) without removing the SD-card. Can I do it from the ‘file manager and editor’ over the browser? Witch files have to be copied?
I couldn’t find an answer here.
Thanks.
It’s not possible right now. A watchdog timer will restart the IoTaWatt after 5 minutes.
You would need to power down the IoTaWatt and carefully remove the SDcard. Then mount it on a computer and copy all of the files and directories to a directory on your computer. It doesn’t take very long. Then replace the SD and restart the IoTaWatt.
99% of the card contents are the datalogs, so just copy everything. Then if your card should fail, you can simply copy the entire backup to a new card and pickup from the point of the backup.
I’m working on adopting some improvements that have the potential of speeding things up enough to allow downloading the history file in a reasonable amount of time. If that works out it should be available in a few months.
Be careful removing the card, they break easily if not pulled straight out.
Thanks a lot. You are doing a great job.
Could you estimate, how long an SD-card wil work? Would it be usefull to change the card precautionary? What specifications are needed to get a card with maximum lifecycles?
IoTaWatt doesn’t cause a lot of wear to an SDcard. Most blocks get written once or twice per year and a typical block should have a life of 100K or so writes. Reads are free.
I’m told there are grades of quality. I’m not so sure about that. I don’t know of any longevity metrics that they are certified to. There are various claims by the brands, but as far as I know there is no independent verification of that. There are definitely faster cards, and I’ve found there is slightly better response in Graph+ with an SDHC class 10 card.
One of my systems has been using the same card for 4 years. It is one of the inexpensive class 4 Sandisk cards that is supplied with a new IoTaWatt. You probably have a higher probability of losing your data by damaging it during removal.
@overeasy, this is an interesting thread, and I just read these as well:
I’m coming up on my 1-year anniversary of my first IoTaWatt (yea!) and have not yet exported my data to the supported external databases. So, I was thinking of using the APIs to extract all the data to save it before the IoTaWatt runs out of space and may flush the oldest data out.
So, from what I gather from these threads, a 1-minute interval is what I should use for the API calls (that’s sufficient resolution for me for long-term data)?
Alternatively, should I decide to remove the microSD card and copy it, what format are the datalogs / history logs, that I may extract the data from there manually?
And, for completeness here, docs say:
https://docs.iotawatt.com/en/02_05_12/query.html
“The datalog contains measurements at 5 second (current log) and 1 minute (history log) intervals.”
I’m not finding how long the datalog goes back in time for the 5-second resolution…?
The Data Logs tab at the bottom of the status display shows the start and end dates of the current and history logs. When the current log fills up, it wraps so you will see the begin date advance each time the a new entry overwrites the oldest entry.
It’s not advisable to download the logs with query. Better to upload to an external batabase with the uploaders. IMO old 5 second data doesn’t have much value. Typical queries of historic data are looking for energy (kWh) rather than power (Watts). TBH I never look at Watts more than a week back, and Wh at less than a minute, or even less than an hour, is micro-managing and not very useful.
You can do that to backup. It’s typically pretty fast. I’d recommend bathing the history log. It’s smaller than the current and really all you need. The format is not anything I’d try to explain in a forum post. It’s pretty complicated and also there are issues with the little-endian byte order of the ESP8266.
But you can restore a backup log to your SDcard. The proper way is to remove the SDcard, delete both logs, and then copy in the backed up log.
I also want to back up the history log and current log and config.txt files from my SDcard (which has 6 months of data). The instructions say I can use File Manager and download (I assume that means from to SD card to my PC) files by right clicking them and saving them.
However, in several threads, you state it is not possible to download the logs, as a watchdog timer will restart the IoTaWatt after 5 minutes. Does that mean the log files are too large to download in less than 5 minutes? But the config.txt could be downloaded because it is small?
My current log is 815 MB and history log is 70 MB. Will these time out or are the small enough to succeed? As you say, probably only the History log is important to backup.
A related question would be, what is the transfer rate when downloading with File Manager?
My main reason for backing up is so that if there is any kind of failure or resetting of the data, I can restore and not lose the historical solar, power company, and consumption data. I want to verify/understand San Diego Gas & Electric’s true-up calculations from net-metering and make sure it was done correctly. I do make plots with monthly data, so I can use them, but raw data in only one place is not backed up.
You also say it is not advisable to download a log with query. But you don’t say why. Then I figured out that with data at one minute intervals, there would be over 500,000 rows of data in a year’s collection. Probably too big for importing a CSV file into Excel!!
So, let me know how large a file I can download with File Manager. If it won’t work, I will just copy the card. Thanks!
The data rate will vary depending on your WiFi, browser etc. You can just initiate a download of your history file and measure your effective data rate.
My test system will transfer at about 380Kb/sec. One day of history log data is about 368Kb. So a rule of thumb might be 1 sec per day of history log.
For what you want to do, you really don’t need the full amount of data in the history log: All metrics for all inputs. Setting up an influxDB on a NAS box or RPi is pretty easy, and the IoTaWatt will continuously upload whatever you want to include.
Understand that while IoTaWatt is servicing any webserver request, it does not sample power. That’s usually not an issue as most request complete in milliseconds, but large queries, and large downloads, will pause sampling. The WTD is set for 5 minutes, and will trip if your transaction exceeds that.
When you use an uploader like influxDB, the transaction is handled asynchronously and has minimal impact on sampling.
1 Like