Right problem, wrong solution. Here’s what’s going on:
When IoTaWatt starts or restarts the uploader, it queries Emoncms for the timestamp of the most recent data written to the inputs for the specified node.
Emoncms uses a caching system called Redis to maintain the inputs database in memory. If anything happens to Emoncms like a power failure or even a restart, the Redis cache is lost. That is not a problem for the Open Energy Monitor products because they do not seamlessly upload history, they are real-time and have no history to upload. When they write new data to the inputs, the inputs are recreated with the new data, timestamps etc.
So, uploading seamless history requires that the inputs be preserved in order to determine where it left off.
Fortunately, there is a solution. Redis has a feature called persistence that records the contents of the database on the SD. There are two different ways to do it:
- RDB (Redis Database): RDB persistence performs point-in-time snapshots of your dataset at specified intervals.
- AOF (Append Only File): AOF persistence logs every write operation received by the server. These operations can then be replayed again at server startup, reconstructing the original dataset. Commands are logged using the same format as the Redis protocol itself.
The reason OEM uses Redis is to minimize SD wear caused by writing every time data is posted. The RDB option with a snapshot interval of an hour would cause negligible increased SD activity but would restore the inputs after a restart to an average of 30 minutes old. IoTaWatt would restart it’s upload from the last update as of the time of the last RDB snapshot.
I don’t know the mechanics of setting Redis persistence in the various flavors of Emoncms, but you could ask on their forum or if you are a RPi/Linix person, figure it out for yourself.