I was wondering if a small tweak to the EmonCMS upload history option might be made where we could specify it only go back 2 weeks and as the days roll by that updates to hold to only 2 weeks ago. I’ve found that sometimes if the Iotawatt gets disconnected from EmonCMS when it reconnects it will go back to that upload history date which sometimes can be as far back as a year when it didn’t need to.
Right problem, wrong solution. Here’s what’s going on:
When IoTaWatt starts or restarts the uploader, it queries Emoncms for the timestamp of the most recent data written to the inputs for the specified node.
Emoncms uses a caching system called Redis to maintain the inputs database in memory. If anything happens to Emoncms like a power failure or even a restart, the Redis cache is lost. That is not a problem for the Open Energy Monitor products because they do not seamlessly upload history, they are real-time and have no history to upload. When they write new data to the inputs, the inputs are recreated with the new data, timestamps etc.
So, uploading seamless history requires that the inputs be preserved in order to determine where it left off.
Fortunately, there is a solution. Redis has a feature called persistence that records the contents of the database on the SD. There are two different ways to do it:
- RDB (Redis Database): RDB persistence performs point-in-time snapshots of your dataset at specified intervals.
- AOF (Append Only File): AOF persistence logs every write operation received by the server. These operations can then be replayed again at server startup, reconstructing the original dataset. Commands are logged using the same format as the Redis protocol itself.
The reason OEM uses Redis is to minimize SD wear caused by writing every time data is posted. The RDB option with a snapshot interval of an hour would cause negligible increased SD activity but would restore the inputs after a restart to an average of 30 minutes old. IoTaWatt would restart it’s upload from the last update as of the time of the last RDB snapshot.
I don’t know the mechanics of setting Redis persistence in the various flavors of Emoncms, but you could ask on their forum or if you are a RPi/Linix person, figure it out for yourself.
In my case I’m actually running EmonCMS in a container on my server. RAID 6 spin drives for storage. So really more writes isn’t a problem for me. I might see about what, if any, setting might be changed to allow for it to write the cache more frequently. Would be a good option for people not using RPis.