How exactly IotaWatt uploader works during internet or server shutdown?

I know in a normal situation, the IotaWatt sends data to Emoncms in intervals and bulk settings specified in the configuration panel and selects the sending time based on the date specified in the upload history from. But, in the case of internet shutdown, server shutdown, and device shutdown, does the IotaWatt continue to start uploading from the last successful post or does just it back to the first time specified in the history field? I saw in some cases the device backs to very old dates and It seems to send redundant data to the Emoncms. Why does this behavior occur? I mean what is the reason behind this behavior so that the device loses the track of the last successful post?

It’s been awhile since this topic has come around, so I’ll save you the forum research and go through it again.

To specifically answer your question, When the Emoncms uploader starts or restarts, it queries Emoncms for the state of the inputs that the IoTaWatt uploader is configured to send. Emoncms will return the date/time of the last data to each input that was posted. The uploader will resume from the most recent of those date/time responses.

Seems simple enough right? So why do you see it sometimes upload from the “upload history from” date sometimes? This is where it gets a little more complicated.

Without a doubt you are using your own instance of Emoncms.This doesn’t happen with Emoncms.org. Local instances of Emoncms use an in memory cache of the inputs to avoid excessive writing to an SDcard, which reduces wear and increases longevity. That cache mechanism is provided by a software component called Redis that replicates and maintains the inputs database in memory. If your Emoncms instance is restarted, the cache will be wiped and when the uploader queries it will look like the input is new and upload will restart from the “upload from” date. If you restart just the IoTaWatt, the Redis database in Emoncms is intact and will return the current date/time for restart.

That may sound complicated, but there is a simple solution. Redis has the ability to record periodic checkpoints that will be used to restore the in memory database when Emoncms is restarted. It’s called Redis Persistence. If you configure RDB persistence in your Emoncms with an interval of about an hour, your uploads will be set back at most an hour when Emoncms is restarted. There will be 24 writes per day to an SD, which is trivial, and there is no real problem with sending the redundant data for the last hour aside from the relatively short time it will take.

I have never researched how to actually set this in Redis as I use Emoncms.org. It would be helpful to the community if someone could provide stepwise instructions for setting it up. I believe it’s just a couple of lines in a configuration script somewhere.

1 Like

Thanks, @overeasy. Your explanation was quite clear to me. This is an important problem to me since I really do not care about the number of writes to the SD card as my Emoncms has been installed on a VPS, and returning back to the old dates because of Redis configuration makes me uncomfortable as I have to wait for all the redundant data to be sent while the status of my device is Inactive in Emoncms feeds. As is evident from your response, I can conclude that this behavior (i.e., returning to the old dates) is only related to the Emoncms configuration and does not have any relation to the device or internet shutdown. Right?. Also according to your suggestion, I will follow the RDB persistence configuration in my Emoncms, and I will inform you in case of any success. Another important question is, does this behavior occur in InfluxDB too?

The simple approach in your case could be to set Redis to say one minute checkpoints.

Influx uses a similar query system to synchronize restart but there is no Redis like cache so it works on cold restart.

Dear @overeasy. I followed your instructions about RDB persistence setting in Emoncms, and I found the Redis script, where the Redis persistence is disabled. You can uncomment the following lines in redis.sh of installation to enable Redis persistence. By default this setting is enabled in Redis, but EmonCMS installer disables them.

sudo sed -i “s/^save 900 1/#save 900 1/” /etc/redis/redis.conf
sudo sed -i “s/^save 300 1/#save 300 1/” /etc/redis/redis.conf
sudo sed -i “s/^save 60 1/#save 60 1/” /etc/redis/redis.conf

1 Like

Influx v2 user here - I found an interesting side effect with tags. Quick answer for @ahmad I run Influx on a PC and wished I could resend iota data - once.

I had Uploader configured with only a few CTs and no tags. Worked great. A month later I wanted to make sure all the CTs were sent to Influx, but uploads only sent the full set of fields to influx from the date of the updated config, not historic ones. Did I really need to make a new bucket and define a new Uploader to start from day one? Apparently not!
One clever author here posted pictures of their Uploader config and I decided Yes I should add tags. At the next uploader restart IotaWatt seems to think that these datapoints had never been uploaded… I could see Influx showing the oldest data appear with the new tags, alongside tag-less fields with the exact same readings. The following day (influx internal cleanup??) all datapoints had the tags. I only did this once and didn’t take notes - and I did create new buckets in the process.
Hope that answers some Influx questions or gives people useful ideas.
(that’s the good part of my story… the upload hangs at some later date so I’m exploring the SD card issue now…)