IotaWatt to EmonCMS reload

Support
1

So, I had a hard drive failure on with my EmonCMS setup, I had an old backup and got it reloaded. Only problem is a huge gap in the data due to the backup being back from Jan. I know that IotaWatt can reload historical data. I also know I have to just delete the inputs.

I would prefer not to if possible since to rebuild them means rebuilding the a ton of processes. Is there a way to force IotaWatt to start reloading this data. Maybe blank out the input data so when IotaWatt polls EmonCMS it gets a reply that the inputs are empty and it restarts historical data loading?

2

There are a few variables here that would determine how to proceed. Ordinarily, a user emoncms instance uses the RPi setup with an in-memory Redis database for the inputs to reduce SDcard wear. The problem with that is that it starts up with no history of the last update, so IoTaWatt query for last update time fails to provide seamless data continuity.

Once a hole is made in an emoncms feed by adding current data, it is not possible to go backfill that data. Emoncms doesn’t accept historical data with the bulk protocol.

You can try this:

  1. Stop the IoTaWatt emoncms service from the emoncms status tab.

  2. Restore the backup copy of emoncms database and restart emoncms.

  3. Manually post input data to one of the emoncms inputs with a timestamp 5 seconds after the last data in the corresponding feed.

  4. Start the IoTaWatt emonservice.

IoTaWatt should use the time of that manual posting as the starting time for data upload and proceed from there.

3

Only part I seem to be stuck at is manually posting input data. Not sure how to do that part.

4

Post the date and time of the last entries and I’ll Post the URL to use.

5

1580421600
Jan 30 2020 5:00PM

6

The various formats are here for reference.

HTTP://[emoncms hostname or IP]/input/post?time=1580421600&node=[nodename]&csv=0,0,0,0

You should receive the response “ok”

7

Awesome, that worked. If you don’t mind. I have found the api list, but haven’t studied it much.

I was looking at what you provided on that link. Would you have a quick breakdown on the link command? I can’t seem to match it with one of the listed ones.

8

IMO that’s not the best way to document an API. They are examples, each of use of a different keyword and/or endpoint. I combined the &time= keyword with the &CSV=. keyword examples.

In any event, it worked and you should be uploading the history from Jan 30 on.

9

It did. I guess my main question comes in should this ever happen again and I need to reload from a different date. The main piece I would look for from the CSV is that date code I provided and set it to that. If nothing else, this can help additional folks who have a reload being required due to HDD/SD card failure.

10

I think it would have worked to simply omit the timestamp transaction and instead set the upload history from to the )an 30 date, but there are some potential situations that I could not be sure of without thoroughly exploring your situation. This Approach was intended as a straightforward way to resolve your problem quickly.

11

I did wonder if setting it to Jan 30 would have worked, but wasn’t totally sure. If I did do that, does IotaWatt check all times on Jan 30 before sending history? Basically, finding where it ends to start the historical reload?

1 Like
12

What I expect would happen is that IoTaWatt would start the upload from 12:01am on Jan 30 and Emoncms would discard the data from prior to 5:00pm. It all hinges on the state of your Emoncms after the restore. I’m not familiar enough with how Emoncms uses Redit to essentially cache the inputs to be certain what would happen in your case. I felt it was better to eliminate the guesswork and explicitly set the last input time with that transaction.

13

Makes sense. At least now this will be available for future use if needed or for anyone who might need it.

14

I had to stop EmonCMS and restart it (forgot to add SSH access). When I stopped IotaWatt’s send to EmonCMS and restarted it I noticed it restarted back at Jan 30 rather than where the last update was. Kind of wonder, will that keep happening should it be stopped in the future or get disconnected briefly? Is there a way to reset the above command back to default?

15

Do you have the upload history date set?

16

I do, to that date. Wondered if that might be the cause.

17

I think it has a lot to do with it, but it’s not the cause. As I understand Emoncms (not talking about the Emoncms.org server), it uses Redis to maintain the inputs database in memory. They do this to reduce the amount of updating to the disk, which in the most common case is an SDcard, subject to wear and failure when written to excessively.

Whenever it’s restarted, deliberately or as a result of power failure, the inputs database is not up to date, and so the date of last posting is not there. Upon query, the inputs appear to IoTaWatt to be new inputs. So at this point, IoTaWatt drops back to the “upload history from” date.

In your case, as I understand it, you use a hard disk drive. There are various settings for Redis to obtain various levels of persistence. If you can find where the Emoncms instance sets the persistence, you might try setting that up.

I haven’t done anything with this, and it would be a boost to the IoTaWatt community if you could investigate this. Specifically I’d be inyerested in using RDB persistence with a daily interval. Even an SDcard version of Emoncms would not suffer significant wear with daily snapshots, and it would limit any restart upload to 24 hours. As previously mentioned, I believe Emoncms will simply discard any redundant data. I would consider even 12 or 6 hours if power failures or restarts are common. With a HDD, one hour or less would be fine.

Given that redundant uploads are not a problem, I wouldn’t attempt to use the AOF journals.

If you figure this out, I’d appreciate posting a tutorial about how to set persistence in the Emoncms distribution. I would probably work that into the IoTaWatt/Emoncms docs.

18

I did use an HDD. Had an old 30gb, but sadly it appears it failed which forced this whole mess to begin with. I am back to a new 16gb SD card and will, moving forward, use sd cards and will swap a new one in upon a new EmonSD release. Still, I can look into what you asked about to see if we might find a way to get an easier history reload. Might help people with history gaps in EmonCMS due to power failure or otherwise.