Upload data from history log?

g4m3r7ag · May 20, 2021, 5:35pm

I have an IotaWatt at a cabin recording data that is uploaded to an emonCMS that is not hosted at said cabin. There was recently a multi-day internet outage at the cabin and during that time the IotaWatt rebooted. When it did, it did not get a response from emonCMS and started attempting to upload the data again from a year ago

SD initialized.
5/18/21 14:27:27z Real Time Clock is running. Unix time 1621348047 
5/18/21 14:27:27z Power failure detected.
5/18/21 14:27:27z Reset reason: Power On
5/18/21 14:27:27z ESP8266 ChipID: 2527501
5/18/21 14:27:27z IoTaWatt 5.0, Firmware version 02_06_02
5/18/21 14:27:27z SPIFFS mounted.
5/18/21 10:27:27 Local time zone: -5:00
5/18/21 10:27:27 Using Daylight Saving Time (BST) when in effect.
5/18/21 10:27:27 device name: IotaWatt
5/18/21 10:27:30 Connecting with WiFiManager.
5/18/21 10:27:33 HTTP server started
5/18/21 10:27:33 emoncms: Starting, interval:5, url:http://192.168.1.30:7080
5/18/21 10:27:33 WiFi connected. SSID=Brandon, IP=192.168.128.105, channel=6, RSSI -67db
5/18/21 10:27:33 MDNS responder started for hostname IotaWatt
5/18/21 10:27:33 LLMNR responder started for hostname IotaWatt
5/18/21 10:27:33 timeSync: service started.
5/18/21 10:27:37 statService: started.
5/18/21 10:27:37 Updater: service started. Auto-update class is MINOR
5/18/21 10:27:37 dataLog: service started.
5/18/21 10:27:39 dataLog: Last log entry 05/18/21 10:25:20
5/18/21 10:27:39 historyLog: service started.
5/18/21 10:27:39 historyLog: Last log entry 05/18/21 10:25:00
5/18/21 10:27:59 emoncms: Query failed -4
5/18/21 10:27:59 emoncms: Start posting at 05/18/20 09:06:50
5/18/21 10:28:00 Updater: Invalid response from server. HTTPcode: -4
5/18/21 10:48:39 WiFi disconnected.
5/18/21 10:50:38 WiFi connected. SSID=Brandon, IP=192.168.128.105, channel=6, RSSI -66db
5/18/21 11:10:20 WiFi disconnected.
5/18/21 11:20:34 WiFi connected. SSID=Brandon, IP=192.168.128.105, channel=6, RSSI -68db
5/19/21 10:27:39 timeSync: No time update in last 24 hours.
5/19/21 17:30:06 Updater: Auto-update is current for class MINOR.

When the internet connection restored it started uploading all of that data to emonCMS again resulting in duplicate data in the emonCMS input/feed. When I realized this I stopped emonCMS on the IotaWatt, deleted my inputs and feeds from emonCMS, then recreated the inputs on emonCMS via API with a the initial timestamp prior to the Time that the IotaWatt has data for in it’s history log.

curl --data "node=MudRidge&data={1:1,2:2,3:3}&time=1567296000&apikey=redacted" "http://192.168.1.30:7080/input/post"

I then deleted the emonCMS configuration in the IotaWatt and rebooted it. Rebuilt the feeds in emonCMS. Then rebuilt the emonCMS upload in the IotaWatt with a upload from date of 09/01/2019. When I saved it and the process started, it started uploading from the oldest date in the emonCMS Current Log (a year ago), and not from the upload from date. Is it not possible to upload the data from the history log and only from the current log?

5/20/21 09:07:14 Restart command received.

** Restart **

SD initialized.
5/20/21 13:07:16z Real Time Clock is running. Unix time 1621516036 
5/20/21 13:07:16z Reset reason: Software/System restart
5/20/21 13:07:16z Trace:  1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 10:2, 10:3
5/20/21 13:07:16z ESP8266 ChipID: 2527501
5/20/21 13:07:16z IoTaWatt 5.0, Firmware version 02_06_02
5/20/21 13:07:16z SPIFFS mounted.
5/20/21 09:07:16 Local time zone: -5:00
5/20/21 09:07:16 Using Daylight Saving Time (BST) when in effect.
5/20/21 09:07:16 device name: IotaWatt
5/20/21 09:07:16 HTTP server started
5/20/21 09:07:16 timeSync: service started.
5/20/21 09:07:16 statService: started.
5/20/21 09:07:16 dataLog: service started.
5/20/21 09:07:19 dataLog: Last log entry 05/20/21 09:07:10
5/20/21 09:07:21 historyLog: service started.
5/20/21 09:07:21 historyLog: Last log entry 05/20/21 09:07:00
5/20/21 09:07:23 WiFi connected. SSID=Brandon, IP=192.168.128.105, channel=6, RSSI -68db
5/20/21 09:07:23 MDNS responder started for hostname IotaWatt
5/20/21 09:07:23 LLMNR responder started for hostname IotaWatt
5/20/21 09:07:23 Updater: service started. Auto-update class is MINOR
5/20/21 09:07:24 Updater: Auto-update is current for class MINOR.
5/20/21 09:11:11 emoncms: Starting, interval:5, url:http://192.168.1.30:7080
5/20/21 09:11:12 emoncms: Start posting at 05/20/20 07:50:05

This is what IotaWatt shows for history/current log

|Current Log:|05/20/2020 12:13:35 - 05/20/2021 13:34:45|1.504 GB|
|History Log:|09/01/2019 12:30:00 - 05/20/2021 13:34:00|220.4 MB|

I knew it would take time to recomplete the entire upload, but I figured it was worth it to have the entire history in emonCMS again but I don’t seem to be able to get it to start the upload from 9/1/2019 only from 5/20/2020?

overeasy · May 20, 2021, 5:48pm

Right, your Emoncms is 5 second intervals. The history log is 1 minute intervals. So the data is just not there. The uploaders will only operate out of the Current log.

It was not necessary to delete your Emoncms history. You could have simply stopped the uploaded, set the upload date to the day of the interruption, and restarted the upload. Emoncms discards any redundant data uploads.

g4m3r7ag · May 20, 2021, 5:54pm

emonCMS was not discarding the duplicate data in this instance. It was adding everything to what was already there a second time. If I change the emonCMS interval to 60 seconds will it upload out of the history log or are you saying no matter what the uploader will only upload from the current log?

overeasy · May 20, 2021, 7:08pm

I’d like to see the input process that was doing that.

That’s how it works. The current log has about one year.

g4m3r7ag · May 20, 2021, 7:34pm

The input process in emonCMS? Here is a screenshot. So my kWh feed prior to the outage was at 19xxx which had the meter feed at 64xxx. After the internet restored and it began re-uploading the data from May of 2020 my kWh feed was up over 21,000 and meter was over 66,000 before I realized what was going on. If I look at the data graph on IotaWatt the kWh is only up to 19801 so it had to be duplicating the data.

edit: After the upload of the last years worth of data is finished I’m going to try manually posting an input to 9/1/2019 on the kWh feed in emonCMS with whatever the difference is to get the Meter feed back to where it should be. Actually might be easier to just add the difference to the + variable.

edit 2: The initial data being logged to the Power feed from the IotaWatt is Watts from leg1+leg2 on the panel.