Is there a way to force Iotawatt to resend all the historical data available to Influxdb ?
The reason of my question, is because first I had a problem with the time received in Influxdb that doesn’t correspond to the timestamp in Iotawatt (strange pb). To solve that I dropped the database in Inlfux, recreate the Webserver connection in Iotawatt, and it works much more better. But I’ve lost my history in Influx…
I’ve tried several stuff:
modifying “upload history from:” to an older date, or the start date of my historical data. Nothing change in Influx.
drop measurements in Influx. Iotawatt recreate them and start sending data from last restart (I think), but not the all history.
delete / recreate the measurement definitions in Iotawatt. Same results.
drop my Influx database, create a new one. Same results.
drop / recreate influx database + delete / recreate the influxdb web service in Iotawatt. Same results.
Just to clarify, that’s not possible as the measurements are explicitly timestamped with the timestamp from the datalog. The time received in influx should be irrelevant.
Dropping the database and recreating should give you a clean slate. When you restart the IoTaWatt, did you have the upload history from set prior to 17/03/2019?
When the influx service starts, it queries each of the measurements to see the time of the last entry. It then starts uploading from the most recent time or the “upload history from” date - whichever is greater. If upload history from is blank, it is assumed to be now.
Understand that if you drop the database or individual measurements while the IoTaWatt is actively uploading, it will be adding newer entries even as the older ones are dropped. You need to stop the updating and remove all of the old measurements.
Try this:
Click STOP in the influx tab of the status display.
Wait for the status to change to stopped.
In the IoTaWatt influx configuration:
Set “upload history from” to 03/17/2019.
Add a tag of “id” with value “iotawatt” to the tag set.
Save
Click START in the influx tab of the status display.
You should see the Last Update change to 03/17/2019 (or 17/03/2019 the way you do it).
The first tag in the tag set is a way to group and identify all of the measurements from the IoTaWatt. (You may have other measurements from other sources). With the id tag, it becomes easier administrate that set of measurements with the simple WHERE id = ‘iotawatt’ clause.
Once you verify that the update is progressing from the supplied date, you can watch the progress. It may take a few hours to accomplish upload of the whole 2.5 months.
Great! You’ve probably figured out that this is a new measurement series because of the tag. The old measurements are still there and can be cleaned out (or not), but your queries, if say using grafana, need to be modified to include the id tag to identify these new measurement series.
Hi @overeasy, many thanks. Using “tag” is a good idea !
If I’ve well understood; when I add a new “measurements” or do some updates in “Edit influxDB Service”, historical data are not back-loaded automatically.
The reason is Iotawatt check the max date in Influxdb (for the defined “tag” in “Edit influxDB Service”) and start sending measurements from this max date for all defined “measurements”.
Thus if I add new “measurements” in “Edit influxDB Service”, I’ve to change the “tag” definition (eg change the “id” as you propose) to force the reload from the “upload history” date defined for all my measurements.
That would probably work (the devil is in the details) but it wasn’t really designed for that purpose. Once you get caught up on measurements, the expectation is that new measurements would not be based on historical data, but rather actual new inputs to IoTaWatt that would have no history. So they would just be added to the measurement set and start up from now.
If you were to discover in hindsight that there is something that the IoTaWatt was measuring and you haven’t been sending to influx, then yes, you would need to upload everything again to achieve that.
But I would not recommend changing to a new id tag. That would have a ripple effect in your queries. I would recommend that you stop the influx service, drop the entries from all series with the present id tag, set the upload history from to the appropriate date, and restart the influx service.
