Resend historical data and tags in Influxdb

Support
1

I’ve tried to look at the code (without actually trying to run it), and also read this:

https://community.iotawatt.com/t/influx-resend-all-historical-data/1002

I am using the same measurement name for all data (so I can do math across it inside InfluxDB), with different tags for the ct (really output) name ($name).

Can you confirm my understanding (or correct it): If I add a new InfluxDB output (under the “measurements” section), because the measurement and field name (watts) remain the same, it does not try to send any old data. I.e. when it queries for the last data, it does not query distinguished by tag.

So if I add a new output (which for me becomes a new tag value) it will start sending, but not send any retroactive data. Right?

So this is where I am getting concerned. Because I am using the same measurement name for both Iotawatt’s, might I get a race condition where “left” believes it has already sent because its query to the database finds data already present? Or not even a race condition, if one Iotawatt is offline for a bit for some reason (but collecting data), when it comes back it will query and not see the gap, if the other Iotawatt has been sending.

To make this work right I either have to use a distinct Measurement (specifically the thing under “upload history from”), or a distinct field-key?

If I use distinct measurement, Influxdb simply will not do math between measurements, so for example I cannot add up info between the two Iotawatts.

So the only way I can retain safety for catch-up transmissions is to use a different field key per Iotawatt, if I want to retain the ability to do arithmetic?

Or am I missing something?

PS. I am somewhat getting tangled up in the terminology, the section labeled “measurements” and the single entry textbox labeled “measurement” are two different things. I think you are using the latter in the “last” query, and the former become $name wherever you use that.

2

It does distinguish by tag - all of the tags.

The algorithm looks for the last of all of the measurements, “where” all of the tags are the same as your tag-set. It then begins uploading all of the measurements from the next interval after that.

Not as long as there is a unique tag specified in each unit.

I understand, but I think the algorithm works as long as there is a unique tag-value in each IoTaWatt.

I had not done much with influx, but having just ported the service to the ESP32 (10x faster upload), I started fooling around with it in light of this issue.

I’m just getting into it, but I’m leaning toward building a database with the units as the measurement name and the output name as the field-key. If I then add code to aggregate all of the outputs with the same measurement name (units), it would be more economical in the upload if say I were only uploading watts and volts. I think the grafana query would allow me to do math on all of the outputs with the same units as it would be “select from Watts”. In your case, leaving off the key that is unique between units would meld all of the Watts from your two devices.

I’m more interested in coming up with a better way to aggregate a lot of IoTaWatt, but it’s the same problem I think.

3

Sorry to persist but not sure what “all the tags” vs “unique tag” means here.

Let’s say I have:

Measurement = Watts  (from $units)
field-key = Watts (as a literal) 
Tag set: ct = $name  (so for example ct=Range, ct=AC, etc.)

So a sample data might include (show series):

Watts, ct=Range
Watts, ct=AC

And values within each have a field name of Watts (same word as, but separate from the measurement name).

So let’s say I added one for ct=Oven that was not there before to the list of measurements to output. I have a new value for the tag ct, but not a new tag (i.e. I still have only tag “ct”). When you say “unique tag” I take it you mean that literally - tag, not tag value?

When I did this, it did not seem to send any retroactive data. Which I think is consistent with what you said? Or is it?

If so, then two IotaWatt devices set up like this, with no overlap of $name’s, may have an issue also?

What I’m using right now to avoid this (recognizing I have not actually produced this hypothetical problem in practice):

Measurement: $units
field-key: $device
tag set: ct=$name

I think I get distinction because the field-key is different, so the startup check will find data specific to each device? The downside is the data overall is mixed between two fields, named for the devices. It’s possible (with some kludging) to add two fields together (or other math), but not two measurements. But it’s ugly and counter intuitive. But the best I have so far.

Example query I’m looking for is “add up all the items with ct<>main, and subtract from the item with ct=main, as other”. Surprisingly difficult to do cleanly, apparently. At least so far. Maybe in the morning when I’m more fresh.

4

When it starts up, the service does a query on each of the measurements, fully qualified by measurement name, all of the tags (tag-key = tag-value) and the field name. So the query for:

where say measurement is Watts, field key is iota1, and ct=kitchen, would be

SELECT LAST(iota1) FROM Watts WHERE ct = 'kitchen'

It will do this for every measurement, and use the most recent time as the starting point to begin uploading all of the measurements. In other words, it does not selectively upload each measurement based on it’s last value. It treats all of the measurements as a group.

So the only way to add a new measurement and upload it’s history from before the latest measurement is to specify only that (and other new measurements) and start it up. When it completes, drop the new measurements where time is greater than where the old measurements left off, and then restart with both the old and new measurements specified.

It’s clunky, but not intended as a retrospective capability except in the initial load.

5

It will do this for every measurement, and use the most recent time as the starting point to begin uploading all of the measurements. In other words, it does not selectively upload each measurement based on it’s last value. It treats all of the measurements as a group.

Got it. That’s what I observed.

So continuing this, if I have field-key as Watts not $device (meaning no part of the setup has $device, they are identical except for unique ct $names), then device A will never see any of the rows for device B. This only becomes a problem if someone puts ct=X on both, duplicating. Then device A might see B’s update and miss its own.

That helps, then I can change field-key back to $units so they are the same from both devices. That’s my goal – to have the two device’s data look like it is all one big device – except of course so each device itself does not get confused.

And if I’ve understood you I can. Thanks.

6

I’m still confused. Here is my current setup:

iotar-setup
iotar-setup608×1153 73.5 KB

Results
Results807×320 33.8 KB

Notice I have an entry, separately named, for both watts and volts, but in the data despite coming across named as expected it contains watts.

Is it not possible to send mixed metrics?

PS. This example is from one iotawatt, not trying to mix at the moment while I debug.

7

The problem is that if you want to send a voltage measurement, you need to specify a function that uses VTs not CTs.

Internally, VTs and CTs use the same data structure to log their values. IoTaWatt doesn’t enforce strong typing, so if you specify CTs as the source of the data, you will get what comes out of CTs - Watts.

I know you will get this, but FBO others who might be reading this for guidance, you can see which data sources can be used with a particular unit by selecting that unit in Graph+. The compatible data sources will be listed.

8

Ah, thanks. Didn’t even occur to me, I just assumed magic would happen and pass through (to the appropriate reference voltage). Makes total sense (well, with a minor thought that type checking is always a good thing). Sigh, so many places to stumble, and I’m finding them all. Is that a marketable talent? :smiley:

9

I’m living proof of that.