InfluxDB Best practice & Heap

Ive been battling with various settings in the config file for Influx and thought I had one working well only to find I couldn’t group 2 values ( post process ) like in grafana with my current approach.

I had to edit the config file directly ( like you - @Linwood ) mentioned in your post re. Breet Beeson’s config. Now i need to essentially swop the “measurement & tag " with " fields” ( the opposite to the attached image ). Can we simply use $name in the field key at the end of the config file?? ( i did try it but by that stage i encountered a “configuration parse failed: Unexpected string in JSON at position 2619” after 52 revisions i think i broke my firmware… re-compiled but scared to try again ;-(

I have also been battling with degraded heap on a full stack export to influx ( i.e 12 channels all sending W,VA,A,PF,Wh even though I had resolution at 1800seconds with 1 bulk send. ( I am hoping I can find away to retain all the parameters I need and reduce heap strain - perhaps some thoughts on different resolution in the upload??

@overeasy, @Linwood - any suggestions , help?

As to names and editing, I ended up not needing it as I decided I really did not need much beyond watts. But… I think that feature has been added, see changes here:

I’m a bit confused by the other aspect. The field names come from what is in field-key, so if you want the field name to be the name you put into the output, you can use $name in the field-key. That ends up being the thing you type in during the setup. I think @overeasy had something similar to that, putting all CT’s in as fields so you get one row with lots of fields.

It looks as if you are using the default
and specified

To swap those specify

Note that the left column is labelled Measurements and Tags and I’m assuming those ch2, ch3 etc are the measurements. If they are tags, and you want the tag to be units (with the measurement presumably “value”) then something like this?

Don’t recommend editing the config file. Any error in json syntax will cause the whole config to be rejected. Whenever I edit one, however trivial the change seems, I always paste it into a Json lint checker to make sure I didn’t make a subtle Json syntax error.

If you do get a configuration parse failure on startup, the web server and file manager should still be working, so you can edit the config file and repair, or download the last save valid config from here:
There is no way to move it directly from the SPIFFS file system to the SD but you can downbload it and then upload it to the SD root.

Resolution only has a significant impact on history upload. But sending 60 measurements at say 40 bytes per line would result in several k per frame, which is a lot of data. Switching to a schema that results in only several measurement names, like measurement = $units, will significantly decrease that in post 02_05_06 releases because it melds like measurements into a single line.

Don’t know what release you are on, but 02_05_07 which is in ALPHA and BETA should resove heap issues and also allows specification of multiple influx outputs with the same name.

Got That now. Great Tx.

Could you explain this in a bit more detail - sounds great?

If your measurements end up with the same name as in:

Watts,sometag='tagvalue' ch1=272.8 1586721538
Watts,sometag='tagvalue' ch2=109.3 1586721538
Watts,sometag='tagvalue' ch3=2704.0 1586721538
Watts,sometag='tagvalue' ch4=0.0 1586721538
Watts,sometag='tagvalue' ch5=0.0 1586721538

IoTaWatt can meld those into one measurement:

Watts,sometag='tagvalue' ch1=272.8,ch2=109.3,ch3=2704.0,ch4=0.0,ch5=0.0 1586721538

It does the same thing but much more efficient, especially if you have 14 measurements for each unit.

Excellent - exactly what i want. Is that only in 02_05_06 - im still using 02_05_03

It’s in 02_05_07 which is currently ALPA and BETA. Should go MINOR at the end of the month. Also understand that in order for it to meld, any tags must have the same value as well, so using $units or $name in a tag value will inhibit the meld.

do you perhaps have a screenshot of the new GUI for the influx config ? im curious to see if i can still add a “key & value” tagset before the data ( i do this to keep the structure nested within a dropdown )… Also will the “meld” or concatenated string be reduced to individual fields when in Influx - i.e as per your example , would we have a “measurement & tag” called ‘watts’ with field entries “ch1”, “ch2”, “ch3” etc. ?

There is no change to the influx configuration app. The melding happens in the IoTaWatt when the data is sent.

It will not meld if there are keys containing the variables $name or $units. If you look at the way it melds into one line, there is only one set of keys that are common to all fields in the measurement. That’s just how influx works.

Not sure if I understand the question. If you do a “show series” for your database, you will see that there is a series created for each unique measurement,tag combination. That happens under the hood in influx. The examples above of verbose and melded measurements all populate the series Watts,sometag='tagvalue'
with the various field=value pairs.

If we Melded “Watts” ( like in the image ) would i get the “M1, M2, M3, M4” unmelded on the other side ? i.e could i then query say just M1 in Influx…

@overeasy - i went over to 02-05-07 . I am posting 54 variables per minute , heap is as cool as a cucumber at 23k and data consumption over 4hours is just over 1MB ( thats 12960 entries ). We are getting to serious grade kit here !! Marvelous results … Be incredibly proud