Telegraf exposes an InfluxDB v2 listener which we can then do transforms (tags/bucket routing etc) before finally being pushed to InfluxDB. The /api/v2/write endpoint is identical to that of InfluxDB, it just doesn’t have any query functionality. Would it make sense for the situation where /api/v2/query returns 404 to instead fall back to the “upload history from new measurement set” as described in the docs? influxDB — IoTaWatt 02_03_20 documentation
I don’t think so. The uploaders are very slow and really bog down everything else when they are uploading a lot of history. It can take hours to upload a day’s worth of history depending on the size of each frame.
The only recourse I can see is to skip ahead and upload real-time leaving a hole.
I wouldn’t necessarily consider that more elegant than doing a query to insure a seamless dataset.
What kind of transformation are you interested in doing that cannot be accomplished within influx using a continuous query?
Yep! That’s what I was thinking of. Just continue where we are at now, ignoring what was skipped. That’s what my change above tries to do.
The idea is to handle the routing/transforms/bucket allocations in the Telegraf agent. I don’t have any at the moment, but for flexibility most of the log/metric systems I’ve been using generally tries to consume raw data from the input, and handle the transforms/routing at a central system where we have more compute power.
A nice side effect is also that we could not directly expose Influx to an entire network.
Ah, currently the uploader just reports a 404 and doesn’t upload anything…
Hey @overeasy - would you be open to accepting the linked PR if I were to work through the review, merge, and test process with you? That maintains the existing behaviour for direct Influx access, but also supports Telegraf as the uploader target instead of failing altogether (which is the current behaviour.)
There was no link, but the answer is probably not. If I understand where you were going, it was to skip the query on a 404 response and just start uploading real-time.
There is no other interest in supporting real-time Telegraf uploads, and all of the IoTaWatt uploaders synchronize with their targets to provide seamless datasets. There are other potential opportunities to get a 404 response to the query without Telegraf.
It’s open source, and you appear capable of making whatever special mods you need. I’d recommend that you maintain a branch with your mods.
I setup Telegraf this week. If you use the influxdb_listener it works without any IoTaWatt changes. If you run influxdb on the same host you need to change the port number from the default like I did.
[[inputs.influxdb_listener]]
## Address and port to host HTTP listener on
service_address = ":8186"