I think I already know the answer here, but what the heck, I’ll give it a try
I’ve been using InfluxDB 2 for years and it has worked great, but I have moved on after a few issues and irritations with it. I’m replacing it with Victoria Metrics which I’m loving so far. It’s supposed to be a drop in replacement in most situations, and I have so far found it to work well with my other apps. However not yet with IoTaWatt.
I’ve tried both v1 and v2 and so far have not had any luck.
From the server logs when firing it up, and it queries to get the last entry which is not really supported.
I’m curious if anyone else has had a clever way to work around this. I mean, short of modifying the code to support the VM API format, I’m thinking I’m dead in the water…
I’m half thinking of just writing a quick custom proxy in python to transform the results back for IoTaWatt. Then I’m just passing through all the rest. More complicated than I wanted, but I really need to get off InfluxDB so I have to figure something out.
I’m stubborn…I need to solve this. I guess my four options:
Write a custom reverse proxy that will accept the queries/writes and translate them to victoria metrics format
Update the source to interact directly with victoria metrics (the query is really pretty simple)
Write a script that will do the same thing IoTaWatt does in reverse - pull the data off by looking for everything since the last update via the API
Keep InfluxDB as just a temp database, and write a sync script to pull the data and migrate it to victoria metrics on a regular basis
None of these appeal to me
I think the best would be update the code and contribute back, however my cpp is very rusty. Also I recall reading a post about someone having issues with the libraries being used are old (which makes sense) and having troubles compiling. I’m generally not afraid to build firmware for devices, but I’m not a fan of hunting down libraries.
I think for now, the last two are the most realistic.
I’m pretty sure this will not matter, but just an FYI here in case someone comes on this thread in a search. In my research for solutions, I once again visited the possibility of the new version of InfluxDB addressing some of my other issues and I could keep running InfluxDB. It does, but, it has dropped support for Flux, so IoTaWatt would not be supported And I thought I was so close to an easy solution to this problem… In case anyone out there is looking to upgrade their InfluxDB to v3…don’t do it.
I have decided to write a python script pull the data. Basically the opposite of the uploader but same logic. It will query the last values saved in Victoria Metric and then get all data from IoTaWatt in blocks until current. I will share the code when I’m done if anyone is interested.
Just curious, what issues and irritations have you had with InfluxDB v2? I too have been using it for years and it has been fine. v3 is now available in a community/oss edition (it had been coming soon for what seems like years). It is VERY different from v2 and v1. But, it does support v1 query and line protocol so it would probably work with IotaWatt (in v1 mode).
I briefly looked at Victoria Metrics and the marketing stuff makes it sound great, but I haven’t tried it yet. What am I missing?
@frogmore my opinions of InfluxDB are probably getting a little off topic, and I certainly don’t want to bash it, it is fine. I too have run it for many years, just never how I want to. In the particular way I run it, I have decided it’s not what I want. Specifically I prefer to run things in application level clusters and fully resilient, on a k3s cluster. I had many issues with the underlying file system, migrations, not clustered, etc. I ended up using longhorn to support migrations, but even that doesn’t work the way I want it to for Infux (spends forever on migration resetting all the permissions for unknown to me reasons). I probably “could” with enough head banging get it sort of working the way I wanted, but I was already investing too much time. On top of not working the way I want, I am not a fan of the direction of the company. The whole shifting around on their customers and all in / all out on Flux. Yes, the latest version does support clustering, but I’m moving on, it was too late, I got into VictoriaMetrics and loved it so much more for so many reasons, I decided to migrate and not look back. As for IoTaWatt though, yeah, it might work to go all the way back to using v1, that’s not the direction I really want to head in. I wrote a sync script in Python. I will share it here once a make a few changes to it, if folks are interested.
I have also been looking for alternatives to influxdb lately. The writing is on the wall that v2 is now effectively abandoned and some of the issues I’ve been having with it will never get resolved, even if I PRd a fix against the code. The fact that their free (as in freedom) v3 product is limited to querying 72 hours is shortsighted on their part I think. Or perhaps a better way to view it is that it isn’t the right fit for IoT sensor readings’ storage. They’re considering to maybe release a community edition that would address the query limit, but it is pretty much guaranteed that this won’t be under a OSI license and thus is not really a solution for me.
What I am trying out right now, though, is clickhouse. To my eyes VictoriaMetrics’ supporting downsampling and retention rules in enterprise versions only is the same kind of kneecap as influx’s 72 hour query limit, when my goal is to store multiple years of historical data. My experiments with clickhouse so far suggest that it is actually a great alternative, but only time will show how it truly is.
Not opposed to adding another uploader, but there needs to be some vetting. I can’t become familiar with eight different offerings and support test platforms for them. To date, there has been only one effort to modify or build an uploader (PVoutput - and a good job as well), despite the way I had built base classes to support any such effort. Much as folks whine about open source, the reality is that it’s just another way to say free - as in no cost. Even the PVoutput users balk at giving that excellent effort ten or fifteen bucks a year for their trouble.
The efforts here by @danielgoepp are another way to go about it, and for some that may be the answer. I have not looked at it directly as my experience in offering solutions involving intermediary processing have not really worked out.
With influx, I just use the AWS cloud version. It’s not really very expensive in the big picture. Same with Emoncms. The two alternatives mentioned above are compared only by the restrictions of the OSI offering. To me, that is probably a secondary consideration after the function, efficiency and stability of the product. I don’t mind paying a retail consumer fee for a robust hosted version.
Agreed. At this point in time it might not even make sense to keep adding uploaders for additional databases (of which there are dozens.) Everybody’s requirements are going to be ever so slightly different, the most appropriate tool will vary and the threads asking for new uploaders keep coming endlessly.
I’d be curious to hear more about your experiences and what didn’t work out.
The industry has been opting to run a separate tool of such sort to decouple the choice of the database and the sources of data (as well as to enable scale, but that’s off topic for home use anyway.) Some examples of such a tools are Telegraf, Vector or otlp-collector. These are quite similar in idea to the IotaWatt scripts linked above, except, maybe, a little more widely applicable and somewhat limiting in terms of backfill. So at the first blush taking such an approach would seem like a very reasonable way forward.
A concern that comes to mind is that all such tools would all be implemented in terms of the query API, blocking sampling of the power for the duration of the request. That said, expanding iotawatt’s API as needed (maybe an endpoint to stream samples as they get sampled?) could turn out to be the maintainable solution in the longer term.
In particular, the ESP8266 based IoTaWatt doesn’t have the resources to support TLS communications. This has inhibited the use of cloud based influx services and other use of the IoTaWatt’s own datalog because of the lack of a secure web server.
A solution is to use NGINX on a RPi or NAS to make the jump to HTTPS. The IoTaWatt can be configured to send uploader transactions to be encrypted. I’ve seen very little evidence that anybody uses it.
Not true. IoTaWatt doesn’t block during HTTP transactions. Those transactions are done asynchronously.
Two things:
The existing query API can provide that data in a pull protocol. Some advantages of using a pull protocol are that the puller specifies what data to needed and the can request backfill data after an outage.
A push protocol, as in the current uploaders, must be configured to specify the upload content and also needs the query portion to backfill after an outage.
That said, the ESP32 version I’ve been working on could support an MQTT stream, which a lot of the DBs can digest.
I stand corrected. I don’t work on that code very much anymore. The ESP32 version I’ve been working on does everything asynchonously under freeRTOS. The ESP8266 uploaders run asynchronously but the web-server is not async. That said, relatively short queries do not significantly affect sampling.