I found lovely documentation for various cloud places to which IoTaWatt can upload data. I read that I can download and run my own copy of some of these servers, but installing yet another package is more bloat than I want to do right now. I suppose if I dug deep enough, I could find out what different protocols were used to write to those places and write my own little server program to run them so that I could collect the data myself without installing a whole package. Is there a simpler way to get data out of the IoTaWatt?
The Hubitat hub has a Maker API application that has a simple URL to POST device events to. Just enter a URL and the device will POST a JSON record for each event to that URL. Logging those events to a file is a two-line CGI shell script.
They also have a websocket available on the hub (undocumented and unsupported, alas) where one can connect and listen for events via the websocket protocol, and I found a lovely little Perl script that does that – no web server needed.
Could IoTaWatt implement a websocket interface that produced JSON records? There would be no need for me run a web server or any of Influx, Emoncms, or PVoutput.
The IoTaWatt has a database and web server. You can plot your data with Graph+ or you can query the data with the query API.
You might want to start with what you are trying to know (about your energy consumption). All by itself Iotawatt does a decent job of generating graphs that can provide one with more insights than most people would want or know what to do with.
I have had an Iotawatt, since around the time they first came out. I actually still use the built in website to get most of my information. I have used emoncms, InfluxDB, Grafana, and a few other services both in the cloud and locally. Each has its advantages and its challenges.
Also, most people would consider it harsh to call well respected tools “bloat”. I am guessing you didn’t really mean it that way, but that is the way it sounded. They might not be what you want to take the time to learn, but most people who do find them quite valuable. They do take an investment in time energy to learn. If you don’t need what they offer, learning them won’t be worthwhile.
I think the graphing features of the IoTaWatt are wonderful. I even wish there were a way to feed other data, such as temperature, into the graphs, so I could combine energy use with temperature on the same screen. The graphing is awesome!
Since I can’t feed my own temperature data into the IoTaWatt graphs, I have to get the energy data out, and I already have my own graphing infrastructure in place (using data from sensors connected via Hubitat), so don’t want to install any of the three systems that IoTaWatt talks to.
Yes, I will look into the query API. I was hoping to avoid polling.
Since all I want to do is have the data pushed to me where I can save it in a file, installing any of those Data Uploader programs and their multiple dependent packages that I will never use really is “bloat” for my purposes. All I want is the data; I don’t need their databases.
IoTaWatt almost has the functionality I need using its Data Uploader facility. It already keeps a queue of what data hasn’t been uploaded when the other site is down; polling does not (unless I write the recovery logic into my polling program).
My problem is, the three services IoTaWatt knows how to upload to aren’t familiar to me and it will be work to figure out some specific protocol so that I can write a little script to mimic the service and log the records written.
The IoTaWatt could have the exact functionality I need if it had a simple Data Uploader service that wasn’t tied to a specific protocol.
The Hubitat hub Maker API URL made it easy to upload data. Given a URL, it would simply POST the JSON records to that URL and my two-line CGI script at the other end would append them to a file. Easy!
I wish that I could specify just the interval and the inputs to be sent and IoTaWatt would POST the data to the URL I supplied, just as Maker API can do. Absent that, I’ll eventually get around to figuring out and decoding one of the other three protocols.
(And if IoTaWatt had a websocket interface that I could listen on to get the same interval data, it would be much more efficient than incurring the overhead of thousands and thousands of separate HTTP Data Uploader connections, though perhaps the ability of IoTaWatt to queue data when the remote site is down would be harder to do.)
Remember that Iotawatt is running on an esp8266 (not even an esp32). It has limited compute power.
While emoncms, InfluxDB, and PV output might not be familiar to you, they are generally well known standards for energy monitoring (emoncms), general time series database, and solar monitoring.
InfluxDB has a well documented line protocol and is even open source, so it would not be too hard to implement something that could handle the output from Iotawatt. Be aware that Iotawatt does use a query to get the last update, so it knows what time from to resume sending data.
I took a look at Hubitat to see what it was as it seems there is a new entrant in the space every week or so. I am sure you have your reasons for selecting it, but I didn’t see anything I couldn’t live with out. It all depends on what you are trying to achieve. There are lots of tools that can generate graphs, some of them very pretty. The question you have to answer is what data do you need to generate good information to allow you to make real insights that enable you to take some action.
I have been monitoring energy usage for decades and it can be really interesting, actually taking action is much harder, but definitely possible. I tried a few home automation tools, but each of them have their own learning curve and limitations. Some of them are very much into the walled garden approach, which makes it complicated to have things from somewhere else.
The good news is Iotawatt is open source, so if you really want a generic web uploader, you can create one yourself. I don’t know what Bob would think of including it in the official build, but you can always flash your own firmware.
Since I use MQTT (as do a lot of people) I wanted to have an uploader for it. There was some discussion about it, but the protocol doesn’t really meet the goals of Iotawatt (energy monitoring with no gaps). I would like to have some data available via MQTT. I am not sure exactly what I would do with it, so I haven’t take the time to write an uploader. I figured I really should write the requirements of what I wanted a achieve first to see if there was another way.
Finally, if you are really serious about this stuff, you should look at Grafana. It is not hard to set up. They have a Docker image and it is not resource intensive. InfluxDB also has a Docker image which makes it easy to get started. There have been a lot of changes and improvements over the last decade or so I have been using them. If you like dashboards that show you what is going on at a glance, the combination is great.
I discovered that Bob rejected the idea of a “generic” web uploader back in September 2021 (Latest on ESP32 - #33 by overeasy) so there’s no point in me spending time coding one. (From that, I assume coding a websocket interface would also be rejected.) My only way forward is to figure out the upload protocols that IoTaWatt currently implements.
The better way is to start with stating what your real requirements are. Right now you have a solution in mind and the requirements of that solution (to the real requirements) is not going to be easy.
This reminds me of the (apocryphal) different approaches the USA and USSR took on the topic of space pen. The story is NASA spent $1million developing the space pen. The USSR gave their astronauts pencils. The real story is even more interesting https://www.nasa.gov/directorates/spacetech/spinoff/How_NASA_Astronauts_Write_in_Space
Or, if you want to go down the path you are, get a network sniffer or just look at the code:
What insights are you really trying to get from the data?
Or is your goal just getting pretty graphs in your Hubitat UI?
I also read the post you linked (again, since I didn’t recall Bob ever saying no to a provided uploader). I haven’t seen anyone doing the work to create a new uploader. I have submitted pull requests for the things I cared about. On each of them there was a little discussion and then agreement on the approach/code before Bob accepted them. On one or two Bob extended the idea even more. On one or more Bob didn’t see the point, but agreed there was little harm.
If you really want a generic uploader write it and test it out. It probably won’t be harder than writing the other end of it. This is the beauty of open source. You are not tied to what the creators think is right. If what you create is great I am sure others would appreciate it and it might end up being included in the code.
I’m a PHP guy but Python or even basic scripting might work to use the Query API to curl or wget from IotaWatt and then push as an upload.
For example, using PHP Curl is as simple as this…
$ch = “http://192.168.2.xxx/query?select=[time.local,VRef,I2Black.Watts,I2Red.Watts]&begin=2023-02-01T07:09:00&end=s&limit=360&group=5s&missing=zero&format=csv”;
$handle = fopen($ch, rb);
$contents = stream_get_contents($handle);
fclose($handle);
Where $contents is a 2 dimensional array of rows and values - ready to be transformed for an upload. I used csv but json is also available.
Here’s a quick Powershell wget → file example
Linux would be very similar.
I keep going back to NODEred being a good way to issue queries and produce MQTT or HTTP(S) with any desired format.
Here’s another rejected one that would be very useful those of use that use prometheus daily for almost everything. It’s probably the most popular metrics collection software in existence. This pull request isn’t uploader, just an api endpoint for prometheus to poll so it adds zero overhead for people that aren’t using it. So yeah, unless you plan to maintain your own fork, I wouldn’t bother writing any code for it.
Fair enough, it is Bob’s project so he is free to say no to contributions. I see it from both sides. Bob is right that there are an interminable number of choices and there really aren’t enough resources to support them all. I participate in some other open source projects that have conditional compilation to allow supporting (where support is defined as it probably compiled once before it got checked in and someone might be willing to help you figure out why it is not working for you) an almost infinite number of things. Since Bob has a higher definition of support, I can understand why he would be reluctant to include (which would then mean support) something that provides value only to a few people.
There are times I would like info from Iotawatt to MQTT. I haven’t wanted it enough to actually implement it. If I ever do, the source is there for me to do it. Or, I can do it with NodeRED as Bob suggested.
I have found that taking the (smallish) hit to learn enough InfluxDB and Grafana gives me a lot more flexibility to get what I want.