I am attempting to redirect my IOTAWATT from a local instance of INFLUXDB to the cloud version of INFLUXDB. I cant figure out how to set the URL to include the AWS url and my tokens. Has anyone done this? Is this possible to do? Would appreciate any help that you can provide…
Thanks,
Mel
The cloud is a big place. Can you be more specific?
Ok, got it.
I had looked at this awhile ago and it seemed inappropriate for the relatively low data volume of IoTaWatt. At the time I believe the lowest plan was the Hobby plan at $50/month. Pretty expensive hobby. If I could get $50 a month from users I’d give them the IoTaWatt. Kinda like printer ink.
I see that they now have a free plan that looks as if it would work ok, except that it only retains the data for 30 days. I suppose that’s fine for looking at recent activity and doing any longer term analysis directly with the IoTaWatt data.
There is also a pay-as-you-go plan, and I haven’t tried to estimate the cost for that. Have you looked into that?
In any event, it looks as if this service requires adding some query fields to the URL and also some method of specifying a token. I’ll have to look into that. IoTaWatt already supports the username/password authorization, so the token is something new.
It looks from the CURL example that it accepts HTTP posts. If it turns out they require HTTPS, that’s a show stopper unless you use a proxy like Nginx to translate.
I’ll take a closer look when I get some time.
I worked on this for a few hours, and have to say it is unlikely I’ll support this anytime soon.
This cloud offering is a new version of influx designated 2.0. The documentation is confusing in that it works with no changes to the influx 1.x (what IoTaWatt supports), yet the API interface in the cloud offering requires a different authorization protocol and seems to do away with the notion of a “database” in favor of identifiers for “org”[anization] and “bucket”.
I made those changes, only to discover that the query system appears to be quite different and the documentation states that it returns CSV tables rather than JSON as the influx 1.x does.
Even if these things can be overcome, the cloud service requires HTTPS. The ESP8266 does not have adequate memory or appropriate tools to do that, although a proxy server can be used to make the jump.
Sorry.
@overeasy Would like to make a quick recommendation if I may … I just spent a great deal of time setting up InfluxDB 2.0 including purchasing an SSL, etc. and when I went to push the data to it, found out it’s not working. Did some digging in the forum and found this thread. Can you update your documentation to state which version of InfluxDB and Emoncms that you support? Thanks!
More than a year ago, when influx released the Alpha of 2.0, they stated:
We wanted to get this early release out for users in the community to play with to give us feedback. There are some core features missing that need to get done before we’re ready to move into the beta phase. We still have to add compatibility (being able to query InfluxDB 2.0 via InfluxQL and the 1.x API), and being able write to InfluxDB 2.0 as though it were a 1.x server. We also need to add features for backup & restore, bulk data import and export, and data deletes.
The Beta release seems to kick the can down the road. With respect to queries:
InfluxQL is the way users query data from InfluxDB 1.x today, and there are millions of queries running in production today. Manually converting those queries to Flux would be a daunting task, something that we are working to automate. The first step is exposing a way for users to convert their existing InfluxQL queries into Flux that can run on the latest InfluxDB. To that end, we have exposed an API for doing that called Influx Transpile. It’s just a starting point and we are looking for users to post their InfluxQL queries to it and open issues where it doesn’t make sense. We’ll be providing a compatibility mode which allows your existing InfluxQL queries to dynamically run as well.
There doesn’t appear to be a smooth transition for posting data using the 1.0 HTTP interface either.
2.0 isn’t really viable yet. They will not commit to non-breaking changes. But the big question is “What does 2.0 do for an IoTaWatt user?”. I can’t see any compelling reason to convert. The 2.0 seems to have been fast tracked to the cloud, presumably to satisfy the needs of large paying customers. I don’t fault them for this. Believe me, I know that it’s difficult to maintain an open project as a business.
There are plenty more time-series database offerings out there. Rather than chase an evolving version of influxDB for questionable return, I think I’d rather expand support into other offerings that are more mature, most of which can easily handle the needs of IoTaWatt.
I’ll put some language in the documentation to caution against influxDB 2.0.
Bob - would you have any interest in integrating with TimescaleDB? My current plan was to write a “shim” service to take HTTP messages for one of the existing (Influx, Emoncms …) and translate them to something that Timescale can use. Let me know, should be working on it in a week or so.
I’ll take a look. Export services aren’t that hard to add, as long as the server (database) can provide context for restart.
New services would be modeled after PVoutput which is implemented as a class.
Great - see the other thread on Prometheus. For direct to Timescale, I would need to find a way to integrate with Posgresql directly.
@overeasy - I’ve read through PVoutput and have a general sense of what it does. I was considering making a cloud adaptor as a Java service that could be plugged into various service contexts (Lambda, Azure Functions, or just a plain old service running on a compute node).
I wanted to see if you had any guidance on which of the output systems you thought would be the easiest to emulate or if you’d be interested in creating a minimal set of calls that a service would need to support to integrate IoTaWatt. I read the PVOutput service specification and there’s lots there that one would have to parse through and decide what to ignore.
Structurally, my intent is to develop all new services as classes. That’s the reason I reference PVoutput. Functionally, my sense is that what you are after is more like what the influxDB and Emoncms services do.
PVoutput runs on local time. It is the only place, except for timestamping the message log,and query where local time is used. The datalogs all use UTC. Both influx and emoncms use UTC timestamps on the data that is uploaded.
The basic way that an upload server works is:
On startup or restart query the server to determine the timestamp of the last upload.
Upload data from the datalog beginning with the next interval after the last upload.
I do plan to rewrite the influx and emoncms as classes, but that may take awhile. Probably will be a retrofit from the ESP32 port. These two services are pretty much alike, but the devil is in the details. There has to be javascript in the configuration app to configure them, and documentation to say how it works with the server.
At the end of the month, it’s a project.
Got it. That sounds simple. If you have any thoughts on which service would be easier to emulate between influx and emoncms, the advice would be appreciated. Basically I’m going to create something like the following:
IoTaWatt --> MyCustom(Influx|Emoncms)Service -->DB(Timescale|MemSQL) <–Grafana Dashboard.
If you have any request / response samples for the startup / restart and upload actions of whichever one you think is easier, it would be pretty helpful. If not, I’ll wait to observe my own IoTaWatt. Once done, I’d be happy to abstract and donate the class if you think any would make use of it.
I’m about to send you pictures of my circuit breaker box and dryer plugs in another thread to that I can finalize my order. I’m trying to get it down to a single IoTaWatt while still getting a good view of energy use (there are 40 total circuits / 22 distinct channels).
Do you have any example payloads for the transactions to Emoncms? I should have my hardware ordered in the coming week but would love to get a peek.
@overeasy - So I started with Emoncms and emulating it went fairly well. Once I got into it I found I could not see where the names or units of the inputs are passed to the server. Am I reading that right?
I think so. The Emoncms input protocols are described here. IoTaWatt uses input/bulk.
Emoncms has no provision for capturing or storing units. InfluxDB does. So you can specify the units you want to send to Emoncms, but they arrive as unitless numbers.
Emoncms was improved to accept named inputs, in addition to ordinal values, but that protocol is very verbose and does not provide for efficiently sending multiple frames (time stamps) in one transaction.
Mel,
@mgehrs
Where you ever able to get this working? I want to store the data on the Influxdb 2.0 cloud and pull it using grafana.
Thanks!
While the initial 2.0 announcement stated the intent to provide 1.x compatibility, that seems to have been completely abandoned. So now 2.0 is actually a completely new server support.
I have not looked at this lately, but as I recall the influxcloud 2.0 required all APIs to be sent as HTTPS, which IoTaWatt simply cannot do. If you point me to some documentation that HTTP to the cloud is supported, I’ll take another look
The authentication in 2.0 appears to use tokens which is reasonably secure for most IoTaWatt users, so if the data posting and query (F.lux) API can be sent via HTTP it would be feasible.
Thanks for the update. If thats the case I plan to use python http client to pull form a local influxdb server and push it to the cloud server.
Appreciate it.
Head of products from InfluxData here.
I’d like to provide some clarity to a number of points made in this thread.
Following the pattern above:
IoTaWatt → influxDB_listener input /[ Telegraf /] influxDBv2_output → InfluxDB 2.0 (OSS or Cloud)
You can then use the native UI to build beautiful dashboards.
If you already have dashboards built, in say Grafana, you can use the influxDBv1_output and this will create the appropriate mappings and land the data precisely as it was in InfluxDB 1.x. You can then continue to leverage your Grafana dashboards by creating an InfluxDB data source connection leveraging the InfluxQL language in Grafana 7.1.x or above. The instructions are here.
You can send the HTTP-based metrics to the influxDB_listener input plugin configured within Telegraf. This essentially makes Telegraf look like an InfluxDB 1.x instance. Telegraf can then be configured with the appropriate security credentials and will translate the inbound payload to what InfluxDB 2.0 expects (via the InfluxDB 2.0 output plugin) using HTTPS (or use InfluxDB v1.x output plugin and the appropriate compatibility APIs will be leveraged).
Further, if you have an existing setup with InfluxDB and you want to dual write — simply to explore the capabilities of InfluxDB 2.0, you can do this with Telegraf as well. Telegraf is capable of having multiple output plugins configured. Meaning you can configure 1 output plugin for your existing 1.x instance while configuring a 2nd output plugin for InfluxDB 2.0 (again, either Cloud or OSS).
We have a very large community of open source projects and applications that have leveraged the capabilities of InfluxDB and we appreciate the support, interest, contributions, and effort that is has taken to grow such a community. We did not want to aggressively deliver a major upgrade to the OSS community without thought, consideration and feedback from our developers who have clearly put their trust into our technology. So, we have been very deliberate about not pushing the OSS edition towards GA haphazardly. All of the new capabilities have been tested in the Cloud edition first and we continue to deliver improvements, feature additions and more approximately 25-30x/week to the Cloud edition. We’ve now been effectively running the Cloud edition at scale for a year and are confident that we are ready to move the OSS edition forward to a GA state starting with Linux packages and then expanding towards Windows and other platforms.
We’ve reached this point by listening to feedback and incorporating these kinds of compatibility modes and techniques in an effort to minimize code changes as much as possible while also trying to move such a large community forward.
Happy to field additional questions or if you would like to dig deeper, we welcome your inputs to our community site as well → community . influxdata . com
Thank you…