REST API, how to specify ISO date, bug?

Hi.

Finally received the CT clamps, and trying to configure that to fit into my existing power monitoring tool (https://mediaserver.avenard.org/power/power)

So I can’t figure out how to specify the query format to retrieve a particular minute of data.

I can use:
http://iotawatt.local/query?select=[time.utc,Main]&begin=2020-03-26&end=2020-03-27&format=csv&group=m

And retrieve a day worth

I can set a particular hour fine:
http://iotawatt.local/query?select=[time.utc,Main]&begin=2020-03-27T00&end=2020-03-27T01&format=csv&group=m

But more details in the date and it no longer works, even if I ask for a full day like:
http://iotawatt.local/query?select=[time.utc,Main]&begin=2020-03-27T00:00:00&end=2020-03-27T00:59:00&format=csv&group=m

Which should give me the exact same thing as the above
Instead I get the error:
{“error”:“invalid query. unspecified”}

As soon as I specify a date in full ISO format, I get that answer.

What am I missing or is this a bug?

Now of course I could retrieve the entire hour and extra the data I want, but according to the doc specifying a full ISO date/time should work, and it’s less work for me :slight_smile:

Thank you

Yes, that problem was reported as issue #252. It was fixed in release 02_05_03. That release through the current development release 02_05_05 are only available as ALPHA and BETA auto0update. MINOR and MAJOR are still on 02_05_02 pending resolution of an mDNS problem.

But notwithstanding the bug, there is a subtle inaccuracy in your posting.

The two queries should not give the exact same thing. The first query ends on the hour, and the second ends a minute before the hour, so the first will return 60 data points, and the second will return 59 (The last will be 00:58:00). Note that a group represents the measurement that begins at the associated time and ends at the time + group or the query end time, whichever comes first.

To be equivalent the second query would be:
http://iotawatt.local/query?select=[time.utc,Main]&begin=2020-03-27T00:00:00&end=2020-03-27T01:00:00&format=csv&group=m

Also note that the ISO time is interpreted as local time for the IoTaWatt, and you are asking for UTC in the response, so the data timestamp will be skewed by your timezone offset. Use the method .local instead of .utc to get local time. I realize that there is an argument that can be made that ISO allows for specifying UTC or a timezone offset, but the times are considered local.

Thank you for your detailed answer.
I’m only interested with the data of the last minute as I will poll the iota that way.

I do want UTC time, not local. That’s how I store it in my RRD database.
BTW, when specifying UTC time as input (using Z as suffix) I found that it made no difference whatsoever. The data returned is always the same.

On the topic of time. I don’t believe the UI allows for properly managing time zones. The GMT offset isn’t enough. The actual time zone is required, between northern and southern hemisphere, daylight savings offset are inverted.

I would also vote for a 30 minutes presentation interval. That’s how all Australians distributors are collecting usage. Makes it for easier comparison between the detailed usage you get from them and what the iotawatt reports.

Once again thank you for this great tool and the time you take to answer. Much appreciated.

Cheers
JY

With the method .utc you will get the UTC associated with the data, it’s just that the time period will not match the begin and end specifications.

I was wrong above when I said the ISO is always local time. As you indicate the specification allows adding a Z for UTC time. The fix for the seconds bug also makes that work. Using this query:

http://iotahome.local/query?select=[time.utc,time.local,Main_1]&begin=2020-03-27T00:00:00Z&end=2020-03-27T01:00:00Z&format=csv&group=30m&header=yes

I’m in the Eastern Time Zone (UTC-5) and DST is in effect. I get this result:

Time, Time, Main_1
2020-03-27T00:00:00, 2020-03-26T20:00:00, 786.1
2020-03-27T00:30:00, 2020-03-26T20:30:00, 656.8

If you check “Allow Daylight Time” it will apply the DST rules for North America, Europe or Australia. Supporting more timezones for DST is complicated, but those three cover most of the installed base. The current DST rule will be applied to the data when using query.

Using this query:
http://iotahome.local/query?select=[time.utc,time.local,Main_1]&begin=2020-03-08T00:00:00&end=2020-03-08T08:00:00&format=csv&group=1h&header=yes

I get this result:

Time, Time, Main_1
2020-03-08T05:00:00, 2020-03-08T00:00:00, 1529.3
2020-03-08T06:00:00, 2020-03-08T01:00:00, 837
2020-03-08T07:00:00, 2020-03-08T03:00:00, 1306.7
2020-03-08T08:00:00, 2020-03-08T04:00:00, 961.6
2020-03-08T09:00:00, 2020-03-08T05:00:00, 1056.6
2020-03-08T10:00:00, 2020-03-08T06:00:00, 1378.7
2020-03-08T11:00:00, 2020-03-08T07:00:00, 1068.6

It should work correctly for Australia, but Africa and South America are not supported.

You can query at 30m intervals. Are you talking about Graph+? That’s doable. If you want to try your hand at it, there is a table in graph2.js:

image

You can add an entry for 30min:

image

Awesome thanks.

So, just to confirm but I got confused with your response early on.
If I set the time (begin and end)wwith the Z suffix, and ask for UTC time out. It will behave like I’m expecting : UTC queries and results.

Modifying the JS can be done on the deivce?

Yes, with a release greater than 02_05_02.

Very easily. Click on the file manager, then click on the graph2.js file in the file list on the left:

Scroll down to the table in the editor frame. Make the change. Press CTL S to save. Go back to the config app and run Graph+.

How are the calculations made?

Like I created an “Export” output. Which is the sum of all input - sum of all solar systems Max 0.
And then I draw the Wh figure in interval of 30 minutes.

You can’t just take the average of the inputs, subtract the average of the production.

It needs to do the subtraction for each data point

So how is the iotawatt doing it?

I would need more context to answer that.

In general, import and export need to be integrated over the finest detail available. The 5 second datalog is good to get an accurate result. Even the 5 minute resolution of PVoutput produces good results. Integrating over 30 minute intervals would probably be less accurate at the start and end of production and on intermittent cloudy days.

IoTaWatt can only report net usage (+import,-export). Within Graph+, users have replicated the PVoutput daily live data graph using sign limiting import and export outputs with Graph+ essentially integrating using the “accrue” capability on a Wh output.

Integration is easily done with with influxDB, Emoncms, or PVoutput.

I’m trying to reproduce the graph as shown by my electricity distributor. It shows the usage in 30 minutes interval, in kWh.

I can have a similar graph with Graph+, but I’m now wondering how the iota is actually calculating such 30 minutes interval. So is it doing the average Wattage during the 30 minutes and multiply by .5 (Wh)

Or is it taking each records and working in that.

To give an example. I get pay 60c for each kWh exported, and pay 30c for energy imported (make it a single tariff for sake of simplicity)

An usage of 1m import of 1kWh followed by an export for 1m of 1kWh would average 0kWh over 30 minutes.
But the billing would not be $0, it would be a credit in my favour.

So what I want the graph to show aren’t averages over 30 minutes or even 1 hour, but the discrete compounded values reported per 30 minutes interval.

Does it make more sense?

Mind you, I have already done such work in an old system I wrote over 10 years ago, but the graph on the iotawatt are way cooler. It looks super neat. Would love to do it all there.

If you are plotting Watts, you will get average Watts over the group interval. If you are plotting Wh, you will get the total net Wh over the group interval.

The IoTaWatt spends 67% of it’s time measuring power. Everything else has to be done in the remaining 33%. That’s sliced up into 16ms measuring 8ms everything else. The everything else is running the web server, uploading any data, maintaining the datalog, and servicing queries. It’s pretty hard to do serious data manipulation like integration. Most folks upload the 5 or 10 second data to a database like influxDB and let it do the heavy lifting.

The structure of the IoTaWatt datalog allows for direct reporting of average Watts and net Wh directly for any intervals. So it’s fast and accurate for what it does provide through Query on to Graph+, but that structure does not provide the import and export values of a PV system, only the net, which is fine for folks with true net-metering.

That said, tariffs like yours are not uncommon, so I’ve been wrestling with a way to do import and export. I have under development code to maintain a third datalog called the export log, which should maintain integrated export and import data for a given user function. That log will use the same internal format as the other logs and so should be able to report an integrated value over any interval using only the bounding log entries.

The devil is in the details. How to configure it in the user interface, and how to make the data available for export in the calculator scripts, and how to make it available through query require careful deliberation. So it’s probably not going to be available for awhile.

If you are plotting Watts, you will get average Watts over the group interval. If you are plotting Wh, you will get the total net Wh over the group interval.

That doesn’t really tell me how how the total net Wh is calculated :slight_smile: You haven’t answered my question really.

Sure, but how that net Wh is calculated is what I’m after.

Fact is, the Wh matches what my meter states quite well, so long as I use a one minute interval, then collate that over 30 minutes.

When I graph using 30 minutes or 1h interval, it’s definitely off and appears to use the average wattage * time interval.

I’m not too sure about what you mean in regards to net metering. To me, it seems that net metering can only be accurately calculated/reported by iotawatt if the CT clamp are placed on the mains.

this isn’t what I did. I placed multiple clamps so that current can only flow in a single direction. The solar feed use different clamps than the load ones, exactly because I wanted to see what was my real usage, not my net one.

I would say that if you have a look at the source code, as this is an open source project with all the details stored on GitHub, you might be able to answer your own questions. Have you had a look at the calculations in the source code? I would take a look in there first and then come back to @overeasy with any specific questions related to his code. Just a suggestion.

Probably, but with project that size the difficulty is to know where to start and what to look at.

FWIW, here is the output from my utility for yesterday

VS the iotatwatt Wh report.

The iotawatt time is 1h late (I don’t understand what’s happening there, the time offset seems to have no effect whatsoever on the graph output)

Not sure why there’s such discrepancy.

It is always nice to see some solar power being generated in these graphs, I like a sunny day myself. :slight_smile:

I’m in the same neck of the woods the Overeasy, and looking back at your post it appears you are in AUS. If so, I assume you have a three-phase install. Although such an install is outside of my ability apart from that I learned here on the forums, are you sure the IoTaWatt is properly monitoring and collecting all your phases? I know Overeasy has put together a quite detailed setup and troubleshooting write up in the DOCS section of the website. There is also a great group of other in your area that have been though it and are working well.

There are no X or Y legend for the Utility graph so it is hard to know the delta apart from the bumps. There is some consistency in areas apart form a few outliers (some rather large) and what looks like a lack of some phases not included in your total measurements.

Is the second chart you provided from Graph+ or from the data you pulled from the device? If pulled, I would suggest you also run the graph on the IoTaWatt as well and see if the delta is the same or not.

Even in a Split-Phase (US) install, there are things that I needed to ensure were correctly installed for me to see a match against my meter and the IoTaWatt.

I was on single phase until last month. It was indeed very simple back then. I had a current cost with two clamps to monitor two things

After upgrading my house to three phase power (to charge my car quicker) I got the iotatwatt.
I have 6 clamps, 3 connected to each phase, all circuits wires go through those clamps according to the phase they are connecting to.
The sub-circuit for my pool/spa is on another clamp (though connected to phase1), and then the 2 solar each have their clamp (solar 1 is on phase2, and solar 2 is on phase1)
I have one reference voltage linked to phase3. Phase1 and Phase2 are derived from that (the powerboard is by my door entrance in the all way. If I was to add two more GPO there my wife would kill me, it already look messy enough)

In the output iota setup I’ve set Total to be : phase1+phase2+phase3+pool
Then I have Total_Solar that is solar1+solar2 (both are negative).

so my net feed should be: Net_Feed Total + Total_Solar

I set Import as Phase1 + Phase2 + Phase3 + Pool + Solar1 + Solar2 max 0
And Export as: Phase1 + Phase2 + Phase3 + Pool + Solar1 + Solar2 min 0

When I look at the data output, it seems right, and match +/- 10% what the inverters are saying.

For the actual power usage, all I can say is that I’m 100% sure every single circuit is covered and going through a clamp.

|Phase3_VT|VT, Powertech MP-3027(Aus), phase:A|
|Phase1|CT, ECS16100, phase:B|
|Phase2|CT, ECS16100, phase:C|
|Phase3|CT, ECS16100, phase:A|
|Pool|CT, ECOL09, phase:B|
|Solar1|CT, ECOL09, phase:C|
|Solar2|CT, ECOL09↺, phase:B|

Getting the phase in the input settings was based on trial and error, by plugging something of a known load and playing with the value until it reads what I was expecting.
I can guarantee you I’ve read the iotawatt docs about a dozen time (superb docs btw, talking from experience that alone must have taken more time to write than the entire project code)

I do have a rather large voltage difference between the three phases, particularly when the solar is producing at its max. I’ve seen up to 5V difference. So +/- 2% which I can live with.

The 2nd graph is the one from the iotawatt directly.

I roughly now how much devices in my house are pulling when running, and so far the Iotawatt’s Data screen have always matched fairly accurately what I was expecting to read. Certainly much better than what the CurrentCost device did (but that’s because it didn’t have a reference voltage and was using fixed 230V)

Even without metrics, it’s plain to see there’s definitely something wrong if the utility meter is assumed correct.

I’m going by the picture you posted here https://community.iotawatt.com/u/jyavenard. So it’s a jungle in there, and I can’t see enough to sort anything out.

So to rephrase, the three CTs on the left capture all of the wires that go to sub-circuits. It looks as if the mains come in at the top of the upper left breaker. There is a three-phase breaker to the right of that with three beefy wires coming out. Is that the Tesla charger? If so, are they fed by wires that are under the shield and possibly not captured in the CT bundles?

That’s really not very close. I expect it should be more accurate. I see that Solar2 is reversed. Are Solar1 and Solar2 CTs oriented differently? They should not be, and if not, then you should not have to check reverse.

I could see the IoTaWatt being lower than the utility in some places if there are mistakes, but I cannot understand how the IoTaWatt got the large spike at 23:00. My guess is it has something to do with the EV charger.

So that’s a pretty big list of issues to sort out. The utility appears to be reporting import and export separately, so you will only get the net of import-export on the Graph+ display. In this case, it should only have effected the 8:30 and 12:30 bars as it was solid export between.

Can you explain why pool is only included in import and the EV is only included in Export?

yes indeed, it was very tight (why I absolutely needed the 9mm CT and not the 10mm opened one).
I still have those 10mm ones you sent, I got an idea on how I could plug them in and separately monitor the Tesla circuit (32A three phases)

It is for the Tesla.
However, the 6mm2 wire comes out of the main CB, go through the CT and then into the 3 phases RCBO (which is for the Tesla sub-circuit).
IIRC the picture I posted a while ago didn’t have the tesla wired through the CT yet. I needed longer 6mm2 wires which I didn’t have.

Yep, I am 100% sure they were oriented exactly the same. I checked twice as I was surprised by the output. However, the one used for Solar2 appears to be internally wired differently and I had to check the reverse option in the settings.
I didn’t know if it was a common problem or not. All three 9mm CT are oriented the same, one behaved differently to the other. Error made when soldering the wire probably.

yes, it is the Tesla charger. Being in lockdown, I don’t drive the car. So it typically recharge to the set settings in about 15 minutes and it’s set to start charging at 11PM which is when off-peak tariffs start. That is what you see on the graph.

an error in my post, the formula used is the same, the difference is min vs max.

Actually that was a rail the electrician had put. But it ended up causing issues and I got the electrician to remove it. The main 2-phases CB and the 3 phases RCBO being of different brand, the inputs didn’t perfectly match. As such, one phase was intermittently not connecting causing error with the Tesla charger.

I’m thinking it’s the utility report that is rubbish. I never ever see a spike at 11PM, and I have the car pulling 12kW at that time. daily