API bug for certain output calculations

To reproduce:

Set up an output “Solar_Surplus”:
Watts = 0 max (0 - Main_1 - Main_2)

Then query it via

curl “http://192.168.1.168/query?select=\[time.iso,Solar_Surplus\]&begin=y&end=m&group=all
or
curl “http://192.168.1.168/query?select=\[time.iso,Solar_Surplus.wh\]&begin=y&end=m&group=all

always results in:

[[“2021-01-01T00:00:00”,0]]

Even though i can see results in Graph+. Same query works fine for other outputs, so i believe the issue is with the calculation.

This might seem like an edge case, but the new Home Assistant Energy dashboard requires a “Return to Grid” value. I don’t see any other way to calculate this. (issue also exists if you use:

0 max (-1 x (Main_1 + Main_2))
(0 min (Main_1 + Main_2)) abs

or even (for a negative value)
0 min (Main_1 + Main_2)

This is a fairly complicated topic. You have a couple of different issues here:

You seem to be trying to do calculations with solar import and export values. Those numbers are not readily available from IoTaWatt over longer timeframes because they require integrating the higher resolution data, which is an I/O and compute intensive activity that the ESP8266 just doesn’t have the resources to do. There are ways to accumulate import and export, but they require sacrificing additional inputs to obtain datalog space to save the data.

I have explained parts of this issue in various posts on the forum, but really need to add a thorough explanation of the problem and various solutions to the documentation. So I’ll try to do that over the next couple of days.

I will say, without actually seeing your data, that I don’t think there is a problem with the output calculation. I believe it is a lack of understanding how the script system works and what data it works on and when.

There are ways to track solar import and export over longer periods, but it requires dedicating additional resources to it or using an external database with the space to save the accumulated values and/or the ability to integrate the detail data quickly.

Thanks - I won’t pretend to understand this (sucked at calculus) but, if i create two outputs as follows:

test1: Watts = 0 min (Main_1 + Main_2)
test2: Watts = 0 max (Main_1 + Main_2)

Then query:
curl “http://192.168.1.168/query?select=\[time.iso,test1.wh\]&begin=y&end=m&group=all
curl “http://192.168.1.168/query?select=\[time.iso,test2.wh\]&begin=y&end=m&group=all

Why does test2 give results, but test1 returns 0?

also, setting wh aside to avoid integration:

curl “/query?select=[time.iso,test1]&begin=y&end=m&group=all”
curl “/query?select=[time.iso,test2]&begin=y&end=m&group=all”
(url removed because it won’t let me post 4 “links” :))

test1 still returns 0, while test2 works

min is a binary function that returns the smaller of the two operands, so if (Main_1 + Main_2) is positive, it will return the lesser value which is zero.

Conversely max is a binary function that returns the larger of the two operands, so if (Main_1 + Main2) is positive, it will return the greater value which is the positive (Main_1 + Main2).

Same answer.

I fixed that…

I am going to document this issue so there is a common understanding of how these net inputs are calculated, but the way min and max works is your issue here and is pretty straightforward.

I know what a min and max mean :slight_smile:, just trying to illustrate my issue and failing to do so.

I think i understand your explanation now though, at least sorta. I’ll try to pull the wh from influxdb.

I’ve posted a more detailed explanation.

thanks this is actually really good info! I’m going to play with using influxdb. Trying to get Home Assistant energy dashboard working - not sure this will be a solution just yet without some code changes to the influxdb integration, but i think this path makes a lot of sense. Any idea when your new feature (sacrificing an input for data storage) might be ready?

FYI, this as well as my thread earlier this week both relate to the Home Assistant Energy Dashboard and the integration with IoTaWatt (see Add energy sensors by agners · Pull Request #6 · gtdiehl/iotawatt_ha · GitHub). I don’t have solar unfortunately, so this problem did not affect me.

That said, something is unclear to me right now. In your detailed explanation you wrote:

With the above metrics, power (Watts) and energy (Wh) for a time period up to a day can be graphed using Graph+ by simply graphing the appropriate inputs and outputs corresponding to the above metrics.

How is that enforced exactly? E.g. what happens when I query Wh over a period of a year with an output making the Import/Export calculations? Is just the integration accuracy dropping or is the query getting slow?

Since Home Assistant stores the data in its own database, I think having accurate Import/Export data for a single day is absolutely enough since we poll the API anyways more often than that.

From what I understand switching to querying the API with begin=d (or even hourly) should be enough?

I should have said Wh … up to a day can be graphed accurately using Graph+. This is in the context of graphing an input with a net value (allow negative values). The post explains how net value can become less accurate as the interval increases. There is no integration happening, that is the problem when dealing with longer intervals. The query for a single interval (group=all) is trivial and very fast whether the interval is a minute or a year.

If HA is storing the discrete import and export Wh, and the query period is small, like a few minutes or less, there should be no problem.

yes, that should work fine.

I saw in the git that someone made the statement that the Wh accumulation starts after the output is defined. That is not true. Wh are accumulated continuously and that has nothing to do with the output definitions.

I need to better understand what you are trying to do. This should be pretty easy if HA is keeping its own database of import/export.