Home Assistant Energy - Return to grid not working

Sciro · August 20, 2021, 11:58am

Hi Community,

Have been using Iotawatt and Grafana for a couple of years and its been awesome. Thank you for the awesome product.

But i cant work out the home assistant energy module. Can you please help me.

Trying to get my return to grid calculations to be correct but the CT on my main feed goes negative when energy is being returned to the grid. So any output calculation that i try for energy returning to the grid is ignore because of the negative value on the main feed.

Anyone worked out a simple way?

Thankyou.

overeasy · August 20, 2021, 12:51pm

I’m still looking into this issue and possible resolutions. The integration relies on obtaining a daily running total of kWh for each input. This is easy for almost all of the inputs and outputs which typically have monotomically increasing values.

However with solar the maintained value is the net of import and export and so the integration does not work properly for that. What is needed is to maintain the independent sum of the negative (export) and positive (import) components of that net value.

What is needed is a way to maintain the integrated values (mathematecal function as opposed to HA integration) for upload. I’m working on a way to do that and will be releasing something soon.

PVoutput and the other external databases can easily provide this using the uploaded data. The HA integration takes a different approach and will require some extra functionality to maintain import and export.

dotnetdan · August 25, 2021, 9:18pm

You have to make 2. One for Input and one for Output

MainsConsumption|Watts|= (Main_In_Red + Main_In_White + Main_In_Blue) max 0
MainsExport|Watts|= ((Main_In_Red + Main_In_White + Main_In_Blue) min 0) abs

Then in HA config

platform: integration
source: sensor.iotawatt_output_mainsexport
name: Total_grid_export
unit_prefix: k
platform: integration
source: sensor.iotawatt_output_mainsconsumption
name: Total_grid_consumption
unit_prefix: k

overeasy · September 1, 2021, 1:37am

Hello @dotnetdan, we’re having some discussions concerning this solution to the import/export problem. Could I ask you to explain this in more detail and show how it is working in your HA instance?

dotnetdan · September 1, 2021, 4:47am

There are 2 seprate entitys needed for the ‘energy grid’. It is unable to have 1 and jsut take the negative number when your exporting which is annoying.
So I had to make 2 entitiys and limit one to only positive values (importing) and one only negative values (exporting)

I also had to create a ‘integration’ entity which is needed to show up in the settings.

overeasy · September 1, 2021, 1:26pm

Thanks @dotnetdan, and so you find that the resulting import/export in HA is accurate and matches any other measurements?

dotnetdan · September 1, 2021, 8:27pm

@overeasy It is close enough. I think HA has a rounding issue.
Comparison to PVOutput which is 100% correct

overeasy · September 1, 2021, 9:00pm

That’s helpful.

HA is integrating 30 second averages where PVoutput is getting 5 minute averages, so I would expect PVoutput to miss a little and be lower.

How do you know that?

dotnetdan · September 2, 2021, 8:29pm

@overeasy I thought that the PVOutput implementation takes ALL values and sends the average over the 5 mins.

The PVOutput results are always very very close to my meter reads.

The HA implementation only get a value every 30 seconds and averages that. Just assumed that this was inherently less accurate.

Sorry, No data behind such claim.

michaelpiron · September 2, 2021, 8:42pm

I confirm that the above approach works. Let me try to make it as clear as possible, by describing a step-by-step approach.

This is my home setup:

I have a 3-phase system without neutral. Total use is measured with 2 CT’s, connected to inputs 1 & 2 of my iotawatt. Solar is measured using CT on input 3.
Inputs configured in Iotawatt:

Looking at my home setup, you’ll see that the power exchanged with the grid is:
Grid = (Bord_Mid + Bord_Rechts) + Solar
(in this configuration, Solar has negative values when my solar panels produce)

I configured following outputs in my Iotawatt:

With these formulas I calculated not only my grid exchanged power, but I splitted this into two separate components: import and export. This split is needed in Home Assistant energy management.

The following graph visually shows how the split is made:

Now let’s go to Home Assistant:
Go to Developer Tools.
In the ‘States’ page, search for all entities which have the attribute “type: Output”:
This will display all outputs you defined in Iotawatt.

You’ll notice that each output is listed twice.
For example, output “export” has entities:

sensor.export : contains the Watts value
sensor.export_wh : contains the Wh value

We will need the Watts values.
Write down the entities names for the import, export and solar measurements.

In my case:

sensor.export
sensor.import
sensor.solar_2

Now open your configuration.yaml file, where we will add some configuration. For more information on how to open this file: Configuration.yaml - Home Assistant .

Add the following lines to the configuration.yaml file:

sensor:
  - platform: integration
    source: sensor.import
    name: import_wh_calculated
    unit_prefix: k
    round: 2
  - platform: integration
    source: sensor.export
    name: export_wh_calculated
    unit_prefix: k
    round: 2
  - platform: integration
    source: sensor.solar_2
    name: solar_wh_2_calculated
    unit_prefix: k
    round: 2

This applies the mathematical Rieman Integral to the Watts values (Integration - Riemann sum integral - Home Assistant).
It converts power (Watts) values into energy (Wh) values. The Home Assistant energy management works with energy values.

Now we go to the Energy configuration screen: Configuration > Energy dashboard.
There, assign the newly created energy entities to your dashboard:

Now, your Energy dashboard in HA will be populated:

The profile matches well with the graph I generate in iotawatt Graph+:

Hope this helps

Kind regards,
Michaël

overeasy · September 2, 2021, 8:56pm

Thanks @dotnetdan,

The 5 minute average is mathematically an average of all the values in the interval - typically 60. But so is the 30 second average to HA. If both methods give the same import during an import only period as at night, or during the middle of a very sunny day when there is no possibility for import, then I would lean toward the smaller 30 second increment giving more accurate results. That’s all good information.

The good news is that both are very close and reasonably accurate.

The bad news is that because PVoutput uses the difference between generation and consumption to calculate net import/export, no amount of integration of the 5 second data can improve it.

The HA Energy model can be made dead accurate to 5 second integration the way it’s setup now. They are talking about tapping into a net-energy value to get import and export. If they do that and remove the current method, they too will have limited accuracy with larger intervals.

A new solution is coming together nicely. It may require an external integration to get around the inappropriate rules for HA acceptance, but we’ll get it done.

overeasy · September 2, 2021, 9:25pm

Thanks Michael,

I was not aware that you are a three-wire system. I don’t have that documented. Did you get the setup from the forum or did I get involved? Is it derived reference and how well does it match your meter?

I always like a line drawing. Takes out all the guesswork. You should send this to our mutual friend. I have a few comments/questions:

Why integrate the Solar? Isn’t it always negative or nearly so with standby power?

Have you upgraded to the new HA release and latest integration?

I can see that the 30 second integration in HA is a little more accurate than the one hour integration in Graph+, especially in the transition hours. Once it gets a head of steam in the midday, they appear identical. Again as expected.

If you click on statistics with Graph+, you should see slightly different Wh numbers for import and export. As an exercise, click group=auto and I suspect they will get much closer as the interval will come down from 1 hour to 2 minutes.

So proof of concept from both you and @dotnetdan. I’m pretty confident in proceeding according to plan.

I’ve been at a disadvantage for years without solar, relying on others to do this. I’ve had the idea to insert a software pseudosolar input that could be used like an input in scripts and queries. The idea would be to produce the power and energy over any period assuming a sine profile from say 8-4 multiplied by a system power constant.

keiran · September 6, 2021, 11:57am

@michaelpiron - wow great wrote up. I just got my iotawatt last week (good timing with hass energy dashboard) but im having an issue i cant see my head thru it. Seems you have totally nailed it so thought id ask you. two problems actually

1/ Basically hass / iotawatt assumes my house is using the ENTIRETY of my solar produced. Thats not the case. It should be more like something like the red hand drawn line below.

2/ second issue is ‘Return to grid’ is showing 0 (yellow hand drawn line). Also i think it actually winds back my ‘consumed from grid’ numbers.

Your solar producing negative confuses me. Should mine be? Its currently positive. Have i got my CT clamp the wrong way around? Ive created 2x outputs (mainsconsumed, mainsexport) as per the hass docs. Ive not created your solar output logic. Any help really appreciated, thank you!

dheatherly · September 6, 2021, 12:56pm

you need to follow @michaelpiron post to the letter - including adding the integration step in configuration.yaml file. Prior to going through the steps - my results looked like yours. Now - they’re showing up correctly with return to grid and consumption graphed correctly - this is my graph from the other day -

keiran · September 6, 2021, 9:02pm

Ok thanks @dheatherly - good to know im in the right place for help! Seems i need the following new output (i only have one phase)
GRID = PHASE1 + SOLAR

(And yes i already have setup entries in my iota + hass for mainsconsumption and mainsexport as per docs here IoTaWatt - Home Assistant so i think im good here for the other two)

My question stands RE getting negative values for solar though. I thought i read you should put the CT clamps over the wires with the arrows pointing in the way the electricity flows. Ie solar system toward switchbox. ?

keiran · September 6, 2021, 9:29pm

Just noticed the original poster (@Sciro) has this output config. Does this assume every circuit in the switchboard is CT clamped? Im not currently doing this (i did purchase enough CT clamps to cover all, but im walking before i can run!)

I am falling down with electricity concepts here. I am technically minded but this hurts my head. I guess my question here is in order to get HASS dashboard working, do i need to clamp ALL my house usage circuits as a prerequisite before i can sort out the return to grid problem (my screenshot three posts above).

overeasy · September 6, 2021, 9:35pm

There are a lot of different ways to setup the IoTaWatt, but most users, and most examples in this forum, adopt the convention of solar being positive, mains being positive when importing power, and branch circuits being positive. If your solar connects to the grid through your load-center, as most in North America, then the mains will go negative when exporting power to the grid.

While @michaelpiron example is fine for his setup, I cringed at the idea of having the example be a three-phase system with the solar opposite to convention, and what is by convention called a type-1 installation where the solar feed is upstream (on the grid side) of the mains. Both are atypical.

Even with the solar observing this convention, the solution is not cookie-cutter. It varies depending on:

Do you have single, split or three-phase service?
Does your solar connect to the grid before or after your mains CTs?

Perhaps if you posted your inputs and outputs configuration, I can take a pass at it. The Hass people don’t seem to appreciate that IoTaWatt provides an intelligent layer between the CTs and HA, and using that adapts to any power system worldwide.

keiran · September 7, 2021, 7:45am

Thanks @overeasy - i really do appreciate your time. i am single phase here is sydney australia and here is my very simple understanding of the setup i have here this far. Including the extent by which CT clamps are currently deployed (red C) and their directionality.

As mentioned i did purchase enough CT clamps to cover all circuits (the remaining black Xs) so if me deploying them all is part of the optimal solution im of course open to that)

Heres my inputs/outputs at the minute:

In terms of your other questions

single phase
RE ‘Does your solar connect to the grid before or after your mains CTs’ - im not entirely sure, BUT my setup also shows negative values on IoTaWatt when solar is producing more than house is consuming and its flowing back to grid, so does that confirm one way or another?

Thanks again.

overeasy · September 7, 2021, 1:24pm

I think this is about as straightforward as it gets, but given that I have no direct experience with this yet, I hope I can get it right.

HA Energy wants to know three things:

How much energy was imported from the grid.
How much energy was exported to the grid.
How much energy was generated locally.

You have said that your solar has the ability to drive your mains negative, so that indicates you have what is by OEM convention a type-2 solar service, meaning the solar inverter feed downstream (on the load side) of your mains. So your single CT on the mains provides all the data you need for items 1 and 2. Import is the positive component and export is the negative component.

I would change MainsConsumption as you have it to Consumption = GRID + SOLARPRODUCTION

Then define a new output MainsImport as GRID max 0 that satisfies Item 1 above. MainsExport satisfies item 2. SOLARPRODUCTION satisfies item 3.

Now change the YAML configuration entry to:

sensor:
  - platform: integration
    source: sensor.MainsImport
    name: import_wh_calculated
    unit_prefix: k
    round: 2
  - platform: integration
    source: sensor.MainsExport
    name: export_wh_calculated
    unit_prefix: k
    round: 2

.
The Energy configuration stays the same, except Solar production is changed to SOLARPRODUCTION_wh

Give that a try. Unlike influxDB or PVoutput, there is no way that I’m aware of to update the history in Hass, so unfortunately this will only take effect going forward.

Thanks for posting this, please post your results in a day or two when it shows results. This is exactly the simpler example I had in mind as it’s more of a typical case. Once (if) this proves out. I’ll elaborate on the simple changes needed for split-phase systems and type 1 (solar feed upstream of the mains) systems.

keiran · September 8, 2021, 4:22am

amazing. Thanks so much for your time helping me through this. Ill try and get to after work today (after 4 hours or so). Ill loop back soon.