Damping on status page causing issues with Home Assistant

I’ve configured a REST sensor in Home Assistant to pull data from http://iotawatt.local/status?inputs=yes&outputs=yes&stats=yes. One input is my hot water system.

When on standby the IoTaWatt graph shows it is consistently using ~1.5W (+/- 0.25W) and so I have configured another template sensor in Home Assistant to report when the HWS is “on” (>100, because it jumps to 75W momentarily on startup) else “standby” (>1) else “off”. But the data reported by the status page linked above consistently shows that input as “0”, with very occasional jumps to “2” or “3” (~10 seconds apart when refreshing the IoTaWatt status page).

So Home Assistant thinks my HWS is always “off” instead of “standby”, but every half hour or so (when its poll happens to return to value of 2 or 3) it reports it as standby then immediately off again.

Is there a better way to get data from IoTaWatt and into Home Assistant? Why does IoTaWatt “damp” a consistent 1.25-1.75W value down to 0?

UPDATE: To be clear, I think a damped value should be OK for Home Assistant, but why is the damped value being reported as “0” 90% of the time when it’s actually a stable ~1.25-1.75W?

Over time, the algorithms for the status page have been refined so as to give a good snapshot of what’s going on. For most folks, 1W should be zero, especially if it’s going from zero to one randomly. It just looks bad, never mind that it’s amazing IoTaWatt can detect 1W at all with a 50A CT.

I know there are folks that are using the status API to get data, and until now I’ve not offered any alternative, so I guess I should expect folks want to redefine what it is. Wait a minute, did he just say “until now”?

Yes I did. With version 02_05_01 now in ALPHA and BETA, there is a new query API that you can use to get data directly from the datalog. You could have done that before with the API used by the old Graph, but it wasn’t documented and doesn’t have a lot of the features of the new query API. With this API, you can ask for the current value, or the last n values. You can get the response in Json or as a CSV table.

The documentation is here https://docs.iotawatt.com/en/staged/query.html.

This query drives the new Graph+ that is also documented in that “staged” version of the documentation. I’ll be publishing a 02_05_01 version when that goes MINOR, probably next week.

2 Likes

Awesome. I’ll take a look at those docs and give that a try instead. I understand rounding 0-1 down to 0, but I still think a consistent value between 1 and 2 should be rounded to 1 or 2, not 0 :smiley:

It’s open source. Have at it and take the cure.

@overeasy I am experimenting with the new show/select API now. I am having a bit of trouble figuring out how to get the current value for each input/output, and get it into Home Assistant in an easily usable format.

The closest I can figure out is to get the most recent 5 second average with http://iotawatt.home/query?select=[series1,series2,...]&begin=s-5s&end=s&group=5s&header=yes

But even then, the format of the JSON returned by IoTaWatt is not easy to access in Home Assistant via REST sensor. I’ve had to resort to rendering a custom JSON string via Jinja in an intermediate template sensor and then converting that string back into an object in a third template sensor:

Could we possibly add a new ha endpoint to the query API that returns the unit type and latest value (or average across the specified time period) for each input/output, as a top level map?

For example:

{'latest': {'voltage': {'unit_type': 'Volts', 'Volts': 249.5}, 'mains': {'unit_type': 'Watts', 'Watts': 123.5, 'Wh': 12345.6}}}

It should then be super easy to add this JSON data directly as an attribute on a simple REST sensor in Home Assistant, that could be used throughout Home Assistant as-is or subsequently used as the basis for further template sensors. Something like this:

sensor:
  - platform: rest
    name: IoTaWatt
    json_attributes:
      - latest
    resource: http://iotawatt.home/query?ha
  - platform: template
    sensors:
      hot_water_system:
        value_template: '{{ state_attr('sensor.iotawatt', 'latest').hot_water_system.watts }}'

Alternatively, an additional option to the existing select API to return a single/average value and input/output name in a map as an extra formatting option?

Have you tried to use the CSV output format?

I’m open to developing a generic HTTP output service provided it could satisfy a larger audience.

As an old warhorse trying desperately to keep up with modern tools, I am constantly blown away by new tools that I discover. So I’m open to suggestion on this. But my current thinking is a basic service that sends a user formatted HTTP request at a specified regular interval to a some given URL.

This would be for current data only. The average of some specified interval relative to the sending time.

The payload could be a string defined with a C++ printf format, and the data fields could be strings and values defined as a set of calculator outputs.

I have not had time to explore this, but I think there are MQTT agents that will accept HTTP, and so this might be able to serve the purpose of directly producing MQTT from IoTaWatt.

If I had confidence that this would satisfy more of the feature needs of the user base, it’s really a pretty simple service.

I have not tried CSV, but I don’t see how it could solve the problem I’m facing.

I want to access current input and output data by name so that changes to input names don’t require that I adjust a tonne of numerical indexes in home assistant configuration.

CSV would still have one row of sequential input and output names and one row of sequential values that I can’t zip together into a map in home assistant.

Having IoTaWatt push data via mqtt or having home assistant poll for data is neither here nor there for me.

I just need a URL to poll that gives me a json map of the data as a top level key (so I can save the whole map as a sensor attribute) with input and output names as the key.

Reformatting the existing output to provide each series of data in a separate list keyed by input name, instead of as a list of lists, could also work. But would require changing your existing consumers.

Adding a new formatting option that returns the data like that, keyed by input and output name, could support other consumers that work differently than the internal graph library.

It would also be nice if I could request “all” series without explicitly naming them all. I haven’t found a way to dynamically build a rest sensor resource url in home assistant that includes these, so I have to hard code them all.

FYI in case anyone is interested in Home Assistant integration, and to show @overeasy the kind of hoops I currently need to jump through, here’s a working (with some caveats) Home Assistant configuration that uses the new IoTaWatt select API:

1 Like

Would it be useful if the current value were included in the show=series query? My thinking is that it might default to the most recent measurement (s-5s, s) or allow specifying begin and end as in the select= query with group=all implied.

I think you’d need to add the current value(s) and display the series as a nested map (with the input name as keys) instead of a list of maps (with the input name as values).

With data like {"series": [{"name": "mains", ...}, ...]} I’ll still have to do some template gymnastics in additional “sensors” in Home Assistant that exist purely to reformat the data.

Might as well just keep show=series as-is and add show=current or show=ha that outputs more (all current) data and in a different format?

For example:

{
    "voltage_series": {
        "voltage": {
            "V": 245.6,
            "Hz": 50.1
        }
    },
    "power_series": {
        "mains": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "oven": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "air_conditioners_1": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "hot_water_system": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "solar_analytics": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "power_points_1": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "lights_1": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "power_points_2": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "lights_2": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "air_conditioners_2": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "fronius_inverter": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "latronics_inverter": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "air_conditioners": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "consumption": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "grid_export": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "grid_import": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "grid_import_neg": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "lights": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "mains_inverted": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "power_points": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "production": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "self_consumption": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        },
        "unmeasured": {
            "Wh": 567.8,
            "W": 123.4,
            "A": 5,
            "kVA": 6.7,
            "PF": "0.988"
        }
    }
}

Then my Home Assistant integration could be as simple as:

sensor:
  - platform: rest
    name: IoTaWatt
    resource: http://iotawatt.home/query?show=current
    json_attributes:
      - voltage_series
      - power_series

And I could access those values by input/output name in my HA automations, or create additional template sensors for the data I need to expose directly:

sensor:
  - platform: template
    sensors:
      iotawatt_air_conditioners_w:
        friendly_name: Air Conditioners
        unit_of_measurement: Watts
        value_template: '{{ state_attr("sensor.iotawatt", "power_series").air_conditioners.W }}'

It could select from begin=d to end=s by default (and allow you to input your own begin/end) and report the last recorded value in the range as W, and Wh for the whole period.

Let e mull that over. I have a few questions:

Is one second data of interest?

If IoTaWatt could be ganged such that one could speak for all, how do you see that working in this scenario? Would it be a flat namespace where all of the ganged units would use unique names, or a hierarchical system where names are qualified by “node”? What would be the Json representation of that to HA?

Does HA only pull, or can you push to it?

On one hand, I’d love to see 1 second data, but I imagine that is overkill for Home Assistant integration. HA’s rest sensor polls at 30s by default, but it is configurable via scan_interval.

I don’t know if there’s a minimum value allowed, but I think 1s would probably produce too much data (HA logs it all to produce graphs of its own) for little actual noticable benefit. For example if an automation to automatically turn off a switch when all its devices enter standby mode triggers in 1s or 30s is of little consequence.

In terms of aggregating the data from multiple units, I don’t think it matters much. HA could be configured to fetch data from each IoTaWatt separately and the input/output names would not need to be unique. Users would access the data first via the rest sensor namespace.

If IoTaWatt were to aggregate the data, the input/output names could either be unique and flattened, or non-unique and grouped under a node. As long as the node itself had a unique and consistent name.

I don’t know if HA supports having data pushed to it. I assume so, but the rest sensor is so simple and easy to integrate with anything that can provide JSON data. I had a quick look at the MQTT docs but it wasn’t immediately obvious to me how it would work, and looks to require more configuration and a bit more learning than a rest sensor.

Plus anything that involves the IoTaWatt pushing data to HA would mean configuring two things, HA receiver and IoTaWatt sender. So I’m in favour of keeping things simple. The rest sensor is simple. It just needs a slightly different JSON format to overcome some limitations in HA and make the data more easily and consistently accessible.

UPDATE: Making the average period configurable so users can make it match the HA scan interval, and perhaps 30s by default, but allowing users to set it as low as 1s if they want to poll that frequently, would be good (if 1s polling is not too heavy for the IoTaWatt).