[Feature request] Unique identification of outputs

falstaff · March 15, 2023, 8:12am

Currently, for a external consumer it is not possible to identify a particular user configured output of IoTAWatt. There are two options, but both have their downside:

Use the name: Those can change when a user changes the name (e.g. a typo). The external system now would see that output as a “new” output, which is not really the intention of the user.
Use the key of the JSON outputs property (from status?outputs=yes). This key seems to reflect to order of the list of outputs. The list gets alphabetically sorted automatically. That means, after creating a new output the existing output potential get a new key (and therefor ID).

For external systems, it would be helpful if each output receives an unique ID e.g. an UUID, so an external system can understand when the user creates a new output through the IoTAWatt configuration interface, and present it as a new output.

This is a requirement for Home Assistant (see unique ID). The current options are not acceptable (see discussion in this pull request), which causes that IoTAWatt output entities are not configurable through the user interface.

overeasy · March 15, 2023, 3:00pm

I am well aware of this ongoing problem with the HASS elite. They have dug in their heels demanding a unique ID when there is simply no such thing. Their example in the PR discussion cites a light bulb example. The light bulb is a physical device and has a unique ID that is factory set. I get the requirement to register those devices using their unique ID rather than an arbitrary name. There is simply no physical way to identify an output. Any added identifier would be just as volatile as the name and would just complicate the user interface.

IoTaWatt does not retain any information about outputs. Their values are not saved in any datalog. They are simply Scripts that are processed real-time when referenced and the result is a function of one or more input values.

Existing outputs specifications are discarded, and new ones created, every time the configuration is processed. Not just the configuration for that particular output or even all of the outputs, but ANY change that is made to the IoTaWatt configuration. You could simply press stop for an uploader. Within milliseconds the entire configuration is reapplied.

Any attributes for outputs would need to be contained in that configuration, and user supplied. How is that any different than supplying a name? It would only complicate the configuration process for all users, and obfuscate the process of understanding how the named outputs are actually handled by HASS.

I get the underlying logic of a physical ID for something that has an immutable identification. There is just no such thing available. IMO the HASS rule about IDs should be amended to say that if an entity has an immutable ID it must be provided, otherwise the common name is sufficient.

At the time the HACS integration was made “official”, I had proposed to Baloob that they consider a push protocol. He told me that it had to be pull. I see that now other energy monitors are using a push protocol. IoTaWatt has the ability to push data to influxDB, Emoncms, and PVoutput. There is an underlying infrastructure to facilitate pushing data.These connections have high integrity because they will first query to see where uploading was interrupted and backfill all missing data. I don’t believe any other energy monitor will do that. The HASS integration was user developed and uses the general purpose query facility to pull data.

I bring this up because the Emoncms uploader does have the notion of an ordinal number associated with each Script that is uploaded. Moreover, the way a push uploader works is that it only uploads what you tell it to upload, so the endpoint is not overwhelmed with every input and output that exists. At present I don’t believe HASS differentiates between outputs that are intended to provide Amps, VA, PF, Watts or Wh.

It would be a bigger project, but that would make some sense and provide data compliant with their unqualified rules.

UPDATE:

Upon reading the unique-ID requirement, I believe it has not been applied correctly in this case. When I look at their examples:

Clearly none of them are applicable to IoTaWatt outputs. They also describe this situation:

That is exactly what IoTaWatt outputs are. Setup by config entries. Perhaps we are arguing semantics. The exception allows for the Config Entry ID. That is the “name” in the IoTaWatt config. An excerpt from an IoTaWatt config:

“outputs”: [
{
“name”: “baseload”,
“units”: “Watts”,
“script”: “@1+@2-@3-@5-@6-@9”
},
{
“name”: “export”,
“units”: “Watts”,
“script”: “!-grid|”
},

Perhaps the ID could be the MAC address of the IoTaWatt (available) combined with the output name. I believe that satisfies the “rules” as I read them.

falstaff · March 15, 2023, 9:33pm

That is pretty much what the linked PR suggests. But the output name is an user configurable identifier.

Let me explain how I understand things: The HA requirements of unique ID aim to provide coherent behavior across services and devices. No matter if a “thing” is virtual or physical: The “thing” comes to live, the “thing” gets changed, the “thing” gets deleted. It all should behave seamlessly: The “thing” appears in HA, if changes are attributes visible in HA, they get transferred to HA, the “thing” gets deleted in HA.

For most physical things such unique ID appear naturally. For virtual things most often as well (usually a database primary key, or a UUID is used to identify a new record of a “thing”).

IoTaWatt outputs are also such a “thing”. The users creates with an intention (e.g. I want the sum of my 2nd floor power usage). So he clicks “add”, enter “Upper Floor”, enters the calculation. In Home Assistant he configures the output attributes such as Icons etc. Now he goes back, to IoTaWatt, presses “Edit”, changes the formula. That will work no matter what. Now he goes back and actually realizes he wants to name it “2nd Floor”. If we were to use the Name as unique ID, that last operation discards all settings in HA, since the Output now has a new “unique id”. That clearly wasn’t the intention of the user. He pressed “Edit” of that output, and just changed its name…

A technical inclined user might understand that the name has been used as identifier, and that therefor the settings for that output got lost.

But I think most users these days expect things to work “more” seamless as in, a “thing” stays the same thing on both sides, is “connected”, no matter what properties I change.

I don’t want to go into politics, I just want to point out where the requirements in HA come from. They help to ensure a certain user experience.

I’ve looked at the POST requests when pressing save, and I see what you are saying. But in the end, this is how it works behind the scenes.

For me as an user, I just click on “Edit” of a particular output, and then change its properties…

Ok, but an id field could easily be part of this configuration and not user changeable (on the web frontend). E.g. “Add” could just increment the highest id currently present, and store it along the other attributes. “Edit” would just not change the existing id. Sure, that wouldn’t be a bullet prove method (as in, the user can fake POST), but that is not the requirement for such an id.

“config entries” in a HA semantics is a thing created by the user through the HA web interface (“Add integration”). The rule allows to use the HA internal identifier if it gets created by the user in HA itself and the thing has no external identifier. When the IoTaWatt (device) config entry gets created it provides an unique ID for that already. The outputs do not get created through the HA interface (“setup by a config entry”), so we can’t use that rule.

overeasy · March 16, 2023, 2:45am

Thank you for your efforts. From the HASS documentation I see that there are fewer than 400 users of this integration. There are perhaps 10,000 IoTaWatt users worldwide. The only problem noted with the proposed MAC||name ID is that changing the name will unlink any history. That can be forewarned to HASS users and call it a day.

The work required and complexity involved is substantial, and the release protocol would require at least four months for any change to be mainstream. I have never pushed out a release on a shorter timeframe. I know the HASS folks throw a release over the wall every month. I don’t do that. I’ve got commercial and industrial customers and limited testing resources.

Of all the complaints I have fielded about the HASS integration, the early ones involved problems with getting accurate import and export numbers and I resolved that with a six month effort to add integrators which has eliminated the issue and I would argue provides more accurate data than can be accomplished with HA Energy alone. The most frequent current issue is that users cannot find the import and export outputs. I cannot recall anyone complaining about the aftereffects of changing an output name. I think this rule has created a solution looking for a problem.

If the PR is rejected, I think the best approach is to offer the integration via HACS. If you do that, I can add a section to the IoTaWatt documentation explaining the how and why of using it.

jenkins · April 10, 2023, 4:39am

Hello!

I got here from HASS Issue #87182 (Error: Platform iotawatt does not generate unique IDs. · Issue #87182 · home-assistant/core · GitHub)

I agree that having the name as ID for outputs should be OK.

But, I got curious of what could be a way to implement really unique identifiers for outputs.

I looked at the way IoTaWatt saves the outputs, and as you already mentioned there is call to fetch the file http://<iotawatt.local>/config.txt when the user loads the UI, and when they create an output the endpoint /edit is called with multipart/form-data with the whole configuration data which I suppose is overridden. The relevant part of that configuration related to outputs is as following:

outputs: [
    {
        name: "Home",
        units: "Watts",
        script: "@1+@2"
    }, {
        name: "T2",
        units: "Watts",
        script: "@3+@4"
    }, {
        name: "T3",
        units: "Watts",
        script: "@5+@6"
    }, {
        name: "Washing",
        units: "Watts",
        script: "@12+@13"
    }
]

Maybe a way to implement UUID could be web UI handled as following:

When the user creates a new output, along with the name, units and scripts, a uuid field is added. We may use Javascript’s crypto.randomUUID() call, for example:

> crypto.randomUUID()
'5347261a-19f7-44cf-89e2-ff267cb8bd8b'

The configuration is sent and stored. For example:

outputs: [
    {
        name: "Home",
        units: "Watts",
        script: "@1+@2",
        uuid: "f66afed4-9935-4af7-aed6-a6fb78946eff"
    }, {
        name: "T2",
        units: "Watts",
        script: "@3+@4",
        uuid: "70a112c6-a138-42c0-8207-c33551051235"
    }, {
        name: "T3",
        units: "Watts",
        script: "@5+@6",
        uuid: "ec335946-0d69-411d-8e83-d791a1d68d9a"
    }, {
        name: "Washing",
        units: "Watts",
        script: "@12+@13",
        uuid: "d92e39e0-8ca5-4223-a6ee-f0403b796875"
    }
]

Special care is taken when a user is editing an entry. Since the UI knows that we are editing an entry (name, units, script is loaded, and there is a delete button) we just need to maintain that UUID and not generate a new one.
The query endpoint is updated to now include that new field. Totally backward compatible.

For the IoTaWatt firmware there is no difference at all, its only responsibility is to store the value, which could really be anything actually.

There must be a way to make this backward compatible with older firmwares. So, when the config is about to be saved again (/edit), check if there is any output without UUID and assign one.

I looked quickly in the frontend, and found that maybeee here (really, not sure at all) this logic could reside: IoTaWatt/index.htm at ca0a36849c1c833de40aac833e73e3f573db3f8a · boblemaire/IoTaWatt · GitHub

After that the iotawattpy package could be updated to generate unique sensors ID, and then the integration could use them.

What do you think? Terrible idea? Good idea? Lazy idea?

Regards!

overeasy · April 10, 2023, 1:38pm

On the face of it, all of these approaches have merit. But simply trading one avoidable and arguably bad user experience for another make little sense. The HA folks believe this would be a simple change for IoTaWatt that completely resolves the potential for a negative user experience.

Although not committed to a set of strict rules, I understand the desire to improve user experience. IoTaWatt has a design philosophy to that end. The UI attempts to avoid complicated error situations by design. For instance, if you “add” a new output with the same name as a previous output, it doesn’t tell you that you are creating a duplicate output, it simply replaces the pre-existing output by the same name.

And that is the crux of the problem here. If you were to create a new output by the same name with the logic you suggest, the UI would create a new UUID for the “new”/old output. Now we would have the situation that HA would not recognize that output as the same output that was previously defined.

Another problem is that IoTaWatt doesn’t have an “Are you sure?” dialog when things are deleted. The idea is that if you delete something by mistake, you can just recreate it. But that’s not the case once you add this UUID. The recreated output would have a new UUID and would not relate to the old in HA.

It seems like a pretty easy thing to explain to someone why changing the name of an IoTaWatt output will cause it to become a new entity in HA. By contrast I would not want to defend the situation where an output by the exact same name has lost its history in HA. Trading a simple intuitive explanation for a complicate and convoluted situation, and for what?

Someone suggested simply maintaining an unofficial integration that uses the name||MAC unique ID to establish an entity. I think that is the most practical and expedient resolution to this impasse.

jon · January 9, 2024, 5:46pm

Hi there!

I was reading this over pre-purchase, so please forgive me if I’m misunderstanding anything. My plan is to integrate with HA, so I was looking in to how well the integration worked and stumbled upon this.

Nothing here has made me think I shouldn’t move forward with IoTaWatt, it still seems like the best solution.

I think a lot of great points have been made on both sides. I also get frustrated by how dug-in the HA team can be on certain things.

I’m curious about these two points:

From a developer’s point of view and that of simplicity, I sort of understand these decisions.
As a user however I would never expect mature software to act this way. I’d be curious what percentage of the 10k+ users of IoTaWatt would object to these UX changes being made. I am having difficultly thinking of situations where it would cause problems with a UI based workflow.

I am cognizant of the minority of users affected by this - 448 users using the core integration, maybe another 50 using something custom, so we’re looking at probably well under 5% of users who care at all about the HA side of the problem. I think, even if the UUIDs are ignored here the other bit is something that makes sense to consider changing from a UX perspective.

overeasy · January 9, 2024, 8:20pm

I think the problem is that folks are talking about a UUID as if it’s something physical. To be sure, I completely agree with the HA folks that where there is an ability to identify a physical resource, that information should be used as a primary ID. There are 15 physical IoTaWatt inputs, and the integration uses that input number as a UUID in registering input entities. You can rename an input and the historical data remains - for better or worse.

On the other hand, outputs are not physical entities. They are metrics calculated from the raw data of inputs. You might just as well call any UUID an alias name. It is just made up - as the name is.

HA would like to push the issue of managing their database to IoTaWatt. An output can be altered to represent a different metric, and the same metric can be given a different name. Absent a bunch of code in the IoTaWatt UI to quiz the user to determine whether the “HA alias” should be changed to get the desired result in HA, it adds no value to IoTaWatt.

As far as I can tell, data from an MQTT feed is created with no UUID. What is the difference?

The HA integration was user contributed. The creator and the follow-up, both competent and well-meaning, appear to have burned out jumping through HA hoops. For my part I made a major improvement to IoTaWatt to mitigate deficiencies in HA energy where Watt-Hours were not correctly handled for PV, battery and other metrics that were not monotonically increasing or decreasing values. To this day, HA does not allow any backfilling of historical data after a communications outage. IoTaWatt does that with other connected databases like emoncms, influx and PVoutput.

IMO while HA has its uses to control IOT stuff and other smart-house functions, HA Energy provides no reliably accurate historical information. Given that IoTaWatt is in the business of doing just that, along with other external database partners, I’m not really interested in opening this can of worms.

jon · January 9, 2024, 10:04pm

I appreciate your reply here and work on the project. Definitely some food for thought. I didn’t realize the deficiencies of HA’s energy monitoring as I’m just starting to tip-toe into it. Thanks again!