[Feature request] Unique identification of outputs

[falstaff]

Currently, for a external consumer it is not possible to identify a particular user configured output of IoTAWatt. There are two options, but both have their downside:

• Use the name: Those can change when a user changes the name (e.g. a typo). The external system now would see that output as a “new” output, which is not really the intention of the user.
• Use the key of the JSON outputs property (from status?outputs=yes). This key seems to reflect to order of the list of outputs. The list gets alphabetically sorted automatically. That means, after creating a new output the existing output potential get a new key (and therefor ID).

For external systems, it would be helpful if each output receives an unique ID e.g. an UUID, so an external system can understand when the user creates a new output through the IoTAWatt configuration interface, and present it as a new output.

This is a requirement for Home Assistant (see unique ID). The current options are not acceptable (see discussion in this pull request), which causes that IoTAWatt output entities are not configurable through the user interface.

[overeasy]

@falstaff

This is a requirement for Home Assistant (see unique ID). The current options are not acceptable (see discussion in this pull request ), which makes causes that IoTAWatt output entities are not configurable through the user interface.

I am well aware of this ongoing problem with the HASS elite. They have dug in their heels demanding a unique ID when there is simply no such thing. Their example in the PR discussion cites a light bulb example. The light bulb is a physical device and has a unique ID that is factory set. I get the requirement to register those devices using their unique ID rather than an arbitrary name. There is simply no physical way to identify an output. Any added identifier would be just as volatile as the name and would just complicate the user interface.

@falstaff

For external systems, it would be helpful if each output receives an unique ID e.g. an UUID, so an external system can understand when the user creates a new output through the IoTAWatt configuration interface, and present it as a new output.

IoTaWatt does not retain any information about outputs. Their values are not saved in any datalog. They are simply Scripts that are processed real-time when referenced and the result is a function of one or more input values.

Existing outputs specifications are discarded, and new ones created, every time the configuration is processed. Not just the configuration for that particular output or even all of the outputs, but ANY change that is made to the IoTaWatt configuration. You could simply press stop for an uploader. Within milliseconds the entire configuration is reapplied.

Any attributes for outputs would need to be contained in that configuration, and user supplied. How is that any different than supplying a name? It would only complicate the configuration process for all users, and obfuscate the process of understanding how the named outputs are actually handled by HASS.

I get the underlying logic of a physical ID for something that has an immutable identification. There is just no such thing available. IMO the HASS rule about IDs should be amended to say that if an entity has an immutable ID it must be provided, otherwise the common name is sufficient.

At the time the HACS integration was made “official”, I had proposed to Baloob that they consider a push protocol. He told me that it had to be pull. I see that now other energy monitors are using a push protocol. IoTaWatt has the ability to push data to influxDB, Emoncms, and PVoutput. There is an underlying infrastructure to facilitate pushing data.These connections have high integrity because they will first query to see where uploading was interrupted and backfill all missing data. I don’t believe any other energy monitor will do that. The HASS integration was user developed and uses the general purpose query facility to pull data.

I bring this up because the Emoncms uploader does have the notion of an ordinal number associated with each Script that is uploaded. Moreover, the way a push uploader works is that it only uploads what you tell it to upload, so the endpoint is not overwhelmed with every input and output that exists. At present I don’t believe HASS differentiates between outputs that are intended to provide Amps, VA, PF, Watts or Wh.

It would be a bigger project, but that would make some sense and provide data compliant with their unqualified rules.

UPDATE:

Upon reading the unique-ID requirement, I believe it has not been applied correctly in this case. When I look at their examples:

Clearly none of them are applicable to IoTaWatt outputs. They also describe this situation:

That is exactly what IoTaWatt outputs are. Setup by config entries. Perhaps we are arguing semantics. The exception allows for the Config Entry ID. That is the “name” in the IoTaWatt config. An excerpt from an IoTaWatt config:

“outputs”: [
{
“name”: “baseload”,
“units”: “Watts”,
“script”: “@1+@2-@3-@5-@6-@9”
},
{
“name”: “export”,
“units”: “Watts”,
“script”: “!-grid|”
},

Perhaps the ID could be the MAC address of the IoTaWatt (available) combined with the output name. I believe that satisfies the “rules” as I read them.

[falstaff]

@overeasy

Perhaps the ID could be the MAC address of the IoTaWatt (available) combined with the output name. I believe that satisfies the “rules” as I read them.

That is pretty much what the linked PR suggests. But the output name is an user configurable identifier.

Let me explain how I understand things: The HA requirements of unique ID aim to provide coherent behavior across services and devices. No matter if a “thing” is virtual or physical: The “thing” comes to live, the “thing” gets changed, the “thing” gets deleted. It all should behave seamlessly: The “thing” appears in HA, if changes are attributes visible in HA, they get transferred to HA, the “thing” gets deleted in HA.

For most physical things such unique ID appear naturally. For virtual things most often as well (usually a database primary key, or a UUID is used to identify a new record of a “thing”).

IoTaWatt outputs are also such a “thing”. The users creates with an intention (e.g. I want the sum of my 2nd floor power usage). So he clicks “add”, enter “Upper Floor”, enters the calculation. In Home Assistant he configures the output attributes such as Icons etc. Now he goes back, to IoTaWatt, presses “Edit”, changes the formula. That will work no matter what. Now he goes back and actually realizes he wants to name it “2nd Floor”. If we were to use the Name as unique ID, that last operation discards all settings in HA, since the Output now has a new “unique id”. That clearly wasn’t the intention of the user. He pressed “Edit” of that output, and just changed its name…

A technical inclined user might understand that the name has been used as identifier, and that therefor the settings for that output got lost.

But I think most users these days expect things to work “more” seamless as in, a “thing” stays the same thing on both sides, is “connected”, no matter what properties I change.

I don’t want to go into politics, I just want to point out where the requirements in HA come from. They help to ensure a certain user experience.

@overeasy

Existing outputs specifications are discarded, and new ones created, every time the configuration is processed.

I’ve looked at the POST requests when pressing save, and I see what you are saying. But in the end, this is how it works behind the scenes.

For me as an user, I just click on “Edit” of a particular output, and then change its properties…

@overeasy

Any attributes for outputs would need to be contained in that configuration, and user supplied. How is that any different than supplying a name?

Ok, but an id field could easily be part of this configuration and not user changeable (on the web frontend). E.g. “Add” could just increment the highest id currently present, and store it along the other attributes. “Edit” would just not change the existing id. Sure, that wouldn’t be a bullet prove method (as in, the user can fake POST), but that is not the requirement for such an id.

@overeasy

That is exactly what IoTaWatt outputs are. Setup by config entries. Perhaps we are arguing semantics. The exception allows for the Config Entry ID. That is the “name” in the IoTaWatt config. An excerpt from an IoTaWatt config:

“config entries” in a HA semantics is a thing created by the user through the HA web interface (“Add integration”). The rule allows to use the HA internal identifier if it gets created by the user in HA itself and the thing has no external identifier. When the IoTaWatt (device) config entry gets created it provides an unique ID for that already. The outputs do not get created through the HA interface (“setup by a config entry”), so we can’t use that rule.

[overeasy]

Thank you for your efforts. From the HASS documentation I see that there are fewer than 400 users of this integration. There are perhaps 10,000 IoTaWatt users worldwide. The only problem noted with the proposed MAC||name ID is that changing the name will unlink any history. That can be forewarned to HASS users and call it a day.

The work required and complexity involved is substantial, and the release protocol would require at least four months for any change to be mainstream. I have never pushed out a release on a shorter timeframe. I know the HASS folks throw a release over the wall every month. I don’t do that. I’ve got commercial and industrial customers and limited testing resources.

Of all the complaints I have fielded about the HASS integration, the early ones involved problems with getting accurate import and export numbers and I resolved that with a six month effort to add integrators which has eliminated the issue and I would argue provides more accurate data than can be accomplished with HA Energy alone. The most frequent current issue is that users cannot find the import and export outputs. I cannot recall anyone complaining about the aftereffects of changing an output name. I think this rule has created a solution looking for a problem.

If the PR is rejected, I think the best approach is to offer the integration via HACS. If you do that, I can add a section to the IoTaWatt documentation explaining the how and why of using it.