Reliable Option for Home Assistant History/Long-Term Statistics

1

I’m working on a project and would like to gauge community interest.

Briefly, it’s a Home Assistant App (add-on) that fixes many the common HA pain points:

  • Missing history during HA reboots or communication dropouts.
  • Inability to have use outputs as HA sensors.
  • Inability to see 5-second granularity data.
  • Inability to automatically import history and long-term statistics into HA.

In a nutshell, the app automatically configures the influx uploader to send the necessary buckets from all inputs (and outputs) at a 5-second post interval (or whatever post interval you like) starting at any available historical date you choose, then records that data for use within HA. It essentially reconstructs the IoTaWatt log fields in a local database. (This is trickier than it sounds.)

Once the uploader catches up, the data also populates the same W/WH and V/hZ sensors as the existing HA integration, as well as any outputs (including ones that use integrators) configured on the IoTaWatt and keeps those updated as sensors as well.

Any time the app restarts, recovers from a lapse in the uploader posts, or any time HA restarts, the app will attempt to fill in any missing history in HA.

The app supports multiple IoTaWatts. (I have 400A service, so I have 2 panels and 2 IoTaWatts.)

For the first version, the app acts as an IoTaWatt to MQTT bridge, which means you don’t need a separate integration if you already have MQTT. In the future I might develop a companion app or create a PR to allow the core integration to pull data from the app rather than directly from the IoTaWatt(s).

There’s more, but this is already a long post - so the rest is some background on how the project came to be.

The history of the IoTaWatt/HA integration is somewhat long and complex. There are fundamental incompatibilities between the HA and IoTaWatt approaches to handling data that can’t be resolved by an integration alone. HA sensors are always “immediate” - so you lose history during the time HA reboots or if there is a connectivity issue with the IoTaWatt. Coming the other way, there is no “unique ID” for IoTaWatt outputs, so they are not eligible to be a device entity in HA. Both of these have been fiercely argued in numerous forums, and the compromise is what we have today. Fundamentally, HA doesn’t consider itself a historian. It’s a recorder. The HA devs have added APIs to allow for modifying history and long-term statistics, but nothing in the core IoTaWatt integration that brings it full circle.

It’s frustrating because all of the granular data is faithfully recorded in the IoTaWatt log for a whole year, and less granular long-term statistics for an additional year - but the standard integration is useless for dealing with this.

There are other problems. It’s common to encounter HTTP exceptions in the HA logs whenever the integration times out on a query, which in my case is relatively common for one of my IoTaWatts that doesn’t have a perfect signal. The responses can be slow, creating a choppy sampling set. It’s also not possible to get 5-second granular data for inputs/outputs using the core integration. I’ve played with changing the integration to support this, and it’s a disaster with timeouts and seems to even cause spurious IoTaWatt reboots.

You can “fix” some of this using a combination of Ingress or EMonCMS and various tools and tricks to import the data into HA - nothing fully automatic and self-contained exists.

So, with some free time and an abundance of frustration with the limitations of the existing system, I started working on this project. I have been through several re-designs as I learned more about how both IoTaWatt and HA do things internally.

I’ve mostly wrangled it down to a well-behaved HA app (or docker container if you prefer) that uses very few resources, as little storage space as possible, and handles most of the corner cases I see with my system.

I still have some wrinkles to iron out before I would consider publishing, but if there’s enough interest, I will continue ironing and do the extra work to push to a public repository and publish as a Home Assistant app.

I’m also open to criticism and to answer questions if anyone is curious about the details.

1 Like
2

None of those are actually issues I have encountered.

  1. I use a UPS to power my network equipment, home assistant and iotawatt. After a (rare) home assistant restart or reboot all energy sensors are restored. There may be small gaps in the power sensor data but that has never been an issue for me.

  2. Outputs do create sensors, just not currently associated with the device. This may change due to recent device updates that are currently under beta testing. During testing of the latest core beta version it was found that the iotawatt integration generates a warning that device ids are required for the sensors. Something that the HA devs have refused to allow in the past. They are aware of this catch 22 but I have no idea how they will solve it. They have 12 months before it becomes a problem.

  3. 5 second data is not required. Integration happens in the iotawatt. At a much quicker rate than that then the energy total is presented to home assistant at a reasonable interval (10 sec). It would just be needless network traffic.

  4. Not sure what you are getting at here. All the sensors support long term statistics. If you mean for data stored in the iotawatt before it was connected to home assistant then that has never been a requirement for me.

It is not common to suffer HTTP timeouts either. That is a problem with your network you should fix.

However, having said all that I would be interested in checking out your app/add-on as long as it is not vibe coded.

3

Hi Wonko,

Thanks very much for working on this. I too have been frustrated by the immediate nature of HA inputs. I presently have my Iotawatt feeding the HA emoncms add-on, and then I access that emoncms data from HA.

I like the idea of an IoTaWatt to MQTT bridge, followed by an MQTT to HA integration that supports backfilling data. I find HA’s integration with MQTT very powerful, especially using HA autodiscovery features.

I like MQTT messages more than API calls, as MQTT has visibility for messages built in.

If your MQTT to HA integration has a provisioning / mapping interface that maps a device id or channel name or number to an HA identifier, that would be useful. Sometimes you want to move a device or change a channel name while keeping the same HA identifier (device replacement), other times you want to change the HA identifier for an unchanged device because the monitored area has changed occupancy or use. If these details can be changed and validated ahead of time, rather than exactly when the physical device is changed, that is helpful.

I’d be happy to try integrating with what you have, and see how the system could be generalized to support my use cases.

Thanks!