InfluxDB: Adding >67 measurements causes config parse failure/invalid config

Support
1

Good evening,

I’m trying to set up the InfluxDB v1 data uploader.
Being the OCD hoarder that I am, I’m trying to upload everything possible about every circuit the IoTaWatt is monitoring.
I’ve noticed that if I try and add more than 66 measurement items, then on save I get the following config parse error in the logs.
Interestingly, on a reboot it claims that it is uploading again, but if I try and edit the uploader and safe I go back to config parse error and stopped again.

7/11/21 18:49:58z influxDB_v1: Config parse failed
7/11/21 18:49:58z influxDB_v1: Invalid configuration.

Deleting any measurement seems to stop the parsing error.

Firmware version: 02_06_02. Happy to provide full config or anything else that’s useful.

Edit: it looks like the reason the uploader starts again on the reboot is that it’s reverted the config rather than it magically working :frowning:

2

There is a limit to how much JSON the ESP8266 can digest with the limited heap available. The firmware already breaks down the config into smaller pieces for that purpose, but 60+ scripts with associated literal strings is apparently too much.

One approach to these kinds of “problems” might be to simply limit the number of things you can do. Limited number of uploaders, limited number of measurements, limited name sizes etc. The approach taken is to simply let you push the limits to the edge and fail as gracefully as possible, allowing you to recognize the limitations and back-off.

The device has 15 inputs. There are three basic measurements for each - Volts, Amps and Watts. Thatr’s all IoTaWatt stores in its datalog, and that’s where everything you upload comes from. So uploading more than three measurements is redundant, and if you are single phase, you can just upload the one voltage measurement and only two per input. Everything else that IoTaWatt can upload can be calculated from those measurements. There’s no justification for heroics to handle 60+ measurements.

3

Really appreciate the response; thank you.
I should point out I’m extremely novice; both with power monitoring and influx, so I took the approach of “record all the things and deal with it later”.

Reference Voltage and Frequency seems to be provided by the 9v transformer.

For the individual inputs am I correct in reading “voltage” as the “VA” reading?

This is probably another stupid question, but is the internally calculated power usage (kwh or wh) likely to be different or more accurate that what I could calculate afterwards with the reported wattage values?

Greatest of thanks - both for the reply and for the unit. It’s flippin fantastic! Actually; how much internal storage does the IoTaWatt have? I’m only really setting up influx as I’m worried it’ll fill up and I’ll loose data.

4

That’s pretty common. Also wanting to monitor everything in the panel is pretty common for the uninitiated. Most folks change their approach once they get a chance to see and think about the data.

My feeling is that there are several different activities requiring data:

Looking at individual appliances, their duty cycles and power profiles is something that requires fairly high resolution data over a relatively recent short period of time. It’s also something that you typically do once or at best occasionally. For that, Graph+ directly from the IoTaWatt current log with 5 second data works great.

The other activity is looking at long term usage as billed, seasonal trends in appliances like HVAC, and demand trends for capacity planning for things like generators. For that, one minute data is fine. Any time period greater than 12 hours will use 1 minute data with graph+.

Another activity is solar generation, consumption and net usage. Usually that’s analyzed a day at a time, so 2 minute data at best, You can do it with Graph+ but PVoutput is a very good free tool for that and it uses 5 minute data.

So it may sound crazy at first, but my recommendation is to upload one minute data to influx or Emoncms, and use Graph+ directly on the IoTaWatt for high resolution analysis. Most folks don’t use high resolution historical data, and shrinking the dataset makes it more manageable for uploading, backup and query.

No, the voltage is taken from the voltage channel associated with the CT channel. For most single-phase users that’s channel zero. VA is just that - Volts x Amps.

Slightly different, just as accurate for all intents and purposes. The IoTaWatt actually doesn’t record Watts. It records cumulative Wh since inception. To get Watts, it divides the cumulative Wh by the cumulative Hours measured. When you upload Watts to influx, it integrates for Wh by doing the reverse - Multiple Watts by Hours in the group (interval). So there can be some precision and rounding effects from the division and decimal digits sent. It doesn’t amount to much significance in the big picture, but it is different.

I assume by context that you mean how much external storage does it have on the SDcard. Newer units have 8 GB. The current log holds up to a year of 5 second resolution data and uses about 1.6 GB. The history log holds up to 10 years of 1 minute resolution data and uses about 1.3GB after ten years. The size of each is displayed in the Data Logs status display.

2 Likes
5

FWIW, one use not mentioned is reconciling data between multiple IoTaWatts. I have a main panel that feeds A/C units, two sub panels, and PV ties in there, and an EV Charger that is on one of the sub-panels. I want to track how much of my EV charging is self-consumed, and I might want to track it for my clothes dryer as well.

I ended up going with Influx so the math could be done there— I can record a daily metric of energy from grid vs from PV. As I try to maximize self-consumption (near) real time, I also like having this in Watts.

There are a few visualization quirks that external tools are helpful for as well, such as using a different sampling interval for energy and power.

(All that said, exporting much more than 20 points per unit does seem excessive. For residential applications tracking apparent power (VA) is likely overkill for anything but your main.)