@frogmore Are you able to share your influx config? I’m also looking to set that up?
Thanks
Look here for a start:
Well, the plot thickens. I noticed that the uploading progress stalled again. Sure enough Grafana showed an annomoly at 11:10AM on 10/30. So I stopped the sending (which had restarted back at the beginning of the datalog and not at the point that I told it to start at). I set the start date a few days ahead and the upload is continuing. I ran some queries and this is what I see in the data:
> select * from iotawatt where Watts < 0
name: iotawatt
time Amps Volts Watts channel device
---- ---- ----- ----- ------- ------
1592858730000000000 -2636 Barn IotaWatt
1592858730000000000 -1.6535469126360393e+51 Utility3 IotaWatt
1594426420000000000 -43.18 HPWH_A IotaWatt
1594426420000000000 -70208482881.11 Utility1 IotaWatt
1599354900000000000 -5.760780923401527e+164 Family IotaWatB
1599354900000000000 -1.894185814057058e+280 k15 IotaWatB
1600551490000000000 -649.18 Bed23 IotaWatt
1600551490000000000 -93103324898650900000 k13 IotaWatt
1600551490000000000 -29.48 WH_Well_A IotaWatt
1600551490000000000 -63.59 Utility3 IotaWatt
1600551490000000000 -29.95 RangeVent IotaWatt
1600551490000000000 -23.32 HPWH_A IotaWatt
1600551490000000000 -639.95 Dryer_A IotaWatt
1602377000000000000 -5.439967369313442e+164 Family IotaWatB
1602377000000000000 -1.7887000316273969e+280 k15 IotaWatB
1604081460000000000 -188951526.75 MB_Floor IotaWatt
1604081470000000000 -1.3228952450799069e+51 Utility3 IotaWatt
1604081470000000000 -2093.83 Barn IotaWatt
> select * from iotawatt where Watts > 10000
name: iotawatt
time Amps Volts Watts channel device
---- ---- ----- ----- ------- ------
1592858730000000000 5.621438149643385e+227 Dryer_A IotaWatt
1594426420000000000 779295162956.35 MB_Floor IotaWatt
1594426420000000000 1.025749877095174e+114 k13 IotaWatt
1594426420000000000 27401210191.42 Washer IotaWatt
1599354900000000000 1.2710192031453963e+73 BarnB IotaWatB
1602377000000000000 1.2002371002851973e+73 BarnB IotaWatB
1604081460000000000 117286850979.92 Dryer_A IotaWatt
1604081460000000000 4.603 23813.22 MainA IotaWatt
1604081460000000000 8.991434343761321e+85 HPWH_A IotaWatt
1604081470000000000 4.49734672892881e+227 Dryer_A IotaWatt
1604081470000000000 1.6003708857643472e+79 HPWH_A IotaWatt
>
IotaWatt is LegA and IotaWatB (8 char limit on the name in IotaWatt) is LegB. This time it shows some clearly incorrect values from IotaWatB. This one has never given me a problem. There are two timestamps that have an issue on Leg B:
1599354900000000000 Saturday, September 5, 2020 6:15:00 PM [GMT-07:00] DST
1602377000000000000 Saturday, October 10, 2020 5:43:20 PM [GMT-07:00] DST
So, it looks like there is something going on. The Sept 5 one is a little before the 6 week automatic restart, but that is probably just a coincidence. The Oct 10 one does have anything interesting around that time, but the next day it did update the FW.
I think I will be done uploading all the data to influxdb in the next day or so. After that, I will rearrange things and after that delete the data and history logs on both devices. I am seeing the following issues:
- Bad data or bad measurements are either being written to the datalog, or the datalog is getting corrupted.
- When uploading data to influxdb, if it hits bad data, it restarts the upload process
- When it restarts the upload process it restarts from the beginning of the datalog, and not the specified start date
- It doesn’t log anything in the message log when it restarts the upload process
It seems the error does not hit all channels at the same time. It is possible that there are more errors. I just searched for obviously wrong values. I don’t have solar, so I left all the channels at the default (allow negative power is unchecked). This means all the negative values are definitely wrong.
Many of the timestamps occurred with nothing interesting around them, so not clear what caused them. Could be a bug, but I doubt this one is going to be easy to figure out.
This is a little unsettling, but I don’t there is enough here to support your conclusions. I have some questions that I’d like to resolve.
What version of influx are you using? The reason I ask is that I don’t know how I could post -5.760780923401527e+164 if I wanted to. Maybe C++ is taking liberties with printf, but I’ll need to see that.
I’m starting to suspect there may be a bug in the datalog search algorithm that is returning the wrong data. It’s definitely a possibility, although this would be the first incident noticed. There is no index for the datalog. It is chronological and so can be searched pretty efficiently. But perhaps there is a boundary condition…
The restarting at the beginning is a separate issue. That’s why I’m asking about the influx version. I’m wondering if there is a translation between the V1 query and flux, and if that might be working improperly because a flux query is being created with too short a look-back window.
There’s a lot going on here, and with a newer version of influx, it may be we’re looking at compound issues. I definitely want to look at the IoTaWatt data from the other machine. I’ll post the date/times and what I’d like to see.
I agree that it is unsettling and I am committed to helping figure out what happened. So I will do what I can to get you the information that you need. I haven’t looked at the data structures used in the log so can’t really explain what might have happened. I also agree that there is more than one issue here, which makes it complicated to figure out what is going on.
So, first question. I am using Influx 1.8.3,
https://dl.influxdata.com/influxdb/releases/influxdb_1.8.3_amd64.deb
I am running this on Ubuntu 20.04 in an LXC (container)
The datalog is showing strange numbers in the graphs for the sample points I looked at. But, tell me what you want to see and I will check it out.
Good news, IotaWatB completed the catch up and got current yesterday. IotaWatt got current sometime in the evening and now I have lots of data in influxdb.