Another interesting problem has cropped up for me. I don’t check my Grafana (using InfluxDB) panels very often, maybe once a week or so. I have a Grafana panel that displays total WHrs that I compare with my monthly electric utility bill. They are usually just slightly off by less than 1%. My November bill said I used 1,566 KWHrs while Grafana said 1,582 KWHrs. I have done nothing to the Grafana panels in the last 2 or 3 months, but this month my utility bill said I used 1,911 KWHrs and Grafana’s reporting 3,856 KWHrs. When I check the data through graph+ in IotaWatt it says 1.93 MWHrs which is pretty close to the utility bill and coincidentally half of what’s in the Grafana panel. When I went back to check data for prior months (and I have 25 months worth of data now), the are all double what they were when I recorded them in a spreadsheet I update monthly. I tried restarting InfluxDB and Grafana but I still get what appears to be doubled values. Here’s a screen shot of my InfluxDB uploader screen from IotaWatt.
Thoughts anyone?
You are saying that data uploaded years ago is now being reported by Grafana at double previous values. Based on that I would say that the problem is with influx query or Grafana, not the IoTaWatt uploader which does not have the ability to upload data from more than one year ago.
What I’m saying is last month, Grafana reported data that basically matched my utility bill. This month, the data in Grafana is double my utility bill. Now, when I display the data for last month, it is also doubled (exactly doubled). After posting this, I actually went back and checked every month since I’ve been recording the data (starting Oct, 2020). It appears the doubling actually started Feb 2022. All of the data prior to that matches what I originally recorded in my spreadsheet. Again, my point is that last month, the numbers matched. This month’s numbers are double what they should be and last month’s are now doubled as well.
So not all 25 months’ worth, but about 10 months’ worth.
When I look at your IoTaWatt uploader setup, the upload history from date is Oct 1, 2022. Can you shed any light on why that was apparently changed in the past few months and what the circumstances were that precipitated that action?
I don’t see any doubled data in that thread. I see some obvious bad data on 10/1 and 11/1, that I think is probably attributable to a datalog issue, but nothing that takes it back to February. And after all, on Nov 2:
[quote=“dcjohnston54, post:6, topic:4926”]
Just checked the uploader and grafana. It appears the data uploaded successfully and is displaying in grafana as it should.
[/quote].
So, there is probably more to the story. I’d suggest looking at the data from March or April and see where the doubling is happening. Could it be that you added or changed tags such that the measurements no longer matched what was already in the database, then started an upload from February?
The doubling actually started in Feb 2022. But again, I would point out that everything appeared to be fine last month. According to the spreadsheet I’ve been keeping, all of the values I recorded basically matched my utility statement up until this month. Now, when I go back and look at the data, the values that matched when I recorded them are doubled (since Feb). The last changes I made to anything were done as a result of that thread I referenced above, and those changes were basically to the database, trying to clean things up. I deleted the data from the point that it was corrupt and changed the upload date to repopulate the database. I’ll take a closer look at the data tomorrow and try to determine exactly what date the doubling began and whether or not all of the fields are doubled or just the Tot-WHrs. I guess if push comes to shove, would I be able to delete the data from Feb 22 on, change the upload date to 31 Mar 2022 and then repopulate the database?
Can you post those doubled values? I could be wrong, but I don’t think influx accepts identical measurements.
This is for influxDB 2.6 Handle duplicate data points when writing to InfluxDB | InfluxDB OSS v2 Documentation
Here’s the influxDB 1.8 documentation
https://docs.influxdata.com/influxdb/v1.8/troubleshooting/frequently-asked-questions/#how-does-influxdb-handle-duplicate-points
You can look at it as the value is replaced by the same value in the new measurement. The net result is that nothing changes.
Well, hopefully this will make some sense, but I sure don’t understand what’s going on. I ran some inquires on the Influx database for a 1 minute time frame that appears to be doubled. Below, you should see a screenshot of that inquiry. I’ve also included a screen shot of a spreadsheet I made to compare the data from IotaWatt and InfluxDB. I’ve included the graph+ screen and a screen showing my grafana panel. Although the numbers are not exactly double, they are close (except for that first value), but there are no duplicate time values. Does any of this make sense?
| Date/Time |
Graph+
(Running Total) |
Graph+
(Data Points) |
InfluxDB |
| 12/22/2022 11:00:00 |
5.971 |
5.971 |
14.2446876 |
| 12/22/2022 11:00:05 |
14.245 |
8.274 |
16.5059949 |
| 12/22/2022 11:00:10 |
22.477 |
8.232 |
16.472522 |
| 12/22/2022 11:00:15 |
30.717 |
8.24 |
16.4366239 |
| 12/22/2022 11:00:20 |
38.913 |
8.196 |
16.4297575 |
| 12/22/2022 11:00:25 |
47.146 |
8.233 |
16.4727857 |
| 12/22/2022 11:00:30 |
55.385 |
8.239 |
16.4700304 |
| 12/22/2022 11:00:35 |
63.616 |
8.231 |
16.4179369 |
| 12/22/2022 11:00:40 |
71.803 |
8.187 |
16.3947329 |
| 12/22/2022 11:00:45 |
80.01 |
8.207 |
16.4304813 |
| 12/22/2022 11:00:50 |
88.233 |
8.223 |
16.4397429 |
| 12/22/2022 11:00:55 |
96.45 |
8.217 |
16.4501566 |
| Totals |
|
96.45 |
195.1654526 |
I just noticed something else interesting. After reviewing the Influx inquiry on the previous post referenced above, I see the data points are all 10 sec apart. On the Invflux inquiry above, they are only 5 sec apart. if you look at the uploader screen, it shows the post interval to be 10 sec. I added another column to my spreadsheet which may explain where the influx numbers are coming from. I added a column that almost exactly duplicates the numbers in the Influx database. If you add the first two entries in the Graph+ (Data Points) column (the 11:00:00 and 11:00:05 points) you get 14.245 which about the value in the InfluxDB column. If you continue that process on down the spreadsheet (11:00:05 + 11:00:10 as the next point) you get 16.506 which matches the InfluxDB column. Don’t know why this might be happening but those numbers are too close to be a coincidence.
| Date/Time |
Graph+
(Running Total) |
Graph+
(Data Points) |
InfluxDB |
|
| 12/22/2022 11:00:00 |
5.971 |
5.971 |
14.2446876 |
14.245 |
| 12/22/2022 11:00:05 |
14.245 |
8.274 |
16.5059949 |
16.506 |
| 12/22/2022 11:00:10 |
22.477 |
8.232 |
16.472522 |
16.472 |
| 12/22/2022 11:00:15 |
30.717 |
8.24 |
16.4366239 |
16.436 |
| 12/22/2022 11:00:20 |
38.913 |
8.196 |
16.4297575 |
16.429 |
| 12/22/2022 11:00:25 |
47.146 |
8.233 |
16.4727857 |
16.472 |
| 12/22/2022 11:00:30 |
55.385 |
8.239 |
16.4700304 |
16.47 |
| 12/22/2022 11:00:35 |
63.616 |
8.231 |
16.4179369 |
16.418 |
| 12/22/2022 11:00:40 |
71.803 |
8.187 |
16.3947329 |
16.394 |
| 12/22/2022 11:00:45 |
80.01 |
8.207 |
16.4304813 |
16.43 |
| 12/22/2022 11:00:50 |
88.233 |
8.223 |
16.4397429 |
16.44 |
| 12/22/2022 11:00:55 |
96.45 |
8.217 |
16.4501566 |
|
| Totals |
|
96.45 |
195.1654526 |
|
Yeah, I saw that and I was about to ask if you ever had your uploader set for 5 second intervals?
As far as I know, it’s never been anything but 10 sec. It’s also interesting that on 22 Dec, I was still getting the 5 sec interval, but on 23 Dec it changed back to the 10 sec interval. I could get an exact time that it changed if that would be helpful.
I’m more interested in when it changed from 10 seconds to 5 seconds, probably back in February when the doubling reportedly started. I would want to see the message log covering that transition as well.
What you may find is that it was actually 5 second before the doubling started but that the vaulues are 5 second values and not 10 second values as you have now.
Here’s a screen shot of exactly when that 10 sec to 5 sec transition occurred. It appears it was in Dec 2021. How do I get the message log for that? or is it too far back??
After looking at my spreadsheet again to figure out why I originally said Feb 22, I discovered that 16 Feb is the bill date on the spreadsheet. The data for that bill date is actually 20 Dec 2021 - 21 Jan 2022. So the above data agrees with with that date range. I would also note that previous to the 5 sec interval occurrence, the seconds count is 0 10 20 30… The latest data in the file now shows it to be 5 15 25 35… Not sure if that’s of any importance but it seems a bit odd.
You can download the whole thing using the file manager. /iotawatt/iotamsgs.txt. Upload it to a me in a PM and I’ll locate it. I think I understand what happened, but the log will help me understand how it happened.
Just sent that text file. Hopefully it’s what you’re looking for.
That’s what I was looking for, but unfortunately it’s truncated at 12/12/2022. The download must have timed out. Looking at what I got I don’t see anything that could have caused the 5 second entries. I’ve looked at the upload code and don’t see how that can happen, so I need to get te rest of the log.
When you look at the message log through the browser app, it simply downloads the last 10,000 characters of the file using a URL query parameter:
http://iotawatt.local/iotawatt/iotamsgs.txt?textpos=-10000
You can then change the URL to indicate that you want to go further back. So you can go back 100000 characters with the URL
http://iotawatt.local/iotawatt/iotamsgs.txt?textpos=-100000
Can you give that a try and see if you can get back to 12/22/2022? Then post it in a PM. Email again is OK or you can click my emoji above and then click “Message” to send a Private Message where you can upload the file remnant.
I’m looking at the txt file right now. I see entries from when I installed my IotaWatt back in 2020. It looks like it starts 18 Aug 2020. There are 6614 lines of text. I can cut out any portion you would like and resend it. I assume you meant 12.22.2021 in your post. I’ll create a file with Oct 2021 through Feb 2022 and send that to you.
I restarted the updater and here’s the last few lines in the log now:
12/12/22 12:12:51 influxDB_v1: Start posting at 12/12/22 12:12:20
12/29/22 23:04:07 influxDB_v1: stopped, Last post 12/29/22 23:03:05
12/29/22 23:04:14 influxDB_v1: Start posting at 12/29/22 23:03:10
Here’s the Status page:
