InfluxDb issues

Sounds like you have your issues under control. Good luck. For the time clause you could make it relative with
where time > now() - 2d

Hi Since last update 02_03_17 one of the 3 iotawatt has an issue with influxDb.

I tried multiple things:

  1. edit the webServer and set no date
  2. edit the server and set the 23.10.2018 (day where the update occurred)
  3. trigger reset…

None of those helped it looks like having problems to recover last entry.
SOmetimes it is hable to start but from july and it stops after a while…

Log:
** Restart **

SD initialized.
10/25/18 04:39:23z Real Time Clock is running. Unix time 1540442363
10/25/18 04:39:23z Version 02_03_17
10/25/18 04:39:23z Reset reason: Software/System restart
10/25/18 04:39:23z Trace: 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:4, 18:5, 1:6, 1:1[10], 1:2[11], 9:0[11], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:5[21]
10/25/18 04:39:23z ESP8266 ChipID: 6910350
10/25/18 04:39:23z SPIFFS mounted.
10/25/18 06:39:24 Local time zone: 2
10/25/18 06:39:24 device name: pinguIW1, version: 3
10/25/18 06:39:27 MDNS responder started for hostname pinguIW1
10/25/18 06:39:27 LLMNR responder started for hostname pinguIW1
10/25/18 06:39:27 HTTP server started
10/25/18 06:39:27 timeSync: service started.
10/25/18 06:39:27 statService: started.
10/25/18 06:39:27 WiFi connected. SSID=pingu, IP=192.168.18.73, channel=1, RSSI -74db
10/25/18 06:39:28 Updater: service started. Auto-update class is ALPHA
10/25/18 06:39:28 dataLog: service started.
10/25/18 06:39:28 dataLog: Last log entry 10/25/18 06:39:20
10/25/18 06:39:28 historyLog: service started.
10/25/18 06:39:29 historyLog: Last log entry 10/25/18 06:39:00
10/25/18 06:39:30 Updater: Auto-update is current for class ALPHA.
10/25/18 06:39:32 EmonService: started. url:80=emoncms.org, node=pinguITW, interval=10
10/25/18 06:39:32 influxDB: started, url=192.168.18.10:8086, db=itw02, interval=5
10/25/18 06:39:33 EmonService: Start posting at 10/24/18 16:13:20
10/25/18 06:39:33 EmonService: Stopped. Last post 10/24/18 16:13:10
10/25/18 06:39:39 influxDB: last entry query failed: -11
10/25/18 06:39:39 influxDB: Stopped. Last post 2/6/6 08:28:16

Updated Log:
** Restart **

SD initialized.
10/25/18 06:22:27z Real Time Clock is running. Unix time 1540448547
10/25/18 06:22:27z Version 02_03_17
10/25/18 06:22:27z Reset reason: Software/System restart
10/25/18 06:22:27z Trace: 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:1, 1:2[1], 9:0[1], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 10:2, 10:3
10/25/18 06:22:27z ESP8266 ChipID: 6910350
10/25/18 06:22:27z SPIFFS mounted.
10/25/18 08:22:28 Local time zone: 2
10/25/18 08:22:28 device name: pinguIW1, version: 3
10/25/18 08:22:31 MDNS responder started for hostname pinguIW1
10/25/18 08:22:31 LLMNR responder started for hostname pinguIW1
10/25/18 08:22:31 HTTP server started
10/25/18 08:22:31 timeSync: service started.
10/25/18 08:22:31 statService: started.
10/25/18 08:22:31 WiFi connected. SSID=pingu, IP=192.168.18.73, channel=1, RSSI -74db
10/25/18 08:22:31 Updater: service started. Auto-update class is ALPHA
10/25/18 08:22:31 dataLog: service started.
10/25/18 08:22:32 dataLog: Last log entry 10/25/18 08:22:25
10/25/18 08:22:32 historyLog: service started.
10/25/18 08:22:32 historyLog: Last log entry 10/25/18 08:22:00
10/25/18 08:22:34 Updater: Auto-update is current for class ALPHA.
10/25/18 08:22:36 EmonService: started. url:80=emoncms.org, node=pinguITW, interval=10
10/25/18 08:22:36 influxDB: started, url=192.168.18.10:8086, db=itw02, interval=5
10/25/18 08:22:37 EmonService: Start posting at 10/24/18 16:13:20
10/25/18 08:22:37 EmonService: Stopped. Last post 10/24/18 16:13:10
10/25/18 08:22:42 influxDB: last entry query failed: -11
10/25/18 08:22:42 influxDB: Stopped. Last post 2/6/6 08:28:16
10/25/18 08:22:47 influxDB: started, url=192.168.18.10:8086, db=itw02, interval=5
10/25/18 08:22:54 influxDB: last entry query failed: -11
10/25/18 08:22:54 influxDB: Stopped. Last post 2/6/6 08:28:16
10/25/18 08:22:56 influxDB: started, url=192.168.18.10:8086, db=itw02, interval=5
10/25/18 08:23:02 influxDB: Start posting from 10/23/18 21:51:00
10/25/18 08:23:02 influxDB: Stopped. Last post 10/23/18 21:50:55
10/25/18 08:23:19 influxDB: started, url=192.168.18.10:8086, db=itw02, interval=5
10/25/18 08:23:21 influxDB: Start posting from 10/23/18 21:51:00
10/25/18 08:23:21 influxDB: Stopped. Last post 10/23/18 21:50:55
10/25/18 08:23:26 influxDB: started, url=192.168.18.10:8086, db=itw02, interval=5
10/25/18 08:23:27 influxDB: Start posting from 10/23/18 21:51:00
10/25/18 08:42:05 Incomplete HTTP request detected, id 7, restarting.

** Restart **

SD initialized.
10/25/18 06:42:07z Real Time Clock is running. Unix time 1540449727
10/25/18 06:42:07z Version 02_03_17
10/25/18 06:42:07z Reset reason: Software/System restart
10/25/18 06:42:07z Trace: 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:4, 18:5, 1:6, 1:3, 1:4, 1:5[21]
10/25/18 06:42:07z ESP8266 ChipID: 6910350
10/25/18 06:42:07z SPIFFS mounted.
10/25/18 08:42:08 Local time zone: 2
10/25/18 08:42:08 device name: pinguIW1, version: 3
10/25/18 08:42:11 Connecting with WiFiManager.
10/25/18 08:42:15 MDNS responder started for hostname pinguIW1
10/25/18 08:42:15 LLMNR responder started for hostname pinguIW1
10/25/18 08:42:15 HTTP server started
10/25/18 08:42:15 timeSync: service started.
10/25/18 08:42:16 statService: started.
10/25/18 08:42:16 WiFi connected. SSID=pingu, IP=192.168.18.73, channel=1, RSSI -74db
10/25/18 08:42:16 Updater: service started. Auto-update class is ALPHA
10/25/18 08:42:16 dataLog: service started.
10/25/18 08:42:17 dataLog: Last log entry 10/25/18 08:42:05
10/25/18 08:42:17 historyLog: service started.
10/25/18 08:42:17 historyLog: Last log entry 10/25/18 08:42:00
10/25/18 08:42:19 Updater: Auto-update is current for class ALPHA.
10/25/18 08:42:20 EmonService: started. url:80=emoncms.org, node=pinguITW, interval=10
10/25/18 08:42:20 influxDB: started, url=192.168.18.10:8086, db=itw02, interval=5
10/25/18 08:42:21 EmonService: Start posting at 10/24/18 16:13:20
10/25/18 08:42:21 EmonService: Stopped. Last post 10/24/18 16:13:10
10/25/18 08:42:27 influxDB: last entry query failed: -11
10/25/18 08:42:27 influxDB: Stopped. Last post 2/6/6 08:28:16

There appear to be two issues here. Timeout of the last post query, and failure to post with the service stopping. I can’t see anything that changed in this release that would cause either problem, so I’d like more information and try something.

First, could you post the log portion from prior to the update to 17 until after the update. That would be at least three full restart cycles.

Next, could you change to auto-update MINOR (02_03_16) it should change right away, and post the log from the update and after.

If influx works fine on that, could you go back to ALPHA and post the logs again?

With version 02_03_16 it is able to recover (even if I have to make some start and stop)
There the 2 logs
20181025_Log1.txt (68.3 KB)
20181025_Log2.txt (9.7 KB)

It does look as though the posting stopped with the upgrade to 02_03_16, although nothing changed in that area, there are no other reports of influx failing after upgrade, and my systems took it in stride.

One thing unique about your usage is the 5 second interval, but that shouldn’t cause this. could you post your influx configuration for this system?

I’d recommend you stay on 02_03_16.

UPDATE: I found the problem. You should stay on 02_03_16 until a new release with a fix.

1 Like

@daniweb

I’ve fixed that problem, and also changed the query for last post time to use multiple individual queries, one for each measurement, using the last() function. Hopefully, this will solve your timeout issue.

This is a not a trivial change. I have installed it on all of my units without incident, but I’m reluctant to publish a release until I have a few days experience with it. I’m also interested in seeing if it solves your problem. The updates are merged into master in the git. If you have the ability to compile and load I’d be interested in the result. Although not critical, you should also download and install index.htm from the git.

Let me know if you need a binary.

I’m not at home, and it will be difficult for me to find time before Tuesday.
Maybe better to get a binary. Do I have to upload it via the file browser of the iotawatt?

I’ve put it into a development release. It may be ALPHA by Tuesday, if not, I can instruct you how to download a non-standard release.

I have the 02_03_18 since 6h and ok for the moment, but I have to admint that I did not try to play to restart the webserver with a different date.

As impression I think that with the change from the 02_03_16 we have 3K or 4K of additional heep free.

Just by virtue of installing the new release, you have restarted influx. I am hopeful that the log does not show timeouts in the last-post query process and that the restart did recognize the correct time. In your case, it may take a minute or more because of the high number of measurements that must be queried.

I expect that if you provide a begin-date that is greater than the last post date, it will sjip ahead to that. If you provide a begin-date for a new first key value it should begin uploading from 00:00:05 on that date.

There a reboot occurred:

** Restart **

SD initialized.
10/30/18 08:18:06z Real Time Clock is running. Unix time 1540887486
10/30/18 08:18:06z Version 02_03_16
10/30/18 08:18:06z Reset reason: Software/System restart
10/30/18 08:18:06z Trace: 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 1:4, 1:3, 10:2, 10:3
10/30/18 08:18:06z ESP8266 ChipID: 6910350
10/30/18 08:18:06z SPIFFS mounted.
10/30/18 10:18:07 Local time zone: 2
10/30/18 10:18:07 device name: pinguIW1, version: 3
10/30/18 10:18:10 MDNS responder started
10/30/18 10:18:10 You can now connect to http://pinguIW1.local
10/30/18 10:18:10 HTTP server started
10/30/18 10:18:10 timeSync: service started.
10/30/18 10:18:10 statService: started.
10/30/18 10:18:10 WiFi connected. SSID pingu, IP 192.168.18.73, channel 1, RSSI -76db
10/30/18 10:18:10 Updater: service started. Auto-update class is ALPHA
10/30/18 10:18:10 dataLog: service started.
10/30/18 10:18:11 dataLog: Last log entry 10/30/18 10:18:00
10/30/18 10:18:11 historyLog: service started.
10/30/18 10:18:11 historyLog: Last log entry 10/30/18 10:18:00
10/30/18 10:18:13 Updater: Update from 02_03_16 to 02_03_18
10/30/18 10:18:13 Updater: download 02_03_18
10/30/18 10:18:40 Updater: Release downloaded 27068ms, size 638232
10/30/18 10:18:40 EmonService: started. url:emoncms.org:80,node:pinguITW,interval:10, unsecure GET
10/30/18 10:18:40 influxDB: started.
10/30/18 10:18:52 Updater: Update downloaded and signature verified
10/30/18 10:18:58 Updater: firmware upgraded to version 02_03_18
10/30/18 10:18:59 Firmware updated, restarting.

** Restart **

SD initialized.
10/30/18 08:19:06z Real Time Clock is running. Unix time 1540887546
10/30/18 08:19:06z Version 02_03_18
10/30/18 08:19:06z Updater: Installing update files for version 02_03_18
10/30/18 08:19:06z Updater: Installing GRAPH.HTM
10/30/18 08:19:06z Updater: Installing GRAPH.JS
10/30/18 08:19:06z Updater: Installing INDEX.HTM
10/30/18 08:19:07z Updater: Installing TABLES.TXT
10/30/18 08:19:07z Updater: Installing CNFSTYLE.CSS
10/30/18 08:19:07z Updater: Installing EDIT.HTM
10/30/18 08:19:08z Updater: Installation complete.
10/30/18 08:19:08z Reset reason: Software/System restart
10/30/18 08:19:08z Trace: 1:2[6], 9:0[6], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:5[19], 1:6, 1:1[6], 1:2[7], 9:0[7], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:5[5]
10/30/18 08:19:08z ESP8266 ChipID: 6910350
10/30/18 08:19:08z SPIFFS mounted.
10/30/18 10:19:09 Local time zone: 2
10/30/18 10:19:09 device name: pinguIW1, version: 3
10/30/18 10:19:10 MDNS responder started for hostname pinguIW1
10/30/18 10:19:10 LLMNR responder started for hostname pinguIW1
10/30/18 10:19:10 HTTP server started
10/30/18 10:19:10 timeSync: service started.
10/30/18 10:19:10 statService: started.
10/30/18 10:19:10 WiFi connected. SSID=pingu, IP=192.168.18.73, channel=1, RSSI -74db
10/30/18 10:19:10 Updater: service started. Auto-update class is ALPHA
10/30/18 10:19:10 dataLog: service started.
10/30/18 10:19:11 dataLog: Last log entry 10/30/18 10:18:30
10/30/18 10:19:11 historyLog: service started.
10/30/18 10:19:11 historyLog: Last log entry 10/30/18 10:18:00
10/30/18 10:19:13 Updater: Auto-update is current for class ALPHA.
10/30/18 10:19:15 EmonService: started. url:80=emoncms.org, node=pinguITW, interval=10
10/30/18 10:19:15 influxDB: started, url=192.168.18.10:8086, db=itw02, interval=5
10/30/18 10:19:16 EmonService: Start posting at 10/30/18 10:18:10
10/30/18 10:19:30 influxDB: Start posting at 10/30/18 10:18:05

** Restart **

SD initialized.
10/31/18 21:22:34z Real Time Clock is running. Unix time 1541020954
10/31/18 21:22:34z Version 02_03_18
10/31/18 21:22:34z Reset reason: Hardware Watchdog
10/31/18 21:22:34z Trace: 18:0, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:4
10/31/18 21:22:34z ESP8266 ChipID: 6910350
10/31/18 21:22:34z SPIFFS mounted.
10/31/18 23:22:35 Local time zone: 2
10/31/18 23:22:35 device name: pinguIW1, version: 3
10/31/18 23:22:35 MDNS responder started for hostname pinguIW1
10/31/18 23:22:35 LLMNR responder started for hostname pinguIW1
10/31/18 23:22:35 HTTP server started
10/31/18 23:22:35 timeSync: service started.
10/31/18 23:22:36 statService: started.
10/31/18 23:22:36 WiFi connected. SSID=pingu, IP=192.168.18.73, channel=1, RSSI -74db
10/31/18 23:22:37 Updater: service started. Auto-update class is ALPHA
10/31/18 23:22:37 dataLog: service started.
10/31/18 23:22:37 dataLog: Last log entry 10/31/18 23:22:20
10/31/18 23:22:37 historyLog: service started.
10/31/18 23:22:38 historyLog: Last log entry 10/31/18 23:22:00
10/31/18 23:22:39 Updater: Auto-update is current for class ALPHA.
10/31/18 23:22:40 EmonService: started. url:80=emoncms.org, node=pinguITW, interval=10
10/31/18 23:22:40 influxDB: started, url=192.168.18.10:8086, db=itw02, interval=5
10/31/18 23:22:41 EmonService: Start posting at 10/31/18 23:22:30
10/31/18 23:22:53 influxDB: Start posting at 10/31/18 23:22:25

Hardware watchdog is a tough one to sort out. The software watchdog has a shorter interval and usually is the one that catches runaway loops and such. Also, the piece of code that the trace reveals doesn’t have any explicit loop, although there may be an implicit one in a call to get the heap size.

More likely this has something to do with WiFi and TCP. I believe some of that stuff operates below the software watchdog purview and is more subject to hardware WDT. The recovery was quick and nothing seems to have been lost, which is what the multi-layer fail-safe design is supposed to do. So keep an eye on it. If it happens a lot, let me know.

PS: It looks like your influx last post issues are gone. It does take up to 15 seconds to query all those measurements, but there’s no real downside to that.

Good day.
I have installed 4 IoTaWatts in 4 different electrical boards. The IotaWatt 1 and 2 are installed in a
building and the IotaWatt 3 and 4 are in another building.
The 4 IotaWatts monitor three-phase lines and the data is being sent to
a local InfluxDB server, which is in another building.

The 4 IotaWatts have the same firmware version 02_03_20.
Iotawatt 1 and 2 was working very well since 2018. On Thursday, January 31 of this year,
They stopped sending data to InfluxDB. The Start / stop button found in the Status section, Influxdb,
has the legend start. I press it with the mouse to change its state to stop, but it does not respond or changes during 2s and then return to start.

I was reading in this forum the possible causes of the failure and the users suggested several actions:

  • reset the influxdb. Yes we did it.
  • increase the posting time> 5s. It was not done.
  • shorten the name of the variables reported by IotaWatt. It was not done.
  • do not calculate the current from the division VA / V, because divisions can be given between 0 and cause an unexpected result. The current is not reported to InfluxDB.

On Wednesday, February 13, I went to the building where the IotaWatt 1 and 2 are. I Disconnected the access point from the power source and the IotaWatt1. Then I went to the building where my PC is and in the chrome I accessed the IotaWatt 1 and 2. The IotaWatt 1 until today it is working well send data to InfluxDB. The IotaWatt 2 that does not disconnect from the power supply remains in the same state,
does not send data to InfluxDB.

The question is, do you have any solution to solve this problem?

There’s something missing here. How do you access these IoTaWatt if the AP is disconnected?

This is the first mention of disconnecting from the power supply. Are you saying that the IoTaWatt disconnects? When? Why?

The first step is to better define the problem:

Are units 3 and 4 working OK?
Do you have the influx configuration and message logs for units 1 and 2 covering the period from January 31 to now?

I do prefer to work on problems using the most current release, which is 02_03_21 which has improvements to influxDB error handling. Is there a good reason why you guys don’t subscribe to an auto-update class? I can work on these problems all day but it doesn’t help if users don’t allow me to deliver the results of that effort and continue to report problems that may have already been addressed.

You are right. I just write “Disconnected” instead “Disconnected/Connected”.

Iotawattt 1 and 2 are in the same building, they are using the same access point. The Iotawatt 2 has the bug (no posting to influxDB), it was not disconneted/conneted from the power supply as Iotawatt1 did.

They are not posting to InfluxDB.

[quote=“overeasy, post:55, topic:402”]
Do you have the influx configuration and message logs for units 1 and 2 covering the period from
[/quote]Iota2_20180129to20190210.txt (9.5 KB)

YesIota1_20180129to20190201.txt (9.9 KB)

Agree. I will update the 4 IoTaWatts.

Here are the InfluxDB config files.

I can see that leading up to the Jan 30 problem, the WiFi was not working. The WiFi appears to have come back sometime around 10:40 on the 29th. One system was able to query and restart, the other was not. As of 2/1 when the logs end, neither was connected to influx.

These timeout failures on restart are exactly what was fixed in 02_03_21. Now, rather than stop, the influx service continues to retry the query indefinitely. You may have other problems, but this one should go away with the upgrade.

When you say that units 3 and 4 are not posting to influxDB, do you mean that they are not configured to post or that they are configured and its not working?

They were configured and were not posting to InfluxDB. But the FW was updated to both of them they already have 02_03_21 version. They started to post to InfluxDB. Until now they are working well.

Hi Overeasy.
As already you know, in my work there are 3 Iotawatts installed. Iota1, Iota2 and Iota4. Iota1 and Iota2 stops sending data to InfluxDB, since last march 12th. Here is the log file of Iota1, I guess both Iotas have the same failure.
Iota1_log_20190315.txt (28.0 KB)