InfluxDB upload fails (code 400) after working fine for a few hours - resolved

So, my influx v1 upload suddenly failed after working fine for a few hours. From my Grafana graph you can see that this happened at about 20:48 local time.

image

Error code and firmware info:

I’ve looked at the other similar issues on the forum, including $device as tag value (does not apply to me), using name tag (also does not apply to me).

See influx v1 tab below.

The strange thing is that it worked fine for a few hours, then suddenly fails. Like there is some aggregation that suddenly reaches a threshold and breaks the upload.

I believe the - in the measurement “iotaWatt-measurement” is causing the problem. Removing the - and replacing with an underscore ‘_’ should resolve this problem.

You might want to rethink the measurement name anyway as it’s very long and is part of each line of every write every 5 seconds. In general the payload for all of that data is quite long by virtue of the long measurement name as well as the long key-data fields represented by $name. It should work fine, but will take a longer time to catch up if there is a long interruption.

Thanks for the quick reply. It does not seem to solve it.

Did a hard restart (pulling 5v power)

I was able to recreate the problem using the latest release 02_07_05 and changing the ‘-’ to ‘_’ resolved it. There are other changes to the influx restart query between your release 02_06_06 and the most recent, so I will need to setup that release and try to recreate.

Alternatively, you could try setting your auto-update to BETA to update to the latest release and see if it resolves. Otherwise, I will look into this over the next few days.

UPDATE: Just noticed that the message is now POST failed as opposed to the previous query failed. This begs the question how you were able to run for a few hours as previously stated. This failed right away with the new resultant series. Could you post your message log so I can see what happened?

Updated to BETA. Same issue

As for the change in error message to POST. I did try various things (removing tag-keys, changing names, changing to BETA and changing back). I believe the change from “query failed” to “post failed” happened when I changed the measurement name to “test” as a test. After changing it back the new “post failed” remained.

So interestingly. As I mentioned the error occurred at 20:48. After looking at the log, that was when the IotaWatt unit did a six week routine restart. After this reboot it was no longer working.


12/07/21 01:34:14 Updater: update tables from invalid to 0.0.4
12/29/21 20:48:16 timeSync: Six week routine restart.

** Restart **

SD initialized.
12/29/21 19:48:18z Real Time Clock is running. Unix time 1640807298
12/29/21 19:48:18z Reset reason: Software/System restart
12/29/21 19:48:18z Trace: 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:1[9], 1:2[10], 9:0[10], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[2], 1:6[2], 1:6[3], 1:5[20], 1:6[4], 20:0, 20:1
12/29/21 19:48:18z ESP8266 ID: 4399443, RTC PCF8523
12/29/21 19:48:18z IoTaWatt 5.0, Firmware version 02_06_06
12/29/21 19:48:18z SPIFFS mounted.
12/29/21 20:48:18 Local time zone: +1:00, using DST/BST when in effect.
12/29/21 20:48:18 device name: IotaWatt
12/29/21 20:48:18 HTTP server started
12/29/21 20:48:18 influxDB_v1: Starting, interval:5, url:http://192.168.1.172:8086
12/29/21 20:48:18 timeSync: service started.
12/29/21 20:48:18 statService: started.
12/29/21 20:48:18 dataLog: service started.
12/29/21 20:48:19 dataLog: Last log entry 12/29/21 20:48:15
12/29/21 20:48:22 WiFi connected. SSID=xFire129_2_4, IP=192.168.1.220, channel=9, RSSI -68db
12/29/21 20:48:22 MDNS responder started for hostname IotaWatt
12/29/21 20:48:22 LLMNR responder started for hostname IotaWatt
12/29/21 20:48:22 Updater: service started. Auto-update class is MINOR
12/29/21 20:48:26 historyLog: service started.
12/29/21 20:48:26 historyLog: Last log entry 12/29/21 20:48:00
12/29/21 20:48:26 Updater: Auto-update is current for class MINOR.
12/29/21 21:13:54 influxDB_v1: stopped, Last post 01/01/70 01:00:00

** Restart **

SD initialized.
12/29/21 20:22:49z Real Time Clock is running. Unix time 1640809369
12/29/21 20:22:49z Reset Reason: Power-fail restart.
12/29/21 20:22:49z ESP8266 ID: 4399443, RTC PCF8523
12/29/21 20:22:49z IoTaWatt 5.0, Firmware version 02_06_06
12/29/21 20:22:49z SPIFFS mounted.
12/29/21 21:22:50 Local time zone: +1:00, using DST/BST when in effect.
12/29/21 21:22:50 device name: IotaWatt
12/29/21 21:22:53 Connecting with WiFiManager.
12/29/21 21:22:56 HTTP server started
12/29/21 21:22:56 influxDB_v1: Starting, interval:5, url:http://192.168.1.172:8086
12/29/21 21:22:56 WiFi connected. SSID=xFire129_2_4, IP=192.168.1.220, channel=9, RSSI -69db
12/29/21 21:22:56 MDNS responder started for hostname IotaWatt
12/29/21 21:22:56 LLMNR responder started for hostname IotaWatt
12/29/21 21:22:56 timeSync: service started.
12/29/21 21:22:56 statService: started.
12/29/21 21:22:56 Updater: service started. Auto-update class is MINOR
12/29/21 21:22:56 dataLog: service started.
12/29/21 21:22:56 dataLog: Last log entry 12/29/21 21:22:20
12/29/21 21:22:58 Updater: Auto-update is current for class MINOR.
12/29/21 21:23:01 historyLog: service started.
12/29/21 21:23:01 historyLog: Last log entry 12/29/21 21:22:00
12/29/21 21:56:47 influxDB_v1: stopped, Last post 01/01/70 01:00:00
12/29/21 21:59:26 influxDB_v1: stopped, Last post 01/01/70 01:00:00
12/29/21 22:05:40 Updater: Auto-update class changed to BETA
12/29/21 22:05:41 Updater: Update from 02_06_06 to 02_07_05
12/29/21 22:05:41 Updater: download 02_07_05
12/29/21 22:05:47 Updater: Release downloaded 6874ms, size 951104
12/29/21 22:05:57 Updater: signature verified
12/29/21 22:06:06 Updater: firmware upgraded to version 02_07_05
12/29/21 22:06:06 Updater: Firmware updated, restarting.

** Restart **

SD initialized.
12/29/21 21:06:15z Real Time Clock is running. Unix time 1640811975
12/29/21 21:06:15z Reset reason: Software/System restart
12/29/21 21:06:15z Trace: 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:4, 18:5, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[2], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[7], 31:1, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[2], 1:6[3], 1:5[5], 1:6[4], 5:0, 5:7
12/29/21 21:06:15z ESP8266 ID: 4399443, RTC PCF8523 (68)
12/29/21 21:06:15z IoTaWatt 5.0, Firmware version 02_07_05
12/29/21 21:06:15z Updater: Installing update files for version 02_07_05
12/29/21 21:06:15z Updater: Installing GRAPH.HTM
12/29/21 21:06:15z Updater: Installing GRAPH.JS
12/29/21 21:06:15z Updater: Installing GRAPH2.HTM
12/29/21 21:06:15z Updater: Installing GRAPH2.JS
12/29/21 21:06:16z Updater: Installing INDEX.HTM
12/29/21 21:06:17z Updater: Installing TABLES.TXT
12/29/21 21:06:17z Updater: Installing CNFSTYLE.CSS
12/29/21 21:06:17z Updater: Installing EDIT.HTM
12/29/21 21:06:17z Updater: Installing FAVICON.ICO
12/29/21 21:06:18z Updater: Installation complete.
12/29/21 21:06:18z SPIFFS mounted.
12/29/21 22:06:18 Local time zone: +1:00, using DST/BST when in effect.
12/29/21 22:06:18 device name: IotaWatt
12/29/21 22:06:18 HTTP server started
12/29/21 22:06:18 influxDB_v1: Starting, interval:5, url:http://192.168.1.172:8086
12/29/21 22:06:18 timeSync: service started.
12/29/21 22:06:18 statService: started.
12/29/21 22:06:19 dataLog: service started.
12/29/21 22:06:19 dataLog: Last log entry 12/29/21 22:05:45
12/29/21 22:06:23 Updater: service started. Auto-update class is BETA
12/29/21 22:06:23 WiFi connected. SSID=xFire129_2_4, IP=192.168.1.220, channel=9, RSSI -71db
12/29/21 22:06:23 MDNS responder started for hostname IotaWatt
12/29/21 22:06:23 LLMNR responder started for hostname IotaWatt
12/29/21 22:06:24 Updater: Auto-update is current for class BETA.
12/29/21 22:06:24 historyLog: service started.
12/29/21 22:06:24 historyLog: Last log entry 12/29/21 22:05:00
12/29/21 22:07:20 influxDB_v1: stopped, Last post 01/01/70 01:00:00
12/29/21 22:08:01 Updater: Auto-update class changed to MINOR
12/29/21 22:08:01 Updater: Update from 02_07_05 to 02_06_06
12/29/21 22:08:01 Updater: download 02_06_06
12/29/21 22:08:08 Updater: Release downloaded 7189ms, size 900176
12/29/21 22:08:19 Updater: signature verified
12/29/21 22:08:27 Updater: firmware upgraded to version 02_06_06
12/29/21 22:08:27 Updater: Firmware updated, restarting.

** Restart **

SD initialized.
12/29/21 21:08:36z Real Time Clock is running. Unix time 1640812116
12/29/21 21:08:36z Reset reason: Software/System restart
12/29/21 21:08:36z Trace: 9:0[13], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[3], 1:5[31], 1:6[4], 31:0, 31:1, 31:2[7], 31:1, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[2], 1:6[3], 1:5[5], 1:6[4], 5:0, 5:7
12/29/21 21:08:36z ESP8266 ID: 4399443, RTC PCF8523
12/29/21 21:08:36z IoTaWatt 5.0, Firmware version 02_06_06
12/29/21 21:08:36z Updater: Installing update files for version 02_06_06
12/29/21 21:08:36z Updater: Installing FAVICON.ICO
12/29/21 21:08:37z Updater: Installing GRAPH.HTM
12/29/21 21:08:37z Updater: Installing GRAPH.JS
12/29/21 21:08:37z Updater: Installing GRAPH2.HTM
12/29/21 21:08:37z Updater: Installing GRAPH2.JS
12/29/21 21:08:38z Updater: Installing INDEX.HTM
12/29/21 21:08:39z Updater: Installing TABLES.TXT
12/29/21 21:08:39z Updater: Installing CNFSTYLE.CSS
12/29/21 21:08:39z Updater: Installing EDIT.HTM
12/29/21 21:08:39z Updater: Installation complete.
12/29/21 21:08:39z SPIFFS mounted.
12/29/21 22:08:39 Local time zone: +1:00, using DST/BST when in effect.
12/29/21 22:08:39 device name: IotaWatt
12/29/21 22:08:39 HTTP server started
12/29/21 22:08:39 influxDB_v1: Starting, interval:5, url:http://192.168.1.172:8086
12/29/21 22:08:39 timeSync: service started.
12/29/21 22:08:40 statService: started.
12/29/21 22:08:40 dataLog: service started.
12/29/21 22:08:40 dataLog: Last log entry 12/29/21 22:08:05
12/29/21 22:08:44 Updater: service started. Auto-update class is MINOR
12/29/21 22:08:44 WiFi connected. SSID=xFire129_2_4, IP=192.168.1.220, channel=9, RSSI -72db
12/29/21 22:08:44 MDNS responder started for hostname IotaWatt
12/29/21 22:08:44 LLMNR responder started for hostname IotaWatt
12/29/21 22:08:45 historyLog: service started.
12/29/21 22:08:45 historyLog: Last log entry 12/29/21 22:08:00
12/29/21 22:08:45 Updater: Auto-update is current for class MINOR.
12/29/21 22:08:45 Updater: update tables from 0.0.3 to 0.0.4
12/29/21 22:11:10 influxDB_v1: stopped, Last post 01/01/70 01:00:00
12/29/21 22:13:18 Updater: Auto-update class changed to MAJOR
12/29/21 22:13:19 Updater: Auto-update is current for class MAJOR.
12/29/21 22:16:00 influxDB_v1: stopped, Last post 01/01/70 01:00:00
12/29/21 23:34:12 influxDB_v1: Start posting at 11/16/21 00:00:05
12/29/21 23:37:43 influxDB_v1: stopped, Last post 11/16/21 00:00:00
12/29/21 23:37:52 influxDB_v1: Start posting at 11/16/21 00:00:05
12/29/21 23:38:52 influxDB_v1: stopped, Last post 11/16/21 00:00:00
12/29/21 23:40:56 influxDB_v1: Start posting at 11/16/21 00:00:05

** Restart **

SD initialized.
12/29/21 22:42:11z Real Time Clock is running. Unix time 1640817731
12/29/21 22:42:11z Reset Reason: Power-fail restart.
12/29/21 22:42:11z ESP8266 ID: 4399443, RTC PCF8523
12/29/21 22:42:11z IoTaWatt 5.0, Firmware version 02_06_06
12/29/21 22:42:11z SPIFFS mounted.
12/29/21 23:42:12 Local time zone: +1:00, using DST/BST when in effect.
12/29/21 23:42:12 device name: IotaWatt
12/29/21 23:42:14 HTTP server started
12/29/21 23:42:15 influxDB_v1: Starting, interval:5, url:http://192.168.1.172:8086
12/29/21 23:42:15 WiFi connected. SSID=xFire129_2_4, IP=192.168.1.220, channel=9, RSSI -70db
12/29/21 23:42:15 MDNS responder started for hostname IotaWatt
12/29/21 23:42:15 LLMNR responder started for hostname IotaWatt
12/29/21 23:42:15 timeSync: service started.
12/29/21 23:42:15 statService: started.
12/29/21 23:42:15 Updater: service started. Auto-update class is MAJOR
12/29/21 23:42:15 dataLog: service started.
12/29/21 23:42:15 dataLog: Last log entry 12/29/21 23:41:50
12/29/21 23:42:16 Updater: Auto-update is current for class MAJOR.
12/29/21 23:42:18 influxDB_v1: Start posting at 11/16/21 00:00:05
12/29/21 23:42:20 historyLog: service started.
12/29/21 23:42:20 historyLog: Last log entry 12/29/21 23:41:00
12/29/21 23:42:58 influxDB_v1: stopped, Last post 11/16/21 00:00:00
12/29/21 23:43:14 influxDB_v1: Start posting at 11/16/21 00:00:05
12/29/21 23:51:50 Updater: Auto-update class changed to BETA
12/29/21 23:51:50 Updater: Update from 02_06_06 to 02_07_05
12/29/21 23:51:50 Updater: download 02_07_05
12/29/21 23:51:57 Updater: Release downloaded 6755ms, size 951104
12/29/21 23:52:08 Updater: signature verified
12/29/21 23:52:16 Updater: firmware upgraded to version 02_07_05
12/29/21 23:52:16 Updater: Firmware updated, restarting.

** Restart **

SD initialized.
12/29/21 22:52:25z Real Time Clock is running. Unix time 1640818345
12/29/21 22:52:25z Reset reason: Software/System restart
12/29/21 22:52:25z Trace: 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 10:13, 1:3, 1:1[9], 1:2[10], 9:0[10], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[3], 1:5[5], 1:6[4], 5:0, 5:7
12/29/21 22:52:25z ESP8266 ID: 4399443, RTC PCF8523 (68)
12/29/21 22:52:25z IoTaWatt 5.0, Firmware version 02_07_05
12/29/21 22:52:25z Updater: Installing update files for version 02_07_05
12/29/21 22:52:25z Updater: Installing GRAPH.HTM
12/29/21 22:52:25z Updater: Installing GRAPH.JS
12/29/21 22:52:25z Updater: Installing GRAPH2.HTM
12/29/21 22:52:25z Updater: Installing GRAPH2.JS
12/29/21 22:52:26z Updater: Installing INDEX.HTM
12/29/21 22:52:27z Updater: Installing TABLES.TXT
12/29/21 22:52:27z Updater: Installing CNFSTYLE.CSS
12/29/21 22:52:27z Updater: Installing EDIT.HTM
12/29/21 22:52:27z Updater: Installing FAVICON.ICO
12/29/21 22:52:28z Updater: Installation complete.
12/29/21 22:52:28z SPIFFS mounted.
12/29/21 23:52:28 Local time zone: +1:00, using DST/BST when in effect.
12/29/21 23:52:28 device name: IotaWatt
12/29/21 23:52:28 HTTP server started
12/29/21 23:52:28 influxDB_v1: Starting, interval:5, url:http://192.168.1.172:8086
12/29/21 23:52:28 timeSync: service started.
12/29/21 23:52:28 statService: started.
12/29/21 23:52:29 dataLog: service started.
12/29/21 23:52:29 dataLog: Last log entry 12/29/21 23:51:55
12/29/21 23:52:32 Updater: service started. Auto-update class is BETA
12/29/21 23:52:33 historyLog: service started.
12/29/21 23:52:33 historyLog: Last log entry 12/29/21 23:51:00
12/29/21 23:52:33 WiFi connected. SSID=xFire129_2_4, IP=192.168.1.220, channel=9, RSSI -73db
12/29/21 23:52:33 MDNS responder started for hostname IotaWatt
12/29/21 23:52:33 LLMNR responder started for hostname IotaWatt
12/29/21 23:52:33 Updater: Auto-update is current for class BETA.
12/29/21 23:52:35 influxDB_v1: Start posting at 11/16/21 00:00:05
12/29/21 23:54:34 influxDB_v1: stopped, Last post 11/16/21 00:00:00
12/29/21 23:54:53 influxDB_v1: Start posting at 11/16/21 00:00:05

** Restart **

SD initialized.
12/29/21 23:02:52z Real Time Clock is running. Unix time 1640818972
12/29/21 23:02:52z Reset Reason: Power-fail restart.
12/29/21 23:02:52z ESP8266 ID: 4399443, RTC PCF8523 (68)
12/29/21 23:02:52z IoTaWatt 5.0, Firmware version 02_07_05
12/29/21 23:02:52z SPIFFS mounted.
12/30/21 00:02:52 Local time zone: +1:00, using DST/BST when in effect.
12/30/21 00:02:53 device name: IotaWatt
12/30/21 00:02:55 HTTP server started
12/30/21 00:02:55 influxDB_v1: Starting, interval:5, url:http://192.168.1.172:8086
12/30/21 00:02:56 WiFi connected. SSID=xFire129_2_4, IP=192.168.1.220, channel=9, RSSI -68db
12/30/21 00:02:56 MDNS responder started for hostname IotaWatt
12/30/21 00:02:56 LLMNR responder started for hostname IotaWatt
12/30/21 00:02:56 timeSync: service started.
12/30/21 00:02:56 statService: started.
12/30/21 00:02:56 Updater: service started. Auto-update class is BETA
12/30/21 00:02:56 dataLog: service started.
12/30/21 00:02:56 dataLog: Last log entry 12/30/21 00:00:30
12/30/21 00:02:58 Updater: Auto-update is current for class BETA.
12/30/21 00:02:59 influxDB_v1: Start posting at 11/16/21 00:00:05
12/30/21 00:03:01 historyLog: service started.
12/30/21 00:03:01 historyLog: Last log entry 12/30/21 00:00:00

That is a clue. The IoTaWatt must restart about every six weeks to avoid some counters rolling over to zero. When it restarted, it did a query to find out where to resume posting to influx and it failed then. What is a mystery is why the restart query didn’t fail when you first started the uploader. Even though there was no data, the problem is syntactical and should have occurred. So It would appear that something changed between when the upload started and the restart.

Are you saying that the failure didn’t change to “post failed” when you first substituted the underscore? If so, was there anything else different from when it failed with the dash? Can I see it setup with the underscore and the “query failed” message?
.

Can I see the complete uploader configuration as it is with the POST error?

Interesting. This is the first 6week restart since I first started the device the 15th of November. The device had been running flawlessly for those 6 weeks. It might be worth mentioning that I did change the influx configuration earlier today (29th of December). Main change was that the field-key was changed from a static value to a variable ($units) and the measurement was changed from “circuit” to “iotaWatt-measurementData”. After that it worked fine for a few hours until the reboot. As such I don’t know if the error was actually caused by the reboot or the change that I did. It might be that the error would have happened regardless.

I am 90% sure that the error changed from “query fail” to “post fail” when I changed the measurement from “iotaWatt-measurementData” to “test”. I am not a 100% sure because I did a lot of other minor changes as well to see if it changed anything. After changing the measurement back to “iotaWatt-measurementData” the error remained at “post fail” instead of reverting back to “query fail”. As for your question, I believe I never was in a position where I used underscore and had a “query fail” message.

Sure.


So, on a whim I changed the date for the “upload history from” from 11/16/2021 to 12/29/2021.

After doing this the problem fixed it self.

The root issue may not be solved, but for my purposes it is working again. It will be interesting to see what happens in 6 weeks.