InfluxDB backlog

Support
1

Hi folks,
I am an 18 month old user and want to comment bob and all involved on such a strong product.

I left a unit I have monitoring offsite offline for over a month of the Xmas period and upon reconnecting it the connection to influx is running but it is only clearing 2 days of backlog in data upload to influx every day. at the moment it looks like i will have to wait 29 days for this to get up to date which is a surprise

I went and disconnected my home unit from the internet for a day yesterday and it was able to upload to the same influx instance in less than 30 minutes.

Can anyone shed some light on this? Is there a way I can force the update rate?

At the moment I am up to date on firmware, have the device to send bulk of one every 10 seconds.

Thanks!

J

2

My first guess would be that the connection is error prone and there’s is a lot of failure/retry. Have you looked at the data that has uploaded and is it ok?

Second, and not enough to explain your stated data rate, are there a lot of measurements in the upload. Some folks have been known to upload 30 or 40 measurements. That said a lot and can generate huge packets.

Can you upload the message log?

3

Thanks Bob.

Connection is up and managed via VPN

My message log is downloading now (11.2Mb) I can include a snippet from the last of the log

"2/01/20 16:54:37z timeSync: service started.
2/01/20 16:54:38z statService: started.
2/01/20 16:54:38z Updater: service started. Auto-update class is MAJOR
2/01/20 16:54:39z dataLog: service started.
2/01/20 16:54:51z dataLog: Last log entry 02/01/20 16:53:50
2/01/20 16:54:51z Updater: Update from 02_04_02 to 02_05_02
2/01/20 16:54:52z Updater: download 02_05_02
2/01/20 16:55:09z Updater: Release downloaded 15840ms, size 762848
2/01/20 16:55:17z Updater: signature verified
2/01/20 16:55:24z Updater: firmware upgraded to version 02_05_02
2/01/20 16:55:24z Updater: Firmware updated, restarting.

** Restart **

SD initialized.
2/01/20 16:55:31z Real Time Clock is running. Unix time 1580576131
2/01/20 16:55:31z Reset reason: Software/System restart
2/01/20 16:55:32z Trace: 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:5[19], 1:6, 1:1[4], 1:2[5], 9:0[5], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:5[5], 5:0, 5:7
2/01/20 16:55:33z ESP8266 ChipID: 2518062
2/01/20 16:55:34z IoTaWatt 4.x, Firmware version 02_05_02
2/01/20 16:55:34z Updater: Installing update files for version 02_05_02
2/01/20 16:55:35z Updater: Installing GRAPH.JS
2/01/20 16:55:36z Updater: Installing GRAPH2.HTM
2/01/20 16:55:42z Updater: Installing GRAPH2.JS
2/01/20 16:55:48z Updater: Installing INDEX.HTM
2/01/20 16:55:53z Updater: Installing TABLES.TXT
2/01/20 16:55:53z Updater: Installing CNFSTYLE.CSS
2/01/20 16:55:54z Updater: Installing EDIT.HTM
2/01/20 16:55:55z Updater: Installing GRAPH.HTM
2/01/20 16:55:57z Updater: Installation complete.
2/01/20 16:55:57z SPIFFS mounted.
2/01/20 16:55:59z Local time zone: +0:00
2/01/20 16:55:59z device name: JohnFitz
2/01/20 16:56:00z MDNS responder started for hostname JohnFitz
2/01/20 16:56:00z LLMNR responder started for hostname JohnFitz
2/01/20 16:56:01z HTTP server started
2/01/20 16:56:02z WiFi connected. SSID=DCSIXBH24, IP=192.168.8.184, channel=2, RSSI -64db
2/01/20 16:56:02z timeSync: service started.
2/01/20 16:56:03z statService: started.
2/01/20 16:56:03z Updater: service started. Auto-update class is MAJOR
2/01/20 16:56:04z dataLog: service started.
2/01/20 16:56:22z dataLog: Last log entry 02/01/20 16:55:05
2/01/20 16:56:23z Updater: Auto-update is current for class MAJOR.
2/01/20 16:57:04z historyLog: service started.
2/01/20 16:57:14z historyLog: Last log entry 02/01/20 00:05:00
2/01/20 19:55:12z influxDB: started, url=192.168.2.187:8086, db=JohnFitzFrm, interval=5
2/01/20 19:55:17z influxDB: Start posting at 12/30/19 18:46:05
2/01/20 19:56:05z influxDB: Restart. Last post 12/30/19 18:46:20
2/01/20 19:56:06z influxDB: started, url=192.168.2.187:8086, db=JohnFitzFrm, interval=5
2/01/20 19:56:10z influxDB: Start posting at 12/30/19 18:46:25
2/01/20 19:58:24z influxDB: Restart. Last post 12/30/19 18:48:20
2/01/20 19:58:25z influxDB: started, url=192.168.2.187:8086, db=JohnFitzFrm, interval=10
2/01/20 19:58:29z influxDB: Start posting at 12/30/19 18:48:30
2/03/20 03:59:45z Updater: Invalid response from server. HTTPcode: -4
2/03/20 04:16:35z influxDB: Post Failed: -4

** Restart **

SD initialized.
2/06/20 09:56:43z Real Time Clock is running. Unix time 1580983003
2/06/20 09:56:43z Power failure detected.
2/06/20 09:56:44z Reset reason: Power on
2/06/20 09:56:44z ESP8266 ChipID: 2518062
2/06/20 09:56:44z IoTaWatt 4.x, Firmware version 02_05_02
2/06/20 09:56:45z SPIFFS mounted.
2/06/20 09:56:46z Local time zone: +0:00
2/06/20 09:56:47z device name: JohnFitz
2/06/20 09:56:50z Connecting with WiFiManager.

** Restart **

SD initialized.
2/06/20 10:00:08z Real Time Clock is running. Unix time 1580983208
2/06/20 10:00:08z Reset reason: Hardware Watchdog
2/06/20 10:00:09z Trace: 15:2[6], 15:3, 15:0[6], 15:2[6], 15:3, 15:0[6], 15:2[6], 15:3, 15:0[6], 15:2[6], 15:3, 15:0[6], 15:2[6], 15:3, 11:8, 11:9, 16:0, 11:10, 17:0, 17:0, 17:0, 17:4, 17:5, 17:7, 17:7, 17:7, 17:7, 17:8, 17:9, 17:10, 11:11, 11:12
2/06/20 10:00:11z ESP8266 ChipID: 2518062
2/06/20 10:00:11z IoTaWatt 4.x, Firmware version 02_05_02
2/06/20 10:00:11z SPIFFS mounted.
2/06/20 10:00:13z Local time zone: +0:00
2/06/20 10:00:13z device name: JohnFitz
2/06/20 10:00:14z MDNS responder started for hostname JohnFitz
2/06/20 10:00:15z LLMNR responder started for hostname JohnFitz
2/06/20 10:00:15z HTTP server started
2/06/20 10:00:16z WiFi connected. SSID=DCSIXBH24, IP=192.168.8.184, channel=6, RSSI -61db
2/06/20 10:00:17z timeSync: service started.
2/06/20 10:00:24z statService: started.
2/06/20 10:00:25z Updater: service started. Auto-update class is MAJOR
2/06/20 10:00:26z dataLog: service started.
2/06/20 10:00:43z dataLog: Last log entry 02/06/20 09:40:30
2/06/20 10:00:44z Updater: Invalid response from server. HTTPcode: -4
2/06/20 10:00:45z influxDB: started, url=192.168.2.187:8086, db=JohnFitzFrm, interval=10
2/06/20 10:01:30z historyLog: service started.
2/06/20 10:01:40z historyLog: Last log entry 02/06/20 09:40:00
2/06/20 10:08:29z influxDB: last entry query failed: -4, retrying.

** Restart **

SD initialized.
2/06/20 15:26:41z Real Time Clock is running. Unix time 1581002801
2/06/20 15:26:41z Power failure detected.
2/06/20 15:26:42z Reset reason: Power on
2/06/20 15:26:42z ESP8266 ChipID: 2518062
2/06/20 15:26:42z IoTaWatt 4.x, Firmware version 02_05_02
2/06/20 15:26:43z SPIFFS mounted.
2/06/20 15:26:44z Local time zone: +0:00
2/06/20 15:26:45z device name: JohnFitz
2/06/20 15:26:48z Connecting with WiFiManager.

** Restart **

SD initialized.
2/06/20 15:30:07z Real Time Clock is running. Unix time 1581003007
2/06/20 15:30:07z Reset reason: Hardware Watchdog
2/06/20 15:30:08z Trace: 15:2[6], 15:3, 15:0[6], 15:2[6], 15:3, 15:0[6], 15:2[6], 15:3, 15:0[6], 15:2[6], 15:3, 15:0[6], 15:2[6], 15:3, 11:8, 11:9, 16:0, 11:10, 17:0, 17:0, 17:0, 17:4, 17:5, 17:7, 17:7, 17:7, 17:7, 17:8, 17:9, 17:10, 11:11, 11:12
2/06/20 15:30:10z ESP8266 ChipID: 2518062
2/06/20 15:30:10z IoTaWatt 4.x, Firmware version 02_05_02
2/06/20 15:30:10z SPIFFS mounted.
2/06/20 15:30:12z Local time zone: +0:00
2/06/20 15:30:12z device name: JohnFitz
2/06/20 15:30:13z MDNS responder started for hostname JohnFitz
2/06/20 15:30:14z LLMNR responder started for hostname JohnFitz
2/06/20 15:30:14z HTTP server started
2/06/20 15:30:15z WiFi connected. SSID=DCSIXBH24, IP=192.168.8.184, channel=6, RSSI -61db
2/06/20 15:30:16z timeSync: service started.
2/06/20 15:30:23z statService: started.
2/06/20 15:30:23z Updater: service started. Auto-update class is MAJOR
2/06/20 15:30:25z dataLog: service started.
2/06/20 15:30:45z dataLog: Last log entry 02/06/20 13:11:45
2/06/20 15:30:45z Updater: Invalid response from server. HTTPcode: -4
2/06/20 15:30:46z influxDB: started, url=192.168.2.187:8086, db=JohnFitzFrm, interval=10
2/06/20 15:31:29z historyLog: service started.
2/06/20 15:31:39z historyLog: Last log entry 02/06/20 13:11:00
2/06/20 15:34:21z influxDB: Start posting at 01/08/20 18:46:40
2/06/20 16:08:16z dataLog: datalog WDT - restarting

** Restart **

SD initialized.
2/06/20 16:08:18z Real Time Clock is running. Unix time 1581005298
2/06/20 16:08:18z Reset reason: Software/System restart
2/06/20 16:08:19z Trace: 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:4, 18:5, 1:6, 1:1[1], 1:2[2], 9:0[2], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:5[19]
2/06/20 16:08:20z ESP8266 ChipID: 2518062
2/06/20 16:08:21z IoTaWatt 4.x, Firmware version 02_05_02
2/06/20 16:08:21z SPIFFS mounted.
2/06/20 16:08:23z Local time zone: +0:00
2/06/20 16:08:23z device name: JohnFitz
2/06/20 16:08:23z MDNS responder started for hostname JohnFitz
2/06/20 16:08:24z LLMNR responder started for hostname JohnFitz
2/06/20 16:08:25z HTTP server started
2/06/20 16:08:25z WiFi connected. SSID=DCSIXBH24, IP=192.168.8.184, channel=6, RSSI -61db
2/06/20 16:08:26z timeSync: service started.
2/06/20 16:08:27z statService: started.
2/06/20 16:08:27z Updater: service started. Auto-update class is MAJOR
2/06/20 16:08:28z dataLog: service started.
2/06/20 16:08:47z dataLog: Last log entry 02/06/20 16:07:45
2/06/20 16:08:48z Updater: Auto-update is current for class MAJOR.
2/06/20 16:08:49z influxDB: started, url=192.168.2.187:8086, db=JohnFitzFrm, interval=10
2/06/20 16:08:54z influxDB: Start posting at 01/08/20 19:47:40
2/06/20 16:09:41z dataLog: datalog WDT - restarting

** Restart **

SD initialized.
2/06/20 16:09:42z Real Time Clock is running. Unix time 1581005382
2/06/20 16:09:42z Reset reason: Software/System restart
2/06/20 16:09:43z Trace: 7:71, 7:71, 7:71, 7:71, 7:71, 7:71, 7:71, 7:71, 7:71, 7:71, 7:71, 7:71, 7:71, 7:71, 7:7, 7:7, 1:6, 1:1[11], 1:2[12], 9:0[12], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 10:13
2/06/20 16:09:44z ESP8266 ChipID: 2518062
2/06/20 16:09:45z IoTaWatt 4.x, Firmware version 02_05_02
2/06/20 16:09:45z SPIFFS mounted.
2/06/20 16:09:47z Local time zone: +0:00
2/06/20 16:09:47z device name: JohnFitz
2/06/20 16:09:48z MDNS responder started for hostname JohnFitz
2/06/20 16:09:48z LLMNR responder started for hostname JohnFitz
2/06/20 16:09:49z HTTP server started
2/06/20 16:09:50z WiFi connected. SSID=DCSIXBH24, IP=192.168.8.184, channel=6, RSSI -63db
2/06/20 16:09:50z timeSync: service started.
2/06/20 16:09:51z statService: started.
2/06/20 16:09:52z Updater: service started. Auto-update class is MAJOR
2/06/20 16:09:53z dataLog: service started.
2/06/20 16:10:14z dataLog: Last log entry 02/06/20 16:09:10
2/06/20 16:10:15z Updater: Auto-update is current for class MAJOR.
2/06/20 16:10:16z influxDB: started, url=192.168.2.187:8086, db=JohnFitzFrm, interval=10
2/06/20 16:10:22z influxDB: Start posting at 01/08/20 19:47:40
2/06/20 16:10:55z historyLog: service started.
2/06/20 16:11:05z historyLog: Last log entry 02/06/20 16:07:00"

Hope this helps, you will see some failed writes as I intentionally renamed the DB to verify that it was in fact writing to influx

The Data seems spot on and I am bringing back 18 measurands.

Thanks,

J

4

You appear to be using the DCSIX product. I really don’t know a lot about it. The person that contacted me last year stopped responding to my inquiries. All I can see by their old website is that they appear to use a GSM based hotspot to communicate, using some kind of proprietary box. As you report another unit running fine against the same influDB instance, my impression is that the difference is the communications link. Both the speed and integrity of the connection are in question. The only metric I can see in the brief log is the time it took to download the 02_05_02 release blob - 15 seconds. That normally takes about 6 seconds, and is cached through Cloudflare so should be OK where you are, which I’m guessing by the timezone and DCSIX is Ireland.