Rebooting Every few minutes

Support
#1

All of a sudden I am seeing my IoTaWatt rebooting every few minutes. Here’s what the Message Log looks like:

3/15/21 08:29:34 historyLog: service started.
3/15/21 08:29:35 historyLog: Last log entry 03/15/21 08:28:00
3/15/21 08:29:43 PVoutput: started
3/15/21 08:30:01 PVoutput: System Xxxxxxx-1838, interval 5, freeload mode
3/15/21 08:30:29 PVoutput: Start status beginning 03/15/21 08:25:00
3/15/21 08:31:29 Heap memory has degraded below safe minimum, restarting.

** Restart **

SD initialized.
3/15/21 12:31:30z Real Time Clock is running. Unix time 1615811490
3/15/21 12:31:30z Reset reason: Software/System restart
3/15/21 12:31:30z Trace: 1:1[10], 1:2[11], 9:0[11], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:5[3], 3:0, 3:11, 1:6, 1:3, 1:4, 1:5[7], 7:0, 7:5, 1:6, 1:3, 1:4, 1:5[21], 21:0, 21:1, 21:10, 21:10
3/15/21 12:31:30z ESP8266 ChipID: 6308187
3/15/21 12:31:30z IoTaWatt 5.0, Firmware version 02_05_12
3/15/21 12:31:30z SPIFFS mounted.
3/15/21 08:31:30 Local time zone: -5:00
3/15/21 08:31:31 Using Daylight Saving Time (BST) when in effect.
3/15/21 08:31:31 device name: IotaWatt
3/15/21 08:31:31 HTTP server started
3/15/21 08:31:31 timeSync: service started.
3/15/21 08:31:31 statService: started.
3/15/21 08:31:31 dataLog: service started.
3/15/21 08:31:34 dataLog: Last log entry 03/15/21 08:31:25
3/15/21 08:31:36 EmonService: started. url=HTTP://192.168.1.105:8081, node=IotaWatt, interval=10, encrypted
3/15/21 08:31:36 influxDB: started, url=HTTP://192.168.1.105:8086, db=iotawatt_db, interval=10
3/15/21 08:31:37 WiFi connected. SSID=Automation, IP=192.168.1.224, channel=11, RSSI -71db
3/15/21 08:31:37 MDNS responder started for hostname IotaWatt
3/15/21 08:31:37 LLMNR responder started for hostname IotaWatt
3/15/21 08:31:37 Updater: service started. Auto-update class is ALPHA
3/15/21 08:31:39 EmonService: Start posting at 03/15/21 08:31:20
3/15/21 08:31:44 Updater: Auto-update is current for class ALPHA.
3/15/21 08:32:32 historyLog: service started.
3/15/21 08:32:33 historyLog: Last log entry 03/15/21 08:31:00
3/15/21 08:32:41 PVoutput: started
3/15/21 08:32:49 PVoutput: System Xxxxxxx-1838, interval 5, freeload mode
3/15/21 08:33:06 PVoutput: Start status beginning 03/15/21 08:30:00
3/15/21 08:33:30 influxDB: Start posting at 03/15/21 08:27:30
3/15/21 08:34:12 Heap memory has degraded below safe minimum, restarting.

** Restart **

SD initialized.
3/15/21 12:34:13z Real Time Clock is running. Unix time 1615811653
3/15/21 12:34:13z Reset reason: Software/System restart
3/15/21 12:34:13z Trace: 1:1[11], 1:2[12], 9:0[12], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:5[3], 3:0, 3:11, 1:6, 1:3, 1:4, 1:5[7], 7:0, 7:9, 1:6, 1:3, 1:4, 1:5[21], 21:0, 21:1, 21:10, 21:10
3/15/21 12:34:13z ESP8266 ChipID: 6308187
3/15/21 12:34:13z IoTaWatt 5.0, Firmware version 02_05_12
3/15/21 12:34:13z SPIFFS mounted.
3/15/21 08:34:13 Local time zone: -5:00
3/15/21 08:34:13 Using Daylight Saving Time (BST) when in effect.
3/15/21 08:34:14 device name: IotaWatt
3/15/21 08:34:14 HTTP server started
3/15/21 08:34:14 timeSync: service started.
3/15/21 08:34:14 statService: started.
3/15/21 08:34:14 dataLog: service started.
3/15/21 08:34:17 dataLog: Last log entry 03/15/21 08:34:10
3/15/21 08:34:19 EmonService: started. url=HTTP://192.168.1.105:8081, node=IotaWatt, interval=10, encrypted
3/15/21 08:34:19 influxDB: started, url=HTTP://192.168.1.105:8086, db=iotawatt_db, interval=10
3/15/21 08:34:23 WiFi connected. SSID=Automation, IP=192.168.1.224, channel=11, RSSI -71db
3/15/21 08:34:23 MDNS responder started for hostname IotaWatt
3/15/21 08:34:23 LLMNR responder started for hostname IotaWatt
3/15/21 08:34:23 Updater: service started. Auto-update class is ALPHA
3/15/21 08:34:26 EmonService: Start posting at 03/15/21 08:34:20
3/15/21 08:34:38 Updater: Auto-update is current for class ALPHA.
3/15/21 08:35:14 historyLog: service started.
3/15/21 08:35:15 historyLog: Last log entry 03/15/21 08:34:00
3/15/21 08:35:24 PVoutput: started
3/15/21 08:35:44 influxDB: Start posting at 03/15/21 08:30:30
3/15/21 08:35:46 PVoutput: System Xxxxxxx-1838, interval 5, freeload mode
3/15/21 08:35:56 PVoutput: Start status beginning 03/15/21 08:30:00
3/15/21 08:36:01 Heap memory has degraded below safe minimum, restarting.

** Restart **

SD initialized.
3/15/21 12:36:02z Real Time Clock is running. Unix time 1615811762
3/15/21 12:36:02z Reset reason: Software/System restart
3/15/21 12:36:02z Trace: 7:9, 1:6, 1:3, 1:4, 1:5[14], 14:0, 14:4, 14:5, 14:7, 14:8, 14:9, 1:6, 1:1[9], 1:2[10], 9:0[10], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:5[21], 21:0, 21:1, 21:10, 21:10
3/15/21 12:36:02z ESP8266 ChipID: 6308187
3/15/21 12:36:02z IoTaWatt 5.0, Firmware version 02_05_12
3/15/21 12:36:02z SPIFFS mounted.
3/15/21 08:36:02 Local time zone: -5:00
3/15/21 08:36:03 Using Daylight Saving Time (BST) when in effect.
3/15/21 08:36:03 device name: IotaWatt
3/15/21 08:36:03 HTTP server started
3/15/21 08:36:03 timeSync: service started.
3/15/21 08:36:03 statService: started.
3/15/21 08:36:03 dataLog: service started.
3/15/21 08:36:06 dataLog: Last log entry 03/15/21 08:36:00
3/15/21 08:36:08 EmonService: started. url=HTTP://192.168.1.105:8081, node=IotaWatt, interval=10, encrypted
3/15/21 08:36:08 influxDB: started, url=HTTP://192.168.1.105:8086, db=iotawatt_db, interval=10
3/15/21 08:36:16 WiFi connected. SSID=Automation, IP=192.168.1.224, channel=6, RSSI -92db
3/15/21 08:36:16 MDNS responder started for hostname IotaWatt
3/15/21 08:36:16 LLMNR responder started for hostname IotaWatt
3/15/21 08:36:16 Updater: service started. Auto-update class is ALPHA
3/15/21 08:36:19 EmonService: Start posting at 03/15/21 08:36:00
3/15/21 08:36:20 Updater: Auto-update is current for class ALPHA.
3/15/21 08:37:04 historyLog: service started.
3/15/21 08:37:05 historyLog: Last log entry 03/15/21 08:36:00
3/15/21 08:37:05 influxDB: Start posting at 03/15/21 08:30:30
3/15/21 08:37:13 PVoutput: started
3/15/21 08:37:13 PVoutput: System Xxxxxxx-1838, interval 5, freeload mode
3/15/21 08:37:16 PVoutput: Start status beginning 03/15/21 08:30:00
3/15/21 08:37:51 Heap memory has degraded below safe minimum, restarting.

** Restart **

SD initialized.
3/15/21 12:37:51z Real Time Clock is running. Unix time 1615811871
3/15/21 12:37:51z Reset reason: Software/System restart
3/15/21 12:37:51z Trace: 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:5[3], 3:0, 3:11, 1:6, 1:3, 1:4, 1:5[7], 7:0, 7:9, 1:6, 1:3, 1:4, 1:5[5], 5:0, 5:2, 1:6, 1:3, 1:4, 1:5[21], 21:0, 21:1, 21:10, 21:10
3/15/21 12:37:51z ESP8266 ChipID: 6308187
3/15/21 12:37:51z IoTaWatt 5.0, Firmware version 02_05_12
3/15/21 12:37:51z SPIFFS mounted.
3/15/21 08:37:51 Local time zone: -5:00
3/15/21 08:37:52 Using Daylight Saving Time (BST) when in effect.
3/15/21 08:37:52 device name: IotaWatt
3/15/21 08:37:52 HTTP server started
3/15/21 08:37:52 timeSync: service started.
3/15/21 08:37:52 statService: started.
3/15/21 08:37:52 dataLog: service started.
3/15/21 08:37:55 dataLog: Last log entry 03/15/21 08:37:50
3/15/21 08:37:57 EmonService: started. url=HTTP://192.168.1.105:8081, node=IotaWatt, interval=10, encrypted
3/15/21 08:37:57 influxDB: started, url=HTTP://192.168.1.105:8086, db=iotawatt_db, interval=10
3/15/21 08:38:03 WiFi connected. SSID=Automation, IP=192.168.1.224, channel=11, RSSI -74db
3/15/21 08:38:03 MDNS responder started for hostname IotaWatt
3/15/21 08:38:03 LLMNR responder started for hostname IotaWatt
3/15/21 08:38:03 Updater: service started. Auto-update class is ALPHA
3/15/21 08:38:05 EmonService: Start posting at 03/15/21 08:37:50
3/15/21 08:38:08 Updater: Auto-update is current for class ALPHA.
3/15/21 08:38:53 historyLog: service started.
3/15/21 08:38:54 historyLog: Last log entry 03/15/21 08:37:00
3/15/21 08:39:02 PVoutput: started
3/15/21 08:39:07 PVoutput: System Xxxxxxx-1838, interval 5, freeload mode
3/15/21 08:39:30 PVoutput: Start status beginning 03/15/21 08:35:00
3/15/21 08:39:44 Heap memory has degraded below safe minimum, restarting.

** Restart **

SD initialized.
3/15/21 12:39:45z Real Time Clock is running. Unix time 1615811985
3/15/21 12:39:45z Reset reason: Software/System restart
3/15/21 12:39:45z Trace: 7:0, 7:5, 1:6, 1:3, 1:4, 1:5[3], 3:0, 3:11, 1:6, 1:3, 1:4, 1:5[7], 7:0, 7:5, 1:6, 1:3, 1:4, 1:5[22], 22:0, 22:1, 22:0, 22:90, 22:92, 22:2, 1:6, 1:3, 1:4, 1:5[21], 21:0, 21:1, 21:10, 21:10
3/15/21 12:39:45z ESP8266 ChipID: 6308187
3/15/21 12:39:45z IoTaWatt 5.0, Firmware version 02_05_12
3/15/21 12:39:45z SPIFFS mounted.
3/15/21 08:39:45 Local time zone: -5:00
3/15/21 08:39:45 Using Daylight Saving Time (BST) when in effect.
3/15/21 08:39:46 device name: IotaWatt
3/15/21 08:39:46 HTTP server started
3/15/21 08:39:46 timeSync: service started.
3/15/21 08:39:46 statService: started.
3/15/21 08:39:46 dataLog: service started.
3/15/21 08:39:49 dataLog: Last log entry 03/15/21 08:39:40
3/15/21 08:39:51 EmonService: started. url=HTTP://192.168.1.105:8081, node=IotaWatt, interval=10, encrypted
3/15/21 08:39:51 influxDB: started, url=HTTP://192.168.1.105:8086, db=iotawatt_db, interval=10
3/15/21 08:39:53 WiFi connected. SSID=Automation, IP=192.168.1.224, channel=11, RSSI -74db
3/15/21 08:39:53 MDNS responder started for hostname IotaWatt
3/15/21 08:39:53 LLMNR responder started for hostname IotaWatt
3/15/21 08:39:53 Updater: service started. Auto-update class is ALPHA
3/15/21 08:39:58 EmonService: Start posting at 03/15/21 08:39:20
3/15/21 08:40:01 Updater: Auto-update is current for class ALPHA.
3/15/21 08:40:47 historyLog: service started.
3/15/21 08:40:48 historyLog: Last log entry 03/15/21 08:39:00
3/15/21 08:40:53 influxDB: Start posting at 03/15/21 08:32:30
3/15/21 08:40:56 PVoutput: started
3/15/21 08:40:58 PVoutput: System Xxxxxxx-1838, interval 5, freeload mode
3/15/21 08:41:00 PVoutput: Start status beginning 03/15/21 08:35:00

#2

There’s no smoking gun in the trace. Heap degradation like this is always associated with abnormal or excessive WiFi activity. Your WiFi RSSI looks good. One of the restarts posted shows PVoutput with an upload issue. All of them show outstanding transactions which is no surprise since it is just starting and you have three uploaders configured.

I run three uploaders in most of the test systems. They run fine steady state when they are all current, but getting to that steady state can cause issues. I’ve done some work on that in the next release to try to make it divide resources equitably with multiple uploaders.

But that doesn’t help you now. I’d suggest trying to stop the uploaders using the STOP button in the status display. It appears there is about a one minute window between restarts to do that. When you press stop, wait about ten seconds before pressing it again. It’s not immediate, and it won’t work at all if the uploader has an I/O in progress.

Once you get them all stopped, try starting them one at a time and waiting a few minutes after it gets current before starting the next. Keep an eye on the heap. It should be above 18,000 when the uploaders are caught up.

If that fails, or you can’t stop the uploaders, we can do something more drastic to stop them.

1 Like
#3

Thanks for the reply - it does seem to be correlated to some WiFi problems. I have been having intermittent issues with my new Ubiquiti Dream Machine Pro and that is likely the culprit. I have a sneaky suspicion that the controller went ahead and did a firmware upgrade in the last week or so that I was unaware of. I just did a firmware upgrade on the UDM and all of sudden my IoTaWatt seems happy again. I’ll monitor and see if it remains stable.

Thanks again!

1 Like