Repeated reboots with no internet

We had a big storm roll through my area and we lost power for 4 days. During that time the iotawatt was off and kept off except for one small startup after I got the generator hooked up and before I unplugged the power adapter (doubt it’s related, but wanted to mention). After the power came back on it took several more days to get internet service restored but I took the chance on the date/time being “close enough” on the iota and turned it on. It seemed ok so I just let it go. What I noticed is the the iota rebooting quite often due to low heap memory so I’m wondering if there’s a memory leak in some error handling code for something that’s accessing the internet. As you can see from the logs I do have a single uploader for pvoutput configured. I did not test stopping the uploader right after a reboot to see if that helped. The current uptime is “Running time: 5d 18h 34m 58s” which is basically since the internet came back so that’s why I’m fairly certain it’s related to that.

Here’s a sample of the logs:

7/20/22 10:45:51z Real Time Clock is running. Unix time 1658313951 
7/20/22 10:45:51z Reset reason: Software/System restart
7/20/22 10:45:51z Trace:  8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[2], 1:6[3], 1:5[22], 1:6[4], 22:0, 22:1, 22:0, 22:120, 22:2, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[2], 1:6[2], 1:6[3], 1:5[21], 1:6[4], 21:0, 21:1, 21:10, 21:10
7/20/22 10:45:51z ESP8266 ID: 567551, RTC M41T81 (68)
7/20/22 10:45:51z IoTaWatt 5.0, Firmware version 02_07_05
7/20/22 10:45:51z SPIFFS mounted.
7/20/22 06:45:51 Local time zone: -5:00, using DST/BST when in effect.
7/20/22 06:45:51 device name: IotaWatt
7/20/22 06:45:51 HTTP server started
7/20/22 06:45:51 timeSync: service started.
7/20/22 06:45:51 statService: started.
7/20/22 06:45:51 dataLog: service started.
7/20/22 06:45:52 dataLog: Last log entry 07/20/22 06:45:50
7/20/22 06:45:55 WiFi connected. SSID=<ssid>, IP=<ip>, channel=6, RSSI -57db
7/20/22 06:45:55 MDNS responder started for hostname IotaWatt
7/20/22 06:45:55 LLMNR responder started for hostname IotaWatt
7/20/22 06:45:55 Updater: service started. Auto-update class is MINOR
7/20/22 06:46:05 historyLog: service started.
7/20/22 06:46:05 historyLog: Last log entry 07/20/22 06:45:00
7/20/22 06:46:05 Updater: Invalid response from server. HTTPcode: -4
7/20/22 06:46:06 PVoutput: started
7/20/22 06:46:12 grid: Started
7/20/22 06:46:12 grid: Last log entry 07/20/22 06:45:50
7/20/22 09:43:12 Heap memory has degraded below safe minimum, restarting.

** Restart **

SD initialized.
7/20/22 13:43:14z Real Time Clock is running. Unix time 1658324594 
7/20/22 13:43:14z Reset reason: Software/System restart
7/20/22 13:43:14z Trace:  1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:3, 1:1[10], 1:2[11], 9:0[11], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[3], 1:5[21], 1:6[4], 21:0, 21:1, 21:10, 21:10
7/20/22 13:43:14z ESP8266 ID: 567551, RTC M41T81 (68)
7/20/22 13:43:14z IoTaWatt 5.0, Firmware version 02_07_05
7/20/22 13:43:14z SPIFFS mounted.
7/20/22 09:43:14 Local time zone: -5:00, using DST/BST when in effect.
7/20/22 09:43:14 device name: IotaWatt
7/20/22 09:43:14 HTTP server started
7/20/22 09:43:14 timeSync: service started.
7/20/22 09:43:14 statService: started.
7/20/22 09:43:14 dataLog: service started.
7/20/22 09:43:15 dataLog: Last log entry 07/20/22 09:43:10
7/20/22 09:43:18 WiFi connected. SSID=<ssid>, IP=<ip>, channel=6, RSSI -55db
7/20/22 09:43:18 MDNS responder started for hostname IotaWatt
7/20/22 09:43:18 LLMNR responder started for hostname IotaWatt
7/20/22 09:43:18 Updater: service started. Auto-update class is MINOR
7/20/22 09:43:28 historyLog: service started.
7/20/22 09:43:29 historyLog: Last log entry 07/20/22 09:43:00
7/20/22 09:43:29 Updater: Invalid response from server. HTTPcode: -4
7/20/22 09:43:29 PVoutput: started
7/20/22 09:43:35 grid: Started
7/20/22 09:43:35 grid: Last log entry 07/20/22 09:43:10
7/20/22 12:29:20 Heap memory has degraded below safe minimum, restarting.

** Restart **

SD initialized.
7/20/22 16:29:21z Real Time Clock is running. Unix time 1658334561 
7/20/22 16:29:21z Reset reason: Software/System restart
7/20/22 16:29:21z Trace:  9:0, 8:4, 8:6, 8:8, 8:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[2], 1:6[3], 1:5[22], 1:6[4], 22:0, 22:1, 22:0, 22:120, 22:2, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[2], 1:6[2], 1:6[3], 1:5[21], 1:6[4], 21:0, 21:1, 21:10, 21:10
7/20/22 16:29:21z ESP8266 ID: 567551, RTC M41T81 (68)
7/20/22 16:29:21z IoTaWatt 5.0, Firmware version 02_07_05
7/20/22 16:29:21z SPIFFS mounted.
7/20/22 12:29:21 Local time zone: -5:00, using DST/BST when in effect.
7/20/22 12:29:21 device name: IotaWatt
7/20/22 12:29:21 HTTP server started
7/20/22 12:29:21 timeSync: service started.
7/20/22 12:29:21 statService: started.
7/20/22 12:29:21 dataLog: service started.
7/20/22 12:29:22 dataLog: Last log entry 07/20/22 12:29:20
7/20/22 12:29:25 WiFi connected. SSID=<ssid>, IP=<ip>, channel=6, RSSI -58db
7/20/22 12:29:25 MDNS responder started for hostname IotaWatt
7/20/22 12:29:25 LLMNR responder started for hostname IotaWatt
7/20/22 12:29:25 Updater: service started. Auto-update class is MINOR
7/20/22 12:29:35 historyLog: service started.
7/20/22 12:29:36 historyLog: Last log entry 07/20/22 12:29:00
7/20/22 12:29:36 Updater: Invalid response from server. HTTPcode: -4
7/20/22 12:29:36 PVoutput: started
7/20/22 12:29:42 grid: Started
7/20/22 12:29:42 grid: Last log entry 07/20/22 12:29:20
7/20/22 15:26:41 Heap memory has degraded below safe minimum, restarting.

** Restart **

SD initialized.
7/20/22 19:26:44z Real Time Clock is running. Unix time 1658345204 
7/20/22 19:26:44z Reset reason: Software/System restart
7/20/22 19:26:44z Trace:  9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:1[4], 1:2[5], 9:0[5], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:6[1], 1:6[2], 1:6[2], 1:6[2], 1:6[3], 1:5[21], 1:6[4], 21:0, 21:1, 21:10, 21:10
7/20/22 19:26:44z ESP8266 ID: 567551, RTC M41T81 (68)
7/20/22 19:26:44z IoTaWatt 5.0, Firmware version 02_07_05
7/20/22 19:26:44z SPIFFS mounted.
7/20/22 15:26:44 Local time zone: -5:00, using DST/BST when in effect.
7/20/22 15:26:44 device name: IotaWatt
7/20/22 15:26:44 HTTP server started
7/20/22 15:26:44 timeSync: service started.
7/20/22 15:26:44 statService: started.
7/20/22 15:26:44 dataLog: service started.
7/20/22 15:26:45 dataLog: Last log entry 07/20/22 15:26:40
7/20/22 15:26:48 WiFi connected. SSID=<ssid>, IP=<ip>, channel=6, RSSI -57db
7/20/22 15:26:48 MDNS responder started for hostname IotaWatt
7/20/22 15:26:48 LLMNR responder started for hostname IotaWatt
7/20/22 15:26:48 Updater: service started. Auto-update class is MINOR
7/20/22 15:26:58 historyLog: service started.
7/20/22 15:26:59 historyLog: Last log entry 07/20/22 15:26:00
7/20/22 15:26:59 Updater: Invalid response from server. HTTPcode: -4
7/20/22 15:26:59 PVoutput: started
7/20/22 15:27:05 grid: Started
7/20/22 15:27:05 grid: Last log entry 07/20/22 15:26:40
7/20/22 16:56:12 PVoutput: System Solar, interval 5, freeload mode  
7/20/22 16:56:12 PVoutput: Start status beginning 05/17/22 00:05:00
7/20/22 17:26:58 Updater: Auto-update is current for class MINOR.
7/20/22 18:11:50 Heap memory has degraded below safe minimum, restarting.

** Restart **

That’s interesting. It seems to take about three hours to degrade. I’ll look into that.