Heap memory has degraded below safe minimum, restarting

Icedmang · June 13, 2022, 12:28am

Good afternoon,

tldr; 2022-06-12: I’m troubleshooting a heap memory issue. The solution isn’t here, yet. Please let me know if something else can be enabled for troubleshooting. I am uploading to influxDB_v2 and hoping to help. This is not urgent.

I’ve come across a new issue and seen it discussed a few times. It’s not urgent, but I wanted to update real-time and will edit here when I have more information. My only intention on posting the amount of details is to help others navigate and hopefully reduce some common questions.
This issue occurred after my recent resolved issue with WiFi connectivity.

Some related topics to my current findings:

What I’ve done so far based on what I could find in other posts:

I’ve modified my config.txt via the ACE Editor. You cannot edit the config.txt file directly through the ACE Editor, by design, to help protect users from errors. (e.g. do not be too confident, the quotes can cause problems)
The file cannot be edited directly followed by a CTRL-S (to save the file). I received an http error code of 403 (permission issues) so additional steps are required.
Added "heap": true, to the config.txt file (I followed the instructions by editing a file on my local machine and named it config+1.txt, running through a JSON lint tool, and then uploaded and restarted my IoTaWatt so the config+1.txt file receives a syntax check and becomes the new config.txt). Personally, I used jsonlint-php in Ubuntu (installed via apt install jsonlint. The command-line tool is called jsonlint-php).
NOTE: Copy/Paste’ing the "heap": true, from this post into the config.txt file is going to give you the odd quotes and cause an error. Make sure you change them.
Restarted IoTaWatt and confirmed the new config.txt contained the "heap": true, setting in the "influxdb2" section.

root@ubuntu:/# jsonlint-php /tmp/config+1.txt
Valid JSON (/tmp/config+1.txt)
root@ubuntu:/#

And this is what my "influxdb2" section looks like in config.txt after restarting with an uploaded config+1.txt file with the added entry for heap (your url entry may be different):

"influxdb2": {
		"type": "influxdb2",
		"revision": 17,
		"heap":true,
		"postInterval": 5,
		"bulksend": 10,
		"url": "http://192.168.10.1:8086",
...

Here’s my message log:


SD initialized.
6/08/22 14:59:41z Real Time Clock is running. Unix time 1654700381 
6/08/22 14:59:41z Reset Reason: Power-fail restart.
6/08/22 14:59:41z ESP8266 ID: 482274, RTC M41T81 (68)
6/08/22 14:59:41z IoTaWatt 5.0, Firmware version 02_07_05
6/08/22 14:59:41z SPIFFS mounted.
6/08/22 14:59:41z Local time zone: +0:00
6/08/22 14:59:41z device name: IotaWatt
6/08/22 14:59:44z Connecting with WiFiManager.
6/08/22 15:01:51z HTTP server started
6/08/22 15:01:51z influxDB_v2: Starting, interval:5, url:http://192.168.10.1:8086
6/08/22 15:01:51z WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=2, RSSI -39db
6/08/22 15:01:51z MDNS responder started for hostname IotaWatt
6/08/22 15:01:51z LLMNR responder started for hostname IotaWatt
6/08/22 15:01:51z timeSync: service started.
6/08/22 15:01:51z statService: started.
6/08/22 15:01:51z Updater: service started. Auto-update class is MINOR
6/08/22 15:01:52z dataLog: service started.
6/08/22 15:01:52z dataLog: Last log entry 06/08/22 14:59:35
6/08/22 15:01:53z influxDB_v2: Resume posting 06/08/22 04:30:40
6/08/22 15:01:54z Updater: Auto-update is current for class MINOR.
6/08/22 15:01:56z historyLog: service started.
6/08/22 15:01:56z historyLog: Last log entry 06/08/22 14:59:00
6/11/22 11:54:47z WiFi disconnected.
6/11/22 11:54:55z WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=7, RSSI -34db
6/11/22 12:59:06z WiFi disconnected.
6/11/22 12:59:14z WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=11, RSSI -39db
6/11/22 14:11:59z WiFi disconnected.
6/11/22 14:12:03z WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=6, RSSI -35db
6/12/22 01:38:49z WiFi disconnected.
6/12/22 01:38:53z WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=10, RSSI -39db
6/12/22 02:06:07z WiFi disconnected.
6/12/22 02:06:12z WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=1, RSSI -37db
6/12/22 02:10:22z Updater: Invalid response from server. HTTPcode: -4
6/12/22 02:30:56z WiFi disconnected.
6/12/22 02:31:07z WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=8, RSSI -44db
6/12/22 04:35:00z Heap memory has degraded below safe minimum, restarting.

** Restart **

SD initialized.
6/12/22 04:35:02z Real Time Clock is running. Unix time 1655008502 
6/12/22 04:35:02z Reset reason: Software/System restart
6/12/22 04:35:02z Trace:  18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:2, 18:3, 18:4, 18:5, 1:6[6], 1:3, 1:3, 1:6[1], 1:6[3], 1:5[21], 1:6[4], 21:0, 21:1, 21:10, 21:10
6/12/22 04:35:02z ESP8266 ID: 482274, RTC M41T81 (68)
6/12/22 04:35:02z IoTaWatt 5.0, Firmware version 02_07_05
6/12/22 04:35:02z SPIFFS mounted.
6/12/22 04:35:02z Local time zone: +0:00
6/12/22 04:35:02z device name: IotaWatt
6/12/22 04:35:02z HTTP server started
6/12/22 04:35:02z influxDB_v2: Starting, interval:5, url:http://192.168.10.1:8086
6/12/22 04:35:02z timeSync: service started.
6/12/22 04:35:02z statService: started.
6/12/22 04:35:02z dataLog: service started.
6/12/22 04:35:02z dataLog: Last log entry 06/12/22 04:35:00
6/12/22 04:35:06z WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=8, RSSI -42db
6/12/22 04:35:06z MDNS responder started for hostname IotaWatt
6/12/22 04:35:06z LLMNR responder started for hostname IotaWatt
6/12/22 04:35:06z Updater: service started. Auto-update class is MINOR
6/12/22 04:35:06z influxDB_v2: Resume posting 06/12/22 04:09:25
6/12/22 04:35:10z historyLog: service started.
6/12/22 04:35:10z historyLog: Last log entry 06/12/22 04:35:00
6/12/22 04:35:30z Updater: Invalid response from server. HTTPcode: -4
6/12/22 05:55:31z WiFi disconnected.
6/12/22 05:55:39z WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=2, RSSI -37db
6/12/22 07:35:18z Updater: Auto-update is current for class MINOR.
6/12/22 16:08:37 Updater: Auto-update class changed to MAJOR
6/12/22 16:08:38 Updater: Auto-update is current for class MAJOR.
6/12/22 16:08:59 Restart command received.

** Restart **

SD initialized.
6/12/22 23:09:00z Real Time Clock is running. Unix time 1655075340 
6/12/22 23:09:00z Reset reason: Software/System restart
6/12/22 23:09:00z Trace:  1:3, 1:1[4], 1:2[6], 9:0[6], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:1[6], 1:2[7], 9:0[7], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 10:2, 10:3
6/12/22 23:09:00z ESP8266 ID: 482274, RTC M41T81 (68)
6/12/22 23:09:00z IoTaWatt 5.0, Firmware version 02_07_05
6/12/22 23:09:00z SPIFFS mounted.
6/12/22 16:09:00 Local time zone: -8:00, using DST/BST when in effect.
6/12/22 16:09:00 device name: IotaWatt
6/12/22 16:09:00 HTTP server started
6/12/22 16:09:00 influxDB_v2: Starting, interval:5, url:http://192.168.10.1:8086
6/12/22 16:09:00 timeSync: service started.
6/12/22 16:09:00 statService: started.
6/12/22 16:09:00 dataLog: service started.
6/12/22 16:09:00 dataLog: Last log entry 06/12/22 16:08:55
6/12/22 16:09:05 historyLog: service started.
6/12/22 16:09:05 historyLog: Last log entry 06/12/22 16:08:00
6/12/22 16:09:05 WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=2, RSSI -37db
6/12/22 16:09:05 MDNS responder started for hostname IotaWatt
6/12/22 16:09:05 LLMNR responder started for hostname IotaWatt
6/12/22 16:09:05 Updater: service started. Auto-update class is MAJOR
6/12/22 16:09:05 influxDB_v2: Resume posting 06/12/22 16:08:15
6/12/22 16:09:06 Updater: Auto-update is current for class MAJOR.
6/12/22 16:26:40 Restart command received.

** Restart **

SD initialized.
6/12/22 23:26:42z Real Time Clock is running. Unix time 1655076402 
6/12/22 23:26:42z Reset reason: Software/System restart
6/12/22 23:26:42z Trace:  1:3, 1:3, 1:3, 1:1[13], 1:2[14], 9:0[14], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:3, 1:3, 1:3, 1:1[14], 1:2, 9:0, 9:0, 8:4, 8:6, 8:8, 8:9, 1:2, 1:3, 10:2, 10:3
6/12/22 23:26:42z ESP8266 ID: 482274, RTC M41T81 (68)
6/12/22 23:26:42z IoTaWatt 5.0, Firmware version 02_07_05
6/12/22 23:26:42z SPIFFS mounted.
6/12/22 16:26:42 Local time zone: -8:00, using DST/BST when in effect.
6/12/22 16:26:42 device name: IotaWatt
6/12/22 16:26:42 HTTP server started
6/12/22 16:26:42 influxDB_v2: Starting, interval:5, url:http://192.168.10.1:8086
6/12/22 16:26:42 timeSync: service started.
6/12/22 16:26:42 statService: started.
6/12/22 16:26:42 dataLog: service started.
6/12/22 16:26:42 dataLog: Last log entry 06/12/22 16:26:40
6/12/22 16:26:47 historyLog: service started.
6/12/22 16:26:47 historyLog: Last log entry 06/12/22 16:26:00
6/12/22 16:26:47 WiFi connected. SSID=<redacted>, IP=192.168.10.233, channel=2, RSSI -38db
6/12/22 16:26:47 MDNS responder started for hostname IotaWatt
6/12/22 16:26:47 LLMNR responder started for hostname IotaWatt
6/12/22 16:26:47 Updater: service started. Auto-update class is MAJOR
6/12/22 16:26:47 influxDB_v2: Resume posting 06/12/22 16:26:15
6/12/22 16:26:49 Updater: Auto-update is current for class MAJOR.

NOTE: I changed auto-update from MINOR to MAJOR in case a new firmware update was available.

Current heap metrics:

overeasy · June 13, 2022, 12:58am

Your degraded heap problem is due to the very poor WiFi RSSI. This only occurs when there is very poor WiFi signal and seems to be a memory leak associated with WiFi error handling.

Heap leaks cannot be found with the influx heap monitor. Only the result can be observed. It takes a lot more than that to find the cause. I don’t recommend anyone use the heap monitor as it is of little value to ordinary users.

I also don’t recommend users manually update their config.txt files. Before the changes that this procedure defeats, a common problem was lost and/or damaged config.txt files. It practically never happens anymore. While using a JSON Linter can eliminate syntax errors, it does not check for content errors. The config app error checks most of the content and should be used to modify the config.

Icedmang · June 13, 2022, 3:05am

Thank you. Am I incorrectly reading the RSSI? I thought the closer to 0, the better. I thought -40 to -30 dB was excellent? The WiFi router is about 3 meters/9 feet away from the IoTaWatt (no impediments)

overeasy · June 13, 2022, 12:35pm

You’re right about the RSSI, worked several problems in that session and the poor RSSI was not yours. That said, degraded heap is associated with WiFi issues and your disconnect/reconnect channel switches would qualify as the probable cause.

I see the degraded heap restart after several channel switch disconnects. I thought you had resolved that.

Icedmang · June 13, 2022, 5:45pm

I understand, you have a significant amount of responsibility that we all appreciate, so thank you.

It’s the first time I’ve come across the heap issue. I didn’t attribute the WiFi toggling for that brief amount of time as a symptom because I haven’t come across the heap issue when WiFi toggling was much, much worse. My RSSI seems excellent. The WiFi toggling, although a lot less, is something I still need to investigate. I’ve attached my iotamsgs.txt (152.5 KB) to see the entire context.

I’m OK with ignoring it and moving on unless you would like me to dig deeper.