EmonCMS moved to AWS instance and RDS backend - failed

fbc · May 14, 2019, 11:57am

EmonCMS moved from a bare-metal server to an Amazon AWS instance with an RDS backend.

the values will periodically change however the inputs and feeds will all say inactive.

Before the move I updated EMONCMS to the latest and greatest.

I recreated the Ubuntu Server environment per the guide. I only didn’t go through the DB creation steps since I already had my own DB and just pointed the config file variable to it.
Logs Show:
019-05-13 09:24:16.127|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-13 19:45:50.940|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-13 19:53:58.960|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-13 20:24:50.947|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-13 21:38:27.019|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-13 22:58:28.353|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-13 23:21:28.644|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-14 00:03:46.869|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-14 00:20:17.924|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-14 00:49:32.964|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-14 00:58:49.929|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-14 01:19:12.171|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-14 01:31:36.915|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4
2019-05-14 06:45:33.436|ERROR|input_controller.php|{“success”: false, “message”: “Format error, json string supplied is not valid”} for User: 4

overeasy · May 14, 2019, 12:36pm

Hi,
This is the event that was on the OEM forum for a few days? It well could be that there is a problem with the Emoncms instance on AWS, but nobody over there in OEM responded. So I want to check to see if the IoTaWatt upload is working properly.

Could you post the message log and a screenshot of your IoTaWatt status display with the Emoncms and datalogs tabs expanded please? I would also like to have a screenshot of your Emoncms service configuration display in IoTaWatt with your write keys obfuscated.

Thanks

fbc · May 14, 2019, 12:38pm

YEs, this is the same even in the OEM… let me try to get you the information you need.

fbc · May 14, 2019, 12:49pm

I’ve sent this request to the remote office. I also updated the original post with log information on the EMONCMS side. My Database instance get hit with up to 150 connections. I assume these are the streams from the REDIS server running on the EMONCMS server.

overeasy · May 14, 2019, 1:20pm

OK, When I get the log info here, I’ll take a look. Understand that I’m not doing anything regarding Emoncms diagnosis. That’s not my area of expertise.

BTW/ In you post in the OEM site you say that you had the remote site change the IP address of the Emoncms server. Did you also have them change the write-key and, if using encrypted protocol, the account number?

fbc · May 14, 2019, 1:27pm

Since I used the same database I assumed everything would stay the same except the IP address. Why would the write key, encryption protocol and account number change if I’m using the same DB?

overeasy · May 14, 2019, 1:37pm

I don’t know. If you say they are the same, that’s fine.

fbc · May 14, 2019, 2:54pm

** Restart **

SD initialized.
5/07/19 21:34:47z Real Time Clock is running. Unix time 1557264887 
5/07/19 21:34:47z Reset reason: Software/System restart
5/07/19 21:34:47z Trace:  1:4, 1:1, 1:2[1], 9:0[1], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 1:4, 1:1[1], 1:2[2], 9:0[2], 9:0, 9:1, 8:4, 8:6, 8:8, 8:9, 9:3, 9:5, 9:9, 1:2, 1:3, 10:2, 10:3
5/07/19 21:34:47z ESP8266 ChipID: 7280075
5/07/19 21:34:47z IoTaWatt 4.x, Firmware version 02_04_00
5/07/19 21:34:47z SPIFFS mounted.
5/07/19 14:34:48 Local time zone: -7:00
5/07/19 14:34:48 device name: Iotawat1
5/07/19 14:34:48 MDNS responder started for hostname Iotawat1
5/07/19 14:34:48 LLMNR responder started for hostname Iotawat1
5/07/19 14:34:48 HTTP server started
5/07/19 14:34:48 timeSync: service started.
5/07/19 14:34:49 statService: started.
5/07/19 14:34:49 Updater: service started. Auto-update class is MINOR
5/07/19 14:34:49 dataLog: service started.
5/07/19 14:34:50 dataLog: Last log entry 05/07/19 14:34:45
5/07/19 14:34:50 historyLog: service started.
5/07/19 14:34:51 historyLog: Last log entry 05/07/19 14:34:00
5/07/19 14:34:51 WiFi connected. SSID=iota, IP=192.168.3.164, channel=1, RSSI -84db
5/07/19 14:34:52 Updater: Auto-update is current for class MINOR.
5/07/19 14:34:53 EmonService: started. url=99.xx.xx.xx:80, node=EDMONTON1, interval=10
5/07/19 14:34:54 EmonService: Start posting at 04/29/19 09:52:30
5/07/19 14:37:09 EmonService: HTTP response -11, retrying.
5/07/19 15:30:53 EmonService: Retry successful after 227 attempts.
5/07/19 15:33:27 EmonService: HTTP response -11, retrying.
5/10/19 08:15:58 WiFi disconnected.
5/10/19 08:20:08 WiFi connected. SSID=iota, IP=192.168.3.164, channel=1, RSSI -89db
5/10/19 09:16:05 WiFi disconnected.
5/10/19 09:16:36 WiFi connected. SSID=iota, IP=192.168.3.164, channel=1, RSSI -86db
5/10/19 09:22:24 WiFi disconnected.
5/10/19 09:23:17 WiFi connected. SSID=iota, IP=192.168.3.164, channel=1, RSSI -90db
5/12/19 00:45:08 Updater: Invalid response from server. HTTPcode: -11
5/12/19 02:45:11 Updater: Invalid response from server. HTTPcode: -11
5/13/19 06:09:51 EmonService: Retry successful after 3076 attempts.
5/13/19 06:12:16 EmonService: HTTP response -11, retrying.
5/13/19 19:48:59 Updater: Invalid response from server. HTTPcode: -11

overeasy · May 14, 2019, 3:56pm

It looks as though the communications with Emoncms are timing out. I can’t say why, but the WiFi RSSI is pretty poor.

That said, it does seem to be successful from time to time. The initial query of the Emoncms inputs indicated that the last post was on April 29 at 9:52. Is that when you switched to the new instance of Emoncms?

Since then, it appears to have advanced only a few hours, and that was probably mostly skipping frames in a last ditch attempt at error recovery.

Since you moved Emoncms, are you able to view data prior to April 29?

fbc · May 14, 2019, 4:04pm

Yeah April 29th was when the old server was shut down and moved to the new service. So I only have up to April 29th data in the emon database. I’m looking that feeds and their EPOCH times.

overeasy · May 14, 2019, 4:17pm

OK, I don’t know what’s going on, and given that the IoTaWatt is remote to you, its going to be hard to diagnose. I can try to access your Emoncms from here if you setup a new account and give me the IP and a write-key via PM.

fbc · May 14, 2019, 4:32pm

ok standby… on my way…

overeasy · May 14, 2019, 5:16pm

I was able to create the new node, but then unable to post to it. Pretty much the same as problem you are seeing. The posts to Emoncms are timing out.

When I query the input list, I can see that I was able to post at least one frame, so it’s not that it flat out not working, but when I do a query from my browser
https://xx.xx.xx.xx/input/list?apikey=…

It takes between 3 and 12 seconds to get a response. IoTaWatt waits 10 seconds for the initial query, but only waits a coupe of seconds for a response to a bulk/post. So something is going on in your instance of Emoncms that makes it slowwwwwwww.

If you read up on the emoncms post protocols, you can create a node and try yourself to post to it using your browser. When you figure out what the problem is and fix it, the IoTaWatt should upload the history. It may require a restart of the IoTaWatt if it has given up on Emoncms.

fbc · May 14, 2019, 5:21pm

Sorry, I had to upgrade the instance size from T2.micro to T2.Small because it was too damn slow. Try it one more time… the time out was probably because it was changing instance type.

fbc · May 14, 2019, 5:23pm

I’m seeing updates from you…

overeasy · May 14, 2019, 5:25pm

Yea, but its timing out.

That’s what you’re seeing on your account as well. I’m wondering if the whole thing is just jammed up from the other IoTaWatt(?s) posting. How many are trying to post to this instance of Emoncms?

fbc · May 14, 2019, 5:34pm

Six IOTAWATTS are on that instance…

overeasy · May 14, 2019, 5:37pm

So I’m wondering if it is overloaded. Are they all running and experiencing this problem?

overeasy · May 14, 2019, 7:13pm

I’m going to assume you’re looking into performance of the AWS instance. FBO anyone just tuning in or curious beyond the basics, this is an old problem for me.

There have been instances where Emoncms.org, for various reasons, has gotten behind in it’s queues to write the feeds. Invariably, the subject of IoTaWatt comes up. None of the Emon products have synchronous protocols to upload history after a failure or slowdown. So when Emoncms drops some stuff from other sources, it’s just gone and then business as usual. But IoTaWatt wants acknowledgement that it’s posted, and will keep sending it if it doesn’t get it.

So when the Emoncms queues get backed up, it naturally is with IoTaWatt data, because there are a lot of them sending a lot of data to Emoncms. Moreover, when an IoTaWatt gets behind, it sends larger transactions with more data, and it sends them as fast as it can.

It was once suggested that I slow down the history upload of IoTaWatt to Emoncms. My position is that if Emoncms will give return a metric of load, I would be happy to respond with an algorithm to delay based on that metric. But as someone who does a slow burn when I’m sitting at a red light for two minutes while there is no cross traffic to be seen, I reject the idea of waiting for the sake of waiting.

I have heard rumblings that the OEM folks are working on a store and forward solution for their node products. Something like that widely deployed may finally provide enough incentive to put flow control into Emoncms as I’ve suggested several times now. It’s simple to do and would turn a problem into a solution.

In the meantime, if overload is the problem here, there are two solutions that I see:

More horsepower in the AWS instance.\
influxDB

fbc · May 14, 2019, 7:17pm

I updated both the DB and EMON CMS instance and a test IOTAWATT in a remote location was just logged in and started to feed data… so I’m hoping once the other begin to do the same it will all come back online.