High server load

Hello,

We have emoncms (10.8.8) installed on our server (Debian Buster). We use 20 IoTaWatts (“IW”), connected to emoncms. Unfortunately, the server load is super high when IW are connected, making the server unusable for the other services installed.
image

As you can see on the screenshot above, the database (MariaDB) is the problem. When I stop the emoncms server (just by renaming the emoncms folder into emoncms2, so that Apache cannot find it anymore), the load goes back to normal (around 0.5) while other services are still running (notably Nextcloud). This is why I’m sure that the load issue is related to emoncms.

It is not confirmed yet, but we have the feeling that the load is higher when several IWs are uploading data that was not uploaded yet (for example after an interruption of the internet connection of the IW).

I don’t know if it’s related to IotaWatt or to emoncms itself. Would you have any advice to solve this?

Thanks in advance!

Here is a few more information about our config:

Debian 10 (buster)
Server version: Apache/2.4.38 (Debian)
PHP 8.0.11 (cli) (built: Sep 23 2021 22:03:11) ( NTS )
mariadb Ver 15.1 Distrib 10.3.29-MariaDB
CPU: AMD E350D x86_64 (2 cores)

Sorry I can’t be of more help, but this is an issue I think better addressed by the Open Energy Monitor folks (EMonCMS) and/or a Linux/Apache expert. I had suggested to the EmonCMS folks to put in an upload throttling feedback header in Emoncms to reduce load from many IoTaWatt recovering from an outage, but that was not adopted.

There may be configuration parameters in Apache that would work to reduce Emoncms load.

I seem to recall reading years back that early versions of Emoncms used MySQL for the data store but that it became too slow. Subsequently the phpfina and other methods were developed to store the feed data more efficiently. The metadata for those feeds is probably stored in the sql database, but I don’t see where that would be a bottleneck unless they update with each write.

As I understand it, the actual data store in the Emoncms.org server uses queues to handle the traffic across multiple threads. I don’t know if the OSS version can do that.

The way inputs are processed into feeds is also quite laborious with the input process being a chain of PHP handlers. If you are doing a lot of that, consider doing what you can in th3 IoTaWatt using the output Scripts (calculator). It is much more efficient.

Thanks for your quick feedback! I also asked to EMONCMS folks, but they think it’s an IotaWatt issue :wink: High server load - #3 by Robert.Wall - Emoncms - OpenEnergyMonitor Community

Regarding your suggestion of using the calculator, I’m not sure to have fully understood: we need to monitor each input individually on emoncms, so I guess that the calculator wouldn’t be helpful in this case, right?

Yeah, that’s a boilerplate that guy hands out when he hears IoTaWatt. You need to ask Trystan. Sure it’s IoTaWatt generating the load, but the problem is that your server with Emoncms can’t handle it. There are a few others over there that could speak to what can be done, if anything, to make your server run more efficiently.

Not sure if you have this, but I believe most use Redis to cache the SQL database and increase throughput. If you do that however, make sure you use one of the persistence options.

If Emoncms isn’t up to the job, maybe you should be looking at influxDB. I’m using influxcloud with influxDB2. It runs on AWS, fast as can be, and is very inexpensive. My bill with 2 units and six month history was $1.02 last month. I use a RPi with NGINX to send the data over https.

Indeed, I use Redis. May I ask you what you mean with the persistence option?

For that, you just ask google. “Redis Persistence”. I’ll save you the trouble https://redis.io/docs/management/persistence/

As to why you would use it, as explained to me by Trystan at OEM, the Emoncms inputs are maintained in Redis. As you post data the inputs are updated to contain the latest value as well as the timestamp of that value.

When you restart emoncms and redis without persistence, the inputs are wiped out. When IoTaWatt queries emoncms to find out the time of last update, it isn’t provided. So if you have an upload history date in your configuration, it will try to upload from that date. If you don’t have an upload history date, it will start from the current time and leave a hole.

This is easily avoided with Redis persistence. If you use RDB persistence and specify a snapshot say every ten minutes, the worse that can happen is that IoTaWatt will start uploading data from ten minutes before the restart.

Is it possible that your problem is that you are restarting Emoncms due to a power problem or other issue, and the IoTaWatt are trying to upload months of data because they have an upload history date and no Redis persistence?

Thank you very much for the explanation! RDB persistence is activated by default in Redis, so I think it’s OK

In fact, my problem is probably related to a wrong configuration of redis, which made it unusable by emoncms. The weird thing is that emoncms was showing that Redis was running well, but in fact it wasn’t. I’ll make a few tests tomorrow to see if it solved the problem and I’ll keep you updated.

Thank you again!!!