Emoncms local install advice

Cliver · August 20, 2022, 8:13am

Hi all,
I have my IoTaWattb kit on order. I now want to prepare a local DB for my data. I think I have settled on Emoncms as my prefered option. Happy to be convinced otherwise though
I could do with some advice on the best machine to install it on.
My options in order of preference are:

Install my Asustor NAS - Searching this option seems it may be problematic?
Install on a RPi.
Install on an existing windows server running other things.
Any other suggestions?
I am fairly confident “fiddling” in linux etc. although I do nee an idiots guide.
Any thoughts, user experience or advice appreciated.
I get my solar install completed next week so looking forward to being able to monitor things at a deeper level than “the app” gives me.

SandyB · August 21, 2022, 1:43am

Hi Cliver,
If you decide on Windows, let’s talk. I have emoncms running under iis with no problems (that ican remember!)
Basically it needs an appropriate web.config.

Cliver · August 21, 2022, 5:48am

Thanks Sandy, I dug out an old RPi and have installed it on that for the time being to see how I get on with it.
I’ll give you a shout out if I decide to try Windows.

khaney · August 21, 2022, 1:58pm

I ended up using influxdb over emoncms because I also wanted to use grafana for graphs and alerting and it has gone very well. If you have docker running somewhere I’d suggest using docker. Perhaps on the NAS?

I’m running influxdb and grafana in docker on a Windows machine, but may move it to my Synology NAS which is also running docker.

I have one docker-compose.yml with my influxdb and grafana information, and do a
docker compose up --detach
To start things, and
docker compose down
To shut them down.

If you haven’t used docker there would be some learning curve there, but once you start using it you won’t go back.

Cliver · August 21, 2022, 2:26pm

Thanks khaney. I’ve never used docker but think I may need to give it a go now.

Cliver · August 22, 2022, 1:56pm

@khaney I can’t seen to get docker compose to run on my asustor nas. Does it run on your synology?
Does it have to be installed seperately?
I did try using Portainer as a front end to docker but ended up going CLI as I couldn’t get it to work.
I’d kind of like to try emoncms and influxdb is seperate containers so I can try both.

khaney · August 22, 2022, 9:49pm

compose isn’t a necessity, I just find it useful to have all the commands, environment variables, ports, etc. in one place (because the next time I’ll need to start it from scratch I’ll forgot something!)

try docker-compose - it was originally one command. at some point, depending on the version of docker, it was made a built in command to docker, so if
docker compose
doesn’t work, try docker-compose

just double checking the synology, I had to use docker-compose.
In a later version of the docker cli the compose command was added to use instead of docker-compose standalone.
Overview of Docker Compose | Docker Documentation

I do have portainer running on both my windows docker and the synology… what sort of problem did you have? I just created the volume, and I think I used the default launch command. windows needed a little tweaking if I recall though.

you may need to make sure your user is in the docker users group… I did have to play with that on the synology (linux). otherwise you need sudo to run the docker commands.

Cliver · August 23, 2022, 9:10am

Many thanks Kevin, I’ll have a look at it later. I’ll let you know how I get on.

Cliver · August 23, 2022, 10:21am

If I install via portainer and try to browse to the emoncms web page I get the following…

Error: environment var ‘MYSQL_HOST’ not defined

Error: environment var ‘MYSQL_DATABASE’ not defined

Error: environment var ‘MYSQL_USER’ not defined

Error: environment var ‘MYSQL_PASSWORD’ not defined

Error: environment var ‘MYSQL_PORT’ not defined

Error: environment var ‘REDIS_ENABLED’ not defined

Error: environment var ‘REDIS_HOST’ not defined

Error: environment var ‘REDIS_PORT’ not defined

Error: environment var ‘REDIS_PREFIX’ not defined

Error: environment var ‘MQTT_ENABLED’ not defined

Error: environment var ‘MQTT_HOST’ not defined

Error: environment var ‘MQTT_USER’ not defined

Error: environment var ‘MQTT_PASSWORD’ not defined

Error: environment var ‘MQTT_BASETOPIC’ not defined

Error: environment var ‘PHPFINA_DIR’ not defined

Error: environment var ‘PHPTIMESERIES_DIR’ not defined

Warning : Redis::connect() expects parameter 2 to be int, string given in /var/www/emoncms/index.php on line 39
Can’t connect to redis at {{REDIS_HOST}}:{{REDIS_PORT}} , it may be that redis-server is not installed or started see readme for redis installation

Cliver · August 24, 2022, 4:10pm

I have it running in the container. My only problem now is I get an Apache forbidden error when I try to open it from a PC on my network. Can’t see what’s wrong with my permissions but there must be something. More digging needed.
I have also discovered that the version of emoncms on the docker hub is way out of date.
Think I’ll have a look at influxdb as an alternative. I have emoncms running on a RPi anyway so I can trial it on that for the time being.

Cliver · August 25, 2022, 7:39pm

Still trying to get emoncms running. As far as I can see the only problem I’m having is the Apache forbiden error. Or, at least if I can get it to open in the browser I will be able to see any other problems
I have a Docker group and it has RW rights to everything in /Docker /Public and /Web
Are there any permissions I need to set up within Docker?
Also I notice the port settings are 8080:80, is that right?
I attache a snapshot of what portainer is showing me for clarity.
Any pointers appreciated as I’m at a loss

overeasy · August 25, 2022, 7:56pm

You might try going over to the OEM forum to ask about installing Emoncms. They have some very knowledgeable people there. It’s best not to mention IoTaWatt as that often elicits a boilerplate response directing you back here.

Cliver · August 25, 2022, 8:28pm

Thanks overeasy, I’ll try on there.

khaney · August 26, 2022, 12:37pm

sorry, catching up on your messages.
I probably won’t be a lot of help specifically with emoncms, but your best bet would be to look through the container logs to see if that helps at all (you can see them from portainer).
When you get the permission error, that when you browser to port 8080 from your pc to your nas address? when that happens, is there a corresponding error in one of the logs (looks like it has 3 services?)
re: 8080, your nas probably already has it’s own web server, so port 80 was likely in use… port mapping (in this case 8080 to 80) is how that is handled (i.e. use different ports). the container will always use 80 internally, but it can be anything (available) from the outside.
interesting the docker version is out of date… maybe their docker support isn’t as good (or they have another container registry with a newer version).
the fact that you are getting an apache error when hitting http://:8080/ does seem to indicate you are hitting the service, so I’d think it should be logging something with the reason.

if you can ssh into the nas, you could also do
docker logs --follow emoncms-docker-web-1
to “tail” the log from the container. ctrc-c to stop.

Cliver · August 27, 2022, 7:08am

Just going through the pointers above, many thanks for them.
The first thing I checked was my docker version.

interesting the docker version is out of date

Where are you getting that from khaney? According to my nas it’s version 20.10.17, which is the latest I can find?

Cliver · August 27, 2022, 1:25pm

I have decided to change tack as I’m not progressing very far. So, as I have Emoncms installed and ready on a RPi I am going to try and get influxdb and grafana running on the nas so that I can try both.
Can I ask you @khaney to expand a bit on how you got it running on your setup please?
Are you using the official influxdb image?
If so how did you get grafana to work with it?
I did find this image which seemed to make it easy but it’s well out of date and not supported. The dev does have an updated and supported version on git hub which looks attractive but I can’t figure out how to get a git hub repository installed in docker.
So, if you can suggest my best plan of action (with a simple idiots guide to get it installed, set up and running with the ability to update in the future that would be most appreciated.
As far as I can figure I need to create/get hold of, a .yml file and run docker-compose on it? Is it really that simple? But, of course, getting the right info into the yml file is another matter
FYI I do have another container with Logitek Media Server installed and running succesfully. So I know docker is up and running OK.

Cliver · August 27, 2022, 1:50pm

So, I’ve been digging some more and I found what I think will work but, I’d like an expert eye cast over things before I run it.
I found an up to date image on the hub which seems to do what I want. I have what I think will install it if I run it in docker.
Could an expert cast an eye and make sure I’m not going completely off the right track please?

docker run -d
–name docker-influxdb-grafana
-p 3003:3003
-p 3004:8083
-p 8086:8086
-p 22022:22
-v /volume1/Docker/influxdb:/var/lib/influxdb
-v /volume1/Docker/grafana:/var/lib/grafana
–restart=always
philhawthorne/docker-influxdb-grafana:latest

khaney · August 27, 2022, 5:46pm

Here is what my NAS docker-compose looks like:

Note that I didn’t do anything specific re: finding versions, so for grafana I am defaulting to latest (looks like I currently have version 9.11 from the logs, but does look like I picked a specific version (2.3.0) for influxdb. 2.4.0 is the latest from a few days ago, so I suppose I could upgrade. I did this for both windows and the NAS… had to move to the NAS to the other day as resources were somewhat constrained on the PC and the two running constantly were impacting performance. Moving from windows to NAS was just a matter of changing the volume mapping (windows would have a c:\foo\bar format, vs /foo/bar on the NAS since it’s linux.

you’re probably better off using the specific version in the docker-compose yml, so you don’t inadvertently upgrade accidentally to a version that might need some steps to upgrade. I probably had that version in there because I found a recent example that worked!

If pulling latest or those specific versions doesn’t work for you on the NAS it may be because of the architecture they are running on. for example, raspberry pi would be arm vs amd for linux and presumably my NAS (intel based).
If you look at the tags page you’ll see that the latest ones are 64 bit which may not be available for arm (looks like arm is only up to version 1.8).

Note the shm_size limits are kind of arbitrary (and maybe too much), haven’t looked into the limit, that was mostly left over from another docker compose file I had.

I created a grafana and influxdb/database folder and made sure those folder were writable by docker.
I also added a /backups volume so I could dump a backup of the db and copy to the new server (used that when I went from windows to NAS). Note to self, remember to stop iotawatt upload to influxdb when doing the backup so you don’t lose some data next time!

one other note, if you don’t have the image locally when you do

docker-compose up --detach
to start things, it’ll pull the latest version (since I didn’t specify) or the specific version, but subsequent runs will use the local image. if I wanted to use a newer version of grafana, I would do

docker-compose down
docker image rm grafana/grafana-enterprise

then either do a pull of the latest images, or just docker-compose up again and it’ll fetch it.

and you can do these things via portainer too (well, maybe not the docker-compose part)

khaney · August 27, 2022, 6:06pm

A note about configuring the influxdb data source in grafana, since they are running together in docker via compose, they are on the same “network” in docker, and have similar network ip addresses. I realized just recently that the “hostname” of the container is the docker name in the compose, so when I configure the address for influxdb in grafana, I can just use http://influxdb:8086 and it’ll find it. I was previously doing it the hard way, using http://host.docker.internal:8086, which points to the machine running docker (so basically coming out of docker, hitting the host, going back into docker!)

you also set the organization id,api token and bucket that you get from influxdb organization pages.

Cliver · August 28, 2022, 9:39am

Wow, thanks @khaney I’ll look at all that in detail.
I did bite the bullet and ran the code I posted above as a command.
I fugured it can’t do much harm and I can delete the container and start again if it all goes wrong.
It appears to have installed fine and runs I can access grafana. Influxdb access is through chronograf though but it seems to work OK. I’ll play with that for a while and see how I get on. I don’t have my IoTaWatt yet so have no data to feed it as yet.