Is there a way in graph+, emoncms or other where I can download all the data.
I do have a sample rate every 5 seconds and wanna download a whole week of data. What do you reccomend me to do?
What do you mean specifically when you say all the data? Do you want:
-
5 second resolution or averages for a greater interval?
-
Watts, Volts, Amps, VA, VAR, Wh, VAh, Hz, PF?
-
Do you want these metrics for all inputs?
Also, how often do you want to do this?
How to approach it depends a lot on the answers to those questions.
To answer your other question, you can download any data that you plot in Graph+ by selecting CSV at the bottom and then either copy or download.
I need the data for a whole week, with a resolution of 5s. Needing Watts, V and PF… I want these metrics for all my inputs. As I’m doing research I need to download every week and to have all the data.
Graph+ only allows to download around 10 minutes with a 5s rate, so if do it on that way I need to do it thousands of times, which is not useful. I’m trying to doing on the query but I’m able to do it for like 9 hrs of data.
OK, got it.
Two options:
-
Get yourself a Rpi, install influxDB V1.x and set the IoTaWatt to upload (you can set to start uploading from any date in the past to capture your history), then use influx query to get the dataset you want, whenever you want it. The Rpi (or other host) should be able to rip that out pretty fast.
-
White a program to issue consecutive queries and append the data to a file. To be safe, query in smaller chunks.
You are looking for a lot of data, and the IoTaWatt’s ESP8266 is always busy sampling power (67% of the time). The web server is not asynchronous, so it blocks for the duration of a query and stops sampling. When you upload data to influxDB (or emoncms or PVoutput), it’s done asynchronously and so doesn’t interfere (much) with sampling.
If the limits were removed from query and it took 2 minutes to serve up the estimated 2MB of data, you would have a two minute hole in your sampling data. So taking smaller chunks allows the IoTaWatt to do a round of sampling between requests.
On the ESP32, where this is not a problem, I have queried several MB of data with no effect at all on sampling. Different architecture and async web server.
Just to be clear with the Rpi, do you mean Raspberry pi?
And second, how can I removed this limits?, I Don’t mind losing some of the sampling data if I’m downloading the rest of the information.
I would go with the influx solution to collect, store and query the data. I found it relatively easy to set up.
Influx (it needs to be version 1.8.x not version 2) has versions for windows and Mac as well as Linux based systems.
Info is at Downloads
1 Like
Yes
You can’t. I would need to change it in the next release. I’ll give some thought to how to do that. Perhaps add an extra parameter to the query with a data limit that defaults to a safe number but lets you set it or defeat it.
I just did a test on a bench unit with the limit removed. I downloaded time, voltage and one input for one week at 5 second intervals. It took 4.27 minutes and produced 3.88 MB.
It also stopped sampling as you can see here:
So the data rate is about 15 Kb. I’ll make the change but do not anticipate putting out a release until at least January.
Thank you a lot for the info!
I will be looking on the update on January !
On the meanwhile I’ll will keep trying in another way !
I have created a dos Script, to Put Data Info Excel. IT dies Download CSV in 15 min intervall vor one year. If Simeone jedes thus, i can Share.
Horst
1 Like
I just got a query for this, so I have whritten down this process:
http://horstwessel.eu/pv-analyse/
I hope, this helps. If you need moe information, please ask.
Horst
1 Like