How to open log files

Is it possible to export the log file so I can open it in, say, Excel? I could download and open iotawatt.log but when I open the file it’s (to me) giberish.

The log is in binary form, and way too large to download anyway. But I’m currently working on a query facility to generate and download .CSV files. Keep an eye out for it in a month or so.

Excellent! And thanks for the response.

Sorry if this is covered elsewhere:

Is there a way to put the SD card in another computer and read the data log files ?

I am installing 5 IotaWatt units at off-grid sites soon and one of them doesn’t have a wifi connection so I was hoping to be able to run it off-line and just collect data from the SD card.

Many thanks

If this doesn’t exist as a feature is there a reason? Otherwise I will attempt to make a way of doing it myself but I’m a bit out of my depth I think!

Thanks

There is not.

Yes, it would be very difficult to do and requires stopping and opening the IoTaWatt. The log file is in binary form and is a unique format.

The new query will support exporting a modest size CSV file. The major hurdle with getting this out is user interface. It’s a substantial JavaScript effort and browser compatibility with things like date/time specification is an obstacle. That’s also holding up a revised new graphing capability.

Even if you don’t have internet, it is advisable to be connected to a WiFi network. That being the case, you can run influxDB or EmonCMS on a RPi for short money and harvest the data that way.

Hi,

The stopping and opening of the IoTaWatt isn’t a big deal - in theory it will only need to be opened to extract data every 6 months anyway. But I totally understand that this is not what the unit is designed for.

In terms of being in binary form and a unique format I would still have thought a script could convert it to CSV but maybe I’m missing something.

So - am I right that, for the off-network locations it would be best if the person who arrives to extract data does so by arriving with a temporary router/hotspot which the IoTaWatt connects to allowing access via the web-browser where they can view graphs and potentially use the new query export function? If the unit has sat for, say, six months with no network connection will it still have been logging data to the sd card?

Many thanks

That something is possible doesn’t mean it’s practical. It would be possible to read the log file directly from the SDcard. It’s not practical to build an external reader because there are only a handful of users interested in doing that. The source is open and readily available. There’s a c++ class with methods that allow reading the records. If you used an ESP8266 or other processor with little-endian encoding, the values stored in the records could be extracted and processed. There is a lot of logic elsewhere in IoTaWatt to develop the various metrics that can be reported from the raw data. That would need to be replicated. The scripting system allows operating on the data to produce other useful metrics.

While it is also possible to run IoTaWatt disconnected from the internet, there are other issues that you will encounter. It isn’t designed for lights-out operation without connectivity. Just as the data retrieval is possible but not pracical, so it goes with running disconnected. The work needed to solve these issues isn’t supported by demand for the capability.

Given the number of requests for this type of operation, I recommend the problem be solved by adding an inexpensive router and an Rpi or other low-cost server to accumulate data for export.

Is this feature now available?
If so how do I generate the csv file?

Thanks

The currently available method of creating CSV data is with the graph feature. The next release will introduce a new query API that can produce CSV directly. A UI will follow sometime later. No schedule available but that’s what I have been working on.

Looking forward to it, thanks

I’ve in the meantime copied the graph data to a csv file and can see all the data, however I’m now confused as Excel doesn’t recognize the digital numeric value assigned to the date and time,

ie. 1569512995 and 1569513060 represents 17:49 26 Sep > 17:51 26 Sep, Length: 0h01 (65 seconds)

How do I decipher this 9 digit number?

Thanks

It’s the number of seconds since Jan 1, 1970, and commonly called “Unix time”. It’s possible to decode it in a spreadsheet, I’ve done it.

But by far the easier way is to select “date-time-string” as a CSV option in the graph program to generate a date that both you and a spreadsheet can understand.