In Australia it is common for houses to have two metered electricity rates: one ‘full tariff’ for normal household power and lights, and one ‘controlled load’ for off-peak hot-water heaters (tanks). When a smart meter is installed (as always happens when you get solar PV Installed), these houses will get a 2-line smart meter to measure both these supplies, the ‘main’ leg recording energy separately for both directions (import/export), since the buy and sell tariffs are different.
To keep track of energy usage with IoTaWatt (or similar), three CTs are sufficient: one on the meter’s ‘main’ output (I label it ‘Main’), one on the meter’s ‘controlled’ output (‘HWS’), and one on the solar inverter’s output (‘Inverter’). Between these three you can see how much energy you are generating and how much you are consuming.
But in terms of my monthly bills, what I’m consuming in my home is not of primary interest. What is of interest is how much of that energy I’m buying from the retailer, and how much am I exporting to the grid and being paid for! While I can see that (visually) on the graphs (see example below), I can’t create outputs for those two quantities.
So my request, if it can be added reasonably easily, is for the Outputs ‘calculator’ to include a provision for ‘conditional’ outputs to be created. Just how that is done is unimportant, but for the programmers amongst us, the C coding method shown below would suffice, and require only four additional keypad characters (the brackets are not needed, and the ‘if’ could also be omitted since the ‘calculator’ knows what is intended, from the question-mark):
Import (from grid): (if Main > 0) ? Main : 0
Export (to grid): (if Main < 0) ? Main : 0
Handling of import/export data is an ongoing issue that has a few current solutions, and is forefront in my mind thinking ahead.
First, to address your request for an equivalent to the ? function in calculator, you can already do what you are asking for.
The syntax of the calculator is very simplistic, and it only supports unary and binary operators. Also, there is no notion of data typing and operands and intermediate results are considered to be the in stated units of the script. So imbedded logical operators that would produce Boolean results are not supported.
However the script system (calculator) has two comparison operators. min and max are binary operators that will evaluate to the smaller and larger of their two operands respectively. So:
4 min 5 = 4
-2 max 0 = 0
-8 min 0 = -8
(Main > 0) ? Main : 0 is the same as Main max 0 (Main < 0) ? Main : 0 is the same as Main min 0
With respect to computing import and export, the real issue is what data the script operates on. If you apply the above import or export script to the net mains energy for a day, you will not get the result you are looking for. If you exported 20kWh and imported 24kWh in the day, the import function applied to daily net will result in zero exported and 4kWh imported. If you integrate the script over all of the 5 second data for the day, you will get the desired result.
If you make import and export scripts using the min and max functions, and plot them for a single day, they will integrate over all the data points in the graph, which for a day are 2 minute net intervals, you will get pretty good results, because the overwhelming number of significant intervals will be all import or all export.
If you try to do this for a week, still not real bad, but the interval is now 15 minutes and some import/export differentiation will be lost in the 15 miinute netting. If you try to do it for a month, it will be wrong.
I do intend to resolve this at some point in the future by providing some way to either accumulate the integrated import/export or to post-process save integrated data, but there are some ways to do that now:
Upload to PVoutput. The resolution there is five minute, and you can zoom out to weeks, months and years.
Upload to influxDB. You can upload the import/export data at 5 second intervals and influx will return the totals over any time period.
Accumulate import/export in IoTaWatt. There is a convoluted way to do this in IoTaWatt. It requires using an additional input channel.
Thanks for your informative reply. As I was trying to get to sleep last night a solution suddenly occurred to me. So I got up and rushed down to my home-office to try it out. I then saw your reply, with a different approach. I first tried mine, which worked fine (see two screen-shots below).
Then I tried yours. But it didn’t work so well, as my original suggestion (C-style conditional) contained an error – I forgot to negate the ‘export’ formula to make the export Wh positive! (I presume this is what you were alluding to when you said my daily export figure would be wrong. Fixing the sign corrects that.)
I did try to correct that, but found I could not negate a variable or expression in the Output ‘calculator’. The ± button placed a - sign before the default 0 in the entry field, but then I could not enter either a left-bracket or the Main Input – clicking those buttons did nothing. I also tried first inserting the Main Input and then clicking ± to negate it (as most hardware calculators would do), but that didn’t change anything either. So I gave up on the ‘min’ expression and reverted to my ‘abs’ method. This screen-shot shows a longer example, for almost 20 hours today (Sat) – very cloudy most of the day, until a late burst of sunshine at 4pm!
But your further comments about the accuracy of accrued values worries me… You seem to be saying that if I look at a longish period (which I will be doing when I want to check the supplier’s monthly invoice against my own measurements), the ‘accruals’ (effectively integrals) won’t be accurate because they are based on widely-separated samples. I really had expected that the firmware would behave, in this respect, more like Excel. In an Excel spreadsheet, all numeric values are kept internally to umpteen digits precision, even though your displayed values might only show (say) two decimal places. All calculations are done using the high-precision internal values, and only the final result is rounded for display.
So I had expected that the IoTaWatt firmware would similarly do all calculations on the maximum-resolution data (5-second intervals, and I presume/hope this is what is stored on the SD card), and only average or accrue these high-precision numbers for 2-minute (or whatever) intervals display on the graph and CSV table.
If that’s not what currently happens, I really hope you can get around to updating the firmware reasonably soon. I don’t know if you have a ‘donate’ page somewhere, but given the work you have put into this project for freeloaders like me to download and build my own, I’d be quite happy to make a donation if it helps with the software effort.
But where you are using it, in an output script, has the limitations on accuracy that I tried to explain above. The script is applied to net intervals, which grow larger as you zoom out in time. As implemented in the post that I just referenced, the algorithm is used on every sample taken, providing maximum accuracy and not subject to the zoom out problem. That is the solution alluded to in
Yea, that’s a quirk of the way the calculator works. The simple workaround is, instead of using the unary minus operator, just add the binary minus operator, as in:
0 - ( Mains min 0 )
It produces the same result.
Right, you need to integrate the 5 second samples rather than add them, which is what you get inherently with the way the IoTaWatt datalog works. Integration is a much more resource intensive activity. There are 518,400 five-second datalog entries in a month. Asking the ESP8266 to read and integrate those entries, while dedicating 66% of it’s CPU time to sample your current usage, would take a very long time. That’s why I say you can export the data to a database that has the resources to do that, or use the method above to effectively save running import and export totals.
The “donate” page is github. This is an open project. It has grown into a full time job just keeping the hardware going out the door. Money isn’t going to help, folks need to contribute solutions.
Thanks again for your patient answers and explanations. My 76YO brain is a bit slow these days, but I think I’m coming to understand what you’re trying to point me to. The linked thread about ‘grid-tie-inverter-negative-watts’ was also helpful.
It seems that in the IoTaWatt data logs (on the SD card), as well as the ‘measured’ (calculated) power values being stored every five seconds, it seems you also might store the accumulated power values too. When scaled according to the sample interval, I guess these could represent reasonably good Wh values for each CT channel.
It also seems, from what you’ve said, that ‘output’ values (calculated from inputs) are not stored in the logs. So I think what you’re implying is that ‘accrued’ graph (and downloaded CSV) values will always be accurate for inputs, since thay are just read from the stored accruals, whereas accrued outputs over a long period won’t be accurate because they are accrued at the display time from widely-spaced data points. Have I understood that correctly?
But let’s assume that I adopted the hardware method in the linked post, except I did it for my ‘Main’ feed from the meter. So I’d have (say) Main_net and Main_abs values being logged every 5-seconds. But neither of those is what I’m after, which is ‘Import’ and ‘Export’ Wh. To get those I’d have to create two outputs using those two inputs.
Doesn’t that put me back where I started, since accrued outputs won’t be accurate over long periods? My previously-specified import/export outputs were also based on inputs logged every 5-seconds, just as these new ones are, so how would it be different?
For the sake of anyone reading this thread now or in the future, this thread has come to be about how to get accurate discrete solar import and export numbers over timeframes longer than a day. Net import is readily available over any time period, and net is what most users with “net metering” are billed by.
IoTaWatt has the data needed to compute long term discrete import/export, but it’s not available in a form that can be quickly extracted and analyzed. Doing so integrating, or adding, all of the individual 5 second datapoints stored in the IoTaWatt datalog, which is just not practical when talking about the half-million or so such points in a month.
Above in this thread, uploading the detail data to a database is offered as a solution. That is fully supported by IoTaWatt. There are other solutions as well, and one of them is to use a non-intuitive method (I called it convoluted above) to save discrete data that can be used to quickly retrieve import and export for longer periods directly from IoTaWatt.
Actually, no. IoTaWatt doesn’t store power values every five seconds. It only stores the accumulated power values as cumulative Wh since the datalog was created. Also included in each 5 second datalog entry is the total accumulated hours that have been monitored.
So for any given interval, given the two cumulative Wh and the cumulative measurement hours, you can easily compute:
_
Wh = Wh_end - Whstart
Hrs = Hrs_end - Hrs_start
Average_Watts = Wh / Hrs
Outputs are scripts. They describe how to compute the desired data as a function of the inputs data in the datalog. When you retrieve an output, you are retrieving the related input values and processing them to produce the desired result. So if you change an output, it will not only change the value of that output going forward, but also historically when you look back. The changed script of the output will be applied to the unchanging historical inputs values.
When you defined import and export as:
What you are doing is essentially the same thing, except at different granularity. “Main abs(absolute value of Mains)” is the same thing as “Main_abs” above, and “Main” is the same thing as “Main_net”. But when that script is evaluated for a monthly period, those data values are computed using the net power each hour. Your net power for the hour including sunrise could be 500Wh, while the import could be 1800 Wh and the export could 1300 Wh after the sun hits the panels.
When you move the computation of Mains_abs down to the 5 second level by not allowing negative values, and the computation of Main_net by allowing negative values, you are essentially using data integrated at 5 second intervals. In the example above, main_abs for the hour interval would show 3100Wh and main _net would show 500 Wh.
That’s wonderful, Bob – I think I understand now. It helps to know how and what IoTaWatt stores in its log files, which is not clear from the ‘manual’ (IoTaWatt Documentation) available from GitHub. (Speaking of which, I’ve converted the latest [Revision 86cd5c62] to a single hyperlinked PDF, if anyone is interested.)
So I can solve my problem with a simple hardware mod (more about this shortly). The following is just for ‘information exchange’…
I’m surprised that you guys (USA) are typically billed by net energy! Does that mean if your net was negative (more export to grid than import) for a billing period that the supplier would pay you for your net export at the full kWh rate? Wow, if that happened here my solar system would be paid for in a few years instead of several!
In Australia’s early ‘solar’ days (~10-years ago), the federal government subsidised feed-in tariffs such that suppliers were offering 40¢/kWh feed-in, while we only paid 20–30¢/kWh for what we import (buy). With the rapid uptake of domestic roof-top solar as a result, the government (might have been a different party in power?) quickly withdrew that subsidy, and left the FIT to market forces. So the situation now is that we still buy energy for 20–30¢/kWh, but suppliers (varies from one to another) only pay 0–18¢/kWh FIT (I’m talking AUD, of course):
So solar systems now take longer to pay for themselves, but should still do so comfortably within the warranty period.
For me to use your suggested hardware solution to generate Main_abs, all I need do is run a wire from the ‘hot’ side of CT1 input (‘Main’) to the corresponding side of (say) CT12 (the last default CT input on my board, and thus least likely to ever be used) with its burden removed. Then I just configure CT12 with ‘Allow negative power value’ NOT ticked, and I have the second hardware measurement I need for Import and Export calculations.
Incidentally I did note your earlier mention of using external websites to do this process, but I don’t wish to avail myself of such services right now.
On a related topic, I tried downloading the iotalog.log file, but it was not in human-readable form. In case I want to do some computations on my captured data (e.g. in a spreadsheet), I’d like to know the format of the stored data. Are you able to explain it to me without wasting too much of your time, or else point me to the code block where IoTaWatt writes the data record every 5-seconds? Thanks.
I cant speak for the whole USA. It varies greatly from state to state. My limited experience is with northern New England - NH and MA. Massachusetts is very progressive, and has had true net metering incentives in place for years. A few towns have municipal electric and use a FIT, but most have the major utilities and true net metering. Rather than pay you for a credit balance, they hold it, usually for a fixed period of up to a year. The majority of the power is produced in the summer, and it can be carried over to winter for increased heating use. Massachusetts has other incentives as well, but they have been diminishing for new installs as the price of new equipment goes down and the cost of grid power has increased.
You will need to read that out of the code on GitHub. The datalog related code is aptly named. Also be aware that the ESP8266. Is little-endian so you may need to do some rearranging if using a big endian processor to read the file.
