PVOutput Support


#1

Hi Bob,

I have recently updated my branch of IoTaWatt to improve the PVOutput web server upload support. This was largely a rewrite to support async http and make the code a lot cleaner.

You can find my source on github:

Are you interested in accepting a pull request for PVOutput support?

If so please review and let me know of changes you would like me to make to improve it before it would be acceptable to be integrated.

Thanks,
Brendon.


#2

Hi Brendon,

Congratulations on being the first to write an output service for IoTaWatt. I’m sure it wasn’t easy. I have scanned over it briefly, but that’s not a trivial bug fix, it’s a lot of code. I had tried your approach briefly some time back, making the service a class, but it got really complicated very fast and I abandoned that approach in favor of the current Emoncms and influxDB services. I feel that the code is starting to stress the limits of the ESP8266 and I have to be careful about the unintended consequences of new things. Also, the service scheduler/dispatcher really needs to have finer granularity than 1 second to support more of these services.

It will be awhile before I can dig into it and try it out. I have a file from when I researched PVoutput a few months ago. As I recall, the post interval is pretty large, and there are two modes where contributors get more functionality and are allowed to upload more history. I had been ruminating on how best to handle that as I think the contributor mode is the most useful.

I’ll give you a shout out when I get a chance to explore it. BTW I didn’t see where there were any changes to the configuration app. Could you post a screenshot of what you did there?

Thanks,
Bob


#3

Thanks. It was ok, the hardware is the hard part the coding is fine :slight_smile: After your recent refactors for config app and async http I had to change a lot of code but it was worth it as it cleaned things up a lot. Your effort in doing this was well worth it thanks.

I originally had coded it the same as the existing services all in a single big function with a switch case state machine and static data for state. I thought it was a bit simpler to read this way but I can easily revert back to a similar style to other services if that is something you prefer.

The concept of the class is still the same as the original code, but I broke things into separate functions instead of all inline in the multiple cases in a switch block it calls functions from the switch and the static data in the Influx style is basically stored as members of the class instead and I have a single static class instance. So anecdotally it should amount to very similar machine assembled code (if in-lining is used).

If there are issues with code size or heap usage let me know and I can take a look. There are lots of things I can do. I will wait for feedback first though about what specific issues you would like me to resolve.

I understand, the ESP is quite limited in various resources. I noticed you spent a lot of time optimizing heap in your changes so tried to minimize it in this code as well though there are more trade offs that can be made to improve this further (reduce logging, remove batching, or recalculate for resend). I understand at some point you wont be able to keep adding services. If that becomes an issue maybe we can talk about producing a new set of “update classes” for the different output services or a single generic http service that can cover all existing cases using string template replacement.

That wont impact PVOutput since its shortest post time is 5 min but maybe the other services.

I understand it is a big change and I assume you already have a bunch of other things planned in the pipeline I am not particularly worried about time, but I would like to avoid having to refactor too much like when the async http change was made if there was no intent of adding it to master.

I also saw another user was after the functionality so thought I would post it in case it is useful.

I plan to use it regardless but currently I have to run my own update server and use my own private key to sign the updates. I wrote some python scripts to pack+sign update images or unpack and test signature of update images that are working fine for me locally.

Yes the post interval is quite large (smallest is 5 min typically I think the default was 15 min).

I went for the free option which will also work for the donator mode. From memory the API is extensible so the free mode API will also work for donators (though I may remove the 14 day history POST restriction to support it). The donator mode adds a lot of extra functionality like collecting other data like temperature which is beyond scope for IoTaWatt. The history for free mode is summarized after 14 days not lost. So you get detailed logs for 14 days in free and then from then on it is daily/monthly/yearly granularity in the PVOutput database. This doesn’t impact the upload of stats.

It may have been difficult to see the config code change. I isolated the PVOutput code as much as possible from the rest of the system the main interface to the module is just two functions:
void PVOutputUpdateConfig(const char* jsonText);
void PVOutputGetStatusJson(JsonObject& pvoutput);

You will find the call to PVOutputUpdateConfig() is made from the config app. Since you moved to the new table structure in the javascript it was much simpler to isolate my changes so you did a great job of refactoring the earlier code. Thanks :slight_smile: it made my job easier.

Here is an screenshot of the config app for IoTaWatt:

One thing I realized I haven’t really supported is 3-phase. I had considered changing the channel config for mains+solar to be able to use the Iota Script instead of a hard coded channel then I assume people would be able to configure for 3-phase (though a bit more involved). The other option is permit N channels to be specified for mains/solar in the config instead. Less flexible but simpler to use and debug IMO though I have not thought through the various 3-phase use cases.


#4

Yea, it does impact. Right now you can wait 1 second (return UNIXtime + 1) or you can wait zero seconds (return 1). So when an async I/O is outstanding and you are waiting for completion with return 1, you are competing with another service that might have something useful to do. At a minimum, it would be nice to be able to wait until after another sample interval (16.6 or 20 ms). I’m weighing two approaches to the problem: Just use the millisecond clock in scheduling; or more elegantly, introduce the notion of blocking semaphores where time is just one event. So you could for instance also wait for unblocking from an asyncHTTPrequest ready state change handler, or an event signaled from a transaction in the webserver. That’s all blue-skies, but I like to keep it in mind so as to leave the door open for it.

I haven’t looked closely enough at the code or reviewed my past notes to make that commitment.

There is a lot of interest in PVoutput. It is a simple (from the user side) way to organize basic usage and PV data in the cloud.

As I recall donator mode allows adding something like six additional data items, which combined with scripted output, can tell a pretty complete story. So I think that’s key. That said, it may be easier and cleaner to have two discrete handlers rather than a bunch of if-else everywhere. Once again, my notes have more on that, but I haven’t come back up to speed…

I saw that. The UpdateConfig is more or less the same as currently. The GetStatusJason is what I would expect with the class approach. That approach is cleaner than collecting the information in the webserver handler.

It’s still a beast. If I had the time i’d start over with jQuery and Bootstrap.

US split-phase also requires adding two mains. Scripting is the way to go on that, plus the extra outputs in donator mode would need to do that. It’s trivial to use inside the firmware. Using the calculator in the config app is a little more complicated. There are tweaks for different applications that make it a bit more fragile.

I’m thinking maybe a month to get to it. Right now I’m in hardware mode with a big manufacturing run in progress, and all software effort dedicated to new features for hardware changes and problem resolution.


#5

Any news on this one making into the main build?


#6

I am currently working on it. Unlike other cloud databases that timestamps with UTC, PVoutput uses local time, so I’m adding support for time zones with daylight time so I can get reliable local time.


#7

Fantastic news.

You already have the timezone setting so not sure what you mean by ‘reliable local time’. Hope its not too hard.

I assume you know this already but you can post a lot of data to the free pvoutput account when going via the API (been doing this on my current energy monitor for years)

I send temp, voltage, power generation, and power consumption

//https://www.pvoutput.org/help.html#api-addstatus
//Parameter Field Required Format Unit Example Since
//d Date Yes yyyymmdd date 20100830 r1
//t Time Yes hh:mm time 14:12 r1
//v1 Energy Generation No number watt hours 10000 r1
//v2 Power Generation No number watts 2000 r1
//v3 Energy Consumption No number watt hours 10000 r1
//v4 Power Consumption No number watts 2000 r1
//v5 Temperature No decimal celsius 23.4 r2
//v6 Voltage No decimal volts 210.7 r2
//c1 Cumulative Flag No number - 1 r1
//n Net Flag No number - 1 r2


#8

The timezone setting is static in current IoTaWatt and doesn’t change with daylight savings time (needs to be manually changed in GUI when DST starts/ends which is what I have been doing). It does make sense to use proper time zone implementation. The fixed integral offset is not really correct.

The stock IoTaWatt has no temperature measurement (though I plan to add a I2C temp sensor to my device later) so that field is skipped but yes the others are all fine.

I am not sure if Bob is planning to use my implementation as a basis or re-writing one that he is comfortable to maintain himself. But mine was only designed for the free account and I know Bob wanted scripting support to add extra for donation mode and also support 3-phase.

My PVOut implementation for IoTaWatt has been running fine for a little while now (with a few minor bug fixes I should really push soon) but it is not official. Below are the stats reported by my IoTaWatt:
https://pvoutput.org/intraday.jsp?id=61657&sid=54871


#9

@bcosta, @dotnetdan,

Brendon is right, and Daylight time is what I am working on now. It’s a very complicated procedure. PVoutput does talk about an “adjust time” feature that corrects standard-time entries to daylight-time, but I can’t seem to turn it on my PVwatts account. I haven’t given up on that, but getting daylight-time adjustment working in IoTaWatt would be a plus anyway.

I have looked in more detail at Brendon’s PVoutput class code. I’m almost convinced to go with the class approach, but there are so many things about that particular implementation that I want to change or extend, I may just rewrite it within a similar framework.

Issues are:

Adopting local-time with daylight time.
Resuming with last posted value after restart.
Support for donation mode features.
Use scripts (calculator) to produce all outputs.


#10

By the way, this should already work using the getstatus API in my branch unless you mean storing locally what you have previously successfully posted. I just query PVOutput what it thinks the last post was:
https://pvoutput.org/help.html#api-getstatus

The donation mode and script support however is a bigger job and not supported in my code.


#11

I was just looking at that last night. I’m still digesting your code. It’s a very different style where the functionality is fragmented over a lot of functions, so it’s taking me awhile to get my head around it.


#12

Hi Brendon,

I’m trying to understand the 23:59:59 status entry. When I look at your site linked above, I see that the last entry of the day is the first entry of the following day. i.e. the last entry for 10/11/18 is 11/11/18 12:00AM and the first entry for 11/11/18 is also 11/11/18 12:00AM. I don’t see anything about a 23:59:59 entry in the API documentation.

It’s hard to imagine that of the many devices that already upload top PVoutput, this issue hasn’t at least been documented. I have to wonder if we’re missing something. One thing I notice is that Add Batch Status service says an addoutput is generated for the last successful status update in the batch. Might that be how a day is finished?

Looking at your PVoutput site, I’m curious why Energy Used stops increasing every day at the same time Energy stops increasing. It increases in the morning when there is no solar generation, but not in the evening. I don’t see anything in the code to cause that, but you are more familiar with how that works.

UPDATE: Just realized I have been looking at an out of date branch.


#13

My first version was a whole lot simpler until I tried to work around this bug.

The fact that I was losing 5 min of data at the end of each day was very surprising to me and I was also surprised that I couldn’t find anything about it online. I assumed for a while I was just doing something wrong. I was planning to post to PVOutput forums and ask, but didn’t get around to it.

I tested a lot of scenarios manually using wget (I might be able to find the tests and result somewhere if they are helpful for you).

One thing I didn’t try though is using cumulative mode. Its possible this works ok as instead of us doing the per-day accumulation calculations ourselves PVOutput do it for us. But I read in the documentation of a significant limitation in the max range of the energy which prevented me attempting that option (I think it was a 250KWh limitation). I read it again just then and it is possible this limitation applies only to the non-cumulative value so that is certainly an avenue worth looking into.

My work-around was to POST the end of day stats in the second before the end of the day.

  • end-of-day post is 23:59:59
  • begin-of-day post is 00:00:00

The 23:59:59 entry gets rounded up to midnight in the reported stats by PVOutput though as it stores data quantized to 5 min instead of the original 23:59:59 so it appears as 12:00AM (prev day) at end of day and the begin of day post of 00:00:00 appears as 12:00AM (curr day)

The end-of-day stats and begin-of-day stats I post are the same instantaneous values but I report the final day cumulative value for the end-of-day and reset the cumulative value for the start of day.

I didn’t notice that before, thanks for pointing it out. That is most likely a bug on my part in the calculation of accumulated energy. I will try and check my logs later this week and verify (lost my dev environment recently so need some time to setup again). Its most likely something to do with the code I use to auto-adjust sign in case I configure the CT incorrectly.


#14

Thanks for the info. Looking at this stuff for a couple of days now, it wouldn’t surprise me that the last status has been falling through the cracks for everyone. I’ll email their support and see what I can get from them. Seems to me that they must be storing this data at UTC.

I’ve got local time adjustment with daylight time working now, so I’m digging into this.