I’m trying to access data via the IoTaWatt Query API, and am having problems with a limit on the amount of data that can be accessed with a single query. For example, a request for time and two power channels fails if the request is for more than four hours of data. The failure mode is strange: the response code, 200, indicates success, but the received data is truncated. At first I assumed this was due to the poor WiFi signal (the RSSI is around -85), but I found that the response is always truncated to a length of 100188 bytes, whereas one would expect some random variation if the cause were just a flaky network connection. Any troubleshooting suggestions would be appreciated.
There is a response limit of 100K bytes in the firmware to limit the amount of time the web server can monopolize the processor to the exclusion of other services.
You can break it up into multiple queries, or if you don’t need fine resolution increase the grouping size to reduce the amount of output.
Thanks, I couldn’t ask for a simpler explanation! I have two suggestions:
- It would be useful if the response code gave some indication that the query resulted in an over-length response. (But perhaps this is only detected when the response length actually reaches the limit, rather than advance, so it isn’t possible to base the response code on this condition?)
- The truncation at a fixed byte length has the undesirable effect of making the received JSON data unparseable. Would it be possible to truncate at the end of the current row once the length limit is exceeded so that the received response is still valid JSON?
Both good suggestions.
Right. The HTTP response code is sent in the response header, then the data is sent using the chunked protocol. So except for doing a rough estimate, there is no way to predict the condition and indicate in the header. Although I’ve never used it, I think the protocol has some mechanism for appending headers at the end. I doubt it would be useful as the HTTP code is not exactly in a header, but more of a prefix.
That’s a possibility. It would require moving the length check from the web-server to the query response generator, but that’s no big deal. That would also open the door to sending the “range” JSON element at the end with the correct values. That would be helpful for a caller doing multiple requests to divide and conquer.
I’ll put that into the queue. It takes quite awhile for changes like this to make it to release, so I’d advise another solution for immediate needs. BTW/ the ESP32 prototype has no limitation and can return megabytes of data, quite fast to boot. But that’s another vaporware solution right now.
Understood. Thanks for the fast response.