EmonService: Invalid response

Yes, exactly

Also Yes

Is there any way you can verify that the transactions make it to the Apache/Emoncms? In other words, I’m wondering if the response you are getting is from NGINX or Apache/Emoncms.

Emoncms doesn’t use a username/password for posting data, it uses the APIkey to identify and validate. That said, if you are logged in from a browser and send a URL, it is accepted without an APIkey, presumably because the browser sends an authorization.

Does NGINX have any sort of authorization capability that might be active here?

(Sorry for the late response - out of town)
I don’t have an easy way to get a serial connection. But I ran a Wireshark trace on the local emoncms server.
iotawatt.xlsx (8.0 KB) (It started as a .pcapng, but I can’t upload those, so I zipped, then renamed to .xlsx).
It’s a trace through an IoTaWatt restart.
The first few posts look good to me; there’s a GET (no. 11/12) and 3 POSTs that come in as (application/x-www-form-urlencoded) with Authorization: Bearer & (I believe) the API key. But then (at No. 67) there’s a POST that doesn’t show as (application/X-www-form-urlencoded) and doesn’t have the Authorization: Bearer … header, and it gets the “username or password empty” response. And then the same from then on.

1 Like

Thanks,
That was very helpful. I think I see what’s going on. I believe it’s a race condition triggered by near immediate acceptance of the asynchronous connection request, causing the request headers that are subsequently set to be discarded.

I think the local IIS and NGINX (@Simsala) are so fast that they accept the connection before IoTaWatt sets the headers. There’s no reason why the headers can’t be set after the connection is established. It’s a bug that can be easily fixed. The headers can be set anytime prior to the send, as that’s when the message is built, so I’ll change asyncHTTPrequest to do that.

If I can find or mitigate a nagging memory leak, I should have a new ALPHA out this week with this fix. Thanks again. I had not used WireShark. Very nice tool. I’m sure it will come in handy going forward.

UPDATE: It’s not actually the immediate acceptance of the connect request, it’s because the previous connection was never disconnected and it is being reused. That’s a good thing. Emoncms.org has always disconnected. As I understand it NGINX, and I suppose IIS, have low overhead to keep connections open, so they do. After this fix, bulk update to NGINX and IIS should be quite a bit quicker than EMoncms under Apache.

1 Like

@SandyB, @Simsala,

If either of you can reproduce your failure conditions I’d be curious to know if release 02_03_09 has fixed it.

Mine’s not on 02_03_09 yet. I just restarted, but it still says:
updater: Auto-update is current for class ALPHA

Thanks,

Snafu with the update script. Should download 02_03_09 within an hour.

I think you’ve got it!!!
Another restart & it picked up 02_03_09.
No more “Invalid response” messages. Just waiting for all the data to send! (~10 days done, 1 to go)
(Getting a few “HTTP response -11, retrying.”, and “Retry successful after ? attempts.”. ? usually 3-7)

11 is timeout. It retries three times before issuing that message, then will keep retrying until it succeeds, logging the final retry count. If they occurred during the bulk upload of 11 days data, I wouldn’t be too concerned. If they keep coming during normal operation, there may be a communication problem. In any event, success after a few retries is still pretty robust, and it should have fully uploaded all of the backlog.

It seems to be working perfectly now; no messages since it finished uploading about 12 hours ago.
Thanks a lot!!

Thank YOU!
That Wireshark log was the key. Took about ten minutes to figure it out with that trace.

Sorry for the late response, I had to get my emoncms test instance up and running again, which caused some problems.

What can I say: Just awesome! It runs wonderfully with NGINX as a web server for emoncms. Thanks for the debugging and the super fast implementation in a new firmware.

That’s exactly what I love about the IotaWatt, someone reports a problem, it is taken seriously and analyzed and a short time later it is fixed and additionally some more functions have been added.