From my understanding, if the IotaWatt losses the wi-fi signal it would continue monitoring power and saving it in the SD card.
I have some questions:
Will the unit upload data to influx once the wifi connection is restored? (Just trying to confirm)
If the backlog is large, how will data be uploaded? At what rate?
Why could the unit stop saving data when the wifi connection is lost?
Here’s the relevant info about this case:
Last data points are on November 20 at 17:27:30. From the CSV output:
The message log shows the wifi disconnection on November 20 at 17:27:31 so it makes sense, but again, I would expect logging to continue with no wifi. The log also shows the unit reset a few minutes before:
Yes, it will upload after WiFi isrestored.
If the backlog is large, it will use larger buffers, more than bulk load, to send more measurements in each post. The rate depends on the number of measurements in each interval, as well as other factors like transmission time and how fast influx responds.
I looked at your log, and I’m not exactly sure what happened. Clearly there was some WiFi drop off. The first event recovered ok, and it kept logging. Then there was an exception. I have seen this before. It happens in some low level esp8266 code after a WiFi failure. It’s usually not a big deal because the unit restarts and moves on.
In your case, for some unknown reason, it spent six days in the WiFi manager code, and then It apparently resolved without a restart. I think there’s more to the story because there was a power cycle a few minutes later.
Nevertheless, this is all definitely WiFi related and in the core or other included code. I believe you have some other units running there as well. If so, what do their logs show for that period?
You are using 02_03_16. The latest general release is 02_03_18, which has newer versions of the core and some of the underlying support code. The newer release has been very solid, and if you look at the forum support topic, there have been very few problems with it. The ESP folks have been working on the newer lwip2 and it’s getting very solid. So I’d encourage you to set auto-update to MINOR and keep current with the improvements.
Thanks for replying. So far, we’ve updated the version now.
Sorry. I should probably have given a few more details. Here they are:
At some time on 11/20 power to the access point, EMILab, was removed (without a warning)
At some time on 11/26 we restored power to the access point, maybe just before 17:30.
At 17:37 we power cycled the IotaWatt. The LED was solid-red before this.
I got a similar problem reported today, and I want to follow up on something came out of that report. Was the device that you used to configure the IoTaWatt within wifi range when the power went off? I’m looking at a scenario where the IoTaWatt comes back up in AP mode before the WiFi router. Then the device that was originally used to connect to the IoTaWatt AP, which might ordinarily wait to connect to the WiFi router, instead connects to the IoTaWatt because it’s available and the WiFi is not. They remain connected indefinitely until something else happens to break them up. Is that a possibility in your case?
If you mean when the Iota was power cycled, then yes. But that’s when things started going well again because the access point was working already.
The whole thing was as follows. Iota was logging and AP was working
At some time on the 20th power to the AP was removed
At 17:04 on the 20th there was an Exception Reset
At 18:27:34 on the 20th there was a Software/System restart because “WiFi disconnected more than 60 minutes”
The Iota got stuck on 11/20/18 18:27:38 Connecting with WiFiManager.
Next message in log is: 11/26/18 17:30:04 No WiFi connection.
This is, apparently, when the AP was powered again.
Note that the device (laptop) for configuring the Iota was probably within range at this point.
The laptop’s wifi was connected to the AP to check for internet access
Tried to access the Iota from the laptop without success (LED was red)
Power cycled the Iota and green led was blinking rapidly (wifi connected)
Searched for the Iota in the network (we can’t use http://Iota3.local, so we use Bonjour software)
Found the Iota, with a different IP than before, btw.
I mean on the 20th, when power to the router (when I say AP I mean the IoTaWatt in AP mode for configuration) was removed, and more specifically at 18:27:38 when the IoTaWatt was trying to connect with WiFi Manager - was that laptop or any other device that might have been previously connected to the Iotaxxxxxx SSID in the vicinity.
When the IoTaWatt starts up, if it cannot see the configured SSID that it uses to connect to a local network router in STAtion mode, it will activate AP mode as SSID Iotannnnn so that the user can connect to it and configure a different network. If the original local network becomes available before you connect to the AP, it’s supposed to make that connection, stop the AP mode, and move on. What happens if it makes a connection before the local network becomes available is the case I’m exploring. It’s possible it just hangs there waiting for a dialogue to configure a new network.
There are a few hundred issues outstanding against the WiFi Manager package. I sympathize with the author because there are so many different routers and clients out there it’s probably impossible to test. I think he may have lost interest in keeping up with this fast changing environment as well. I have elevated the priority of making an alternative.
Most clients rely on DHCP (Dynamic Host Configuration Protocol) to get their IP address assigned by the router. A client can try to use a fixed IP, but if it conflicts with the DHCP pool, there could be a conflict. If you would like your IoTaWatt to have a fixed IP address (which is handy when accessing from the internet), then you would need to configure whatever facility your router offers to fix it’s IP assignment based on the MAC address.
No. Not a chance. The Iotas are only used by us involved in the project and we’re usually like 400 ft away from that particular IotaWatt.
As for the problem you’re investigating, if we find something we’ll let you know. That’s a very interesting problem and in hindsight something we probably faced before while doing some early tests, when we would sometimes plug and unplug the IotaWatts several times a day and also change routers.
Yeah, we’ve been using DHCP in all of out units, but I don’t recall any of them changing their IP just because the wifi was down, and we use the same DHCP server for all of our network. Anyway, I though that could possibly be related to the issue. Guess it’s not.
Could be. Might indicate something weird happened to the router, or could just be business as usual. I’ve already put in changes so that it only invokes WiFi Manager after a power fail restart. That fixes a known problem and maybe helps this issue as well because it will not invoke WiFi Manager after a “WiFi disconnected more than 60 minutes” restart. Every little bit helps.
Since we’ve been having access (wifi/internet) problems with the IotaWatt that stopped logging data (Iota3) we decided to set up another unit next to this one. The access problems have been always present, but we wanted to correlate setting up this second unit (Iota4).
We just had a disconnect issue today (12/13) and we couldn’t access neither unit from our cubes. The Iotas are on a different subnetwork and we believe there’s network connectivity access from there (hence the failed Influx posts; the Influx server is in our cubes) but there’s also the WiFi disconnections.
Since I really don’t understand most of the messages in the log, you’ll know more of what happened.
Consider that both units were restarted early today for a time zone change (from -5 to -6) we had neglected since the end of DST… Also, the LED in both units was blinking green at the time we went to check them at about 14:30… but there was no internet access (connecting from a laptop to EMILab AP); although access was intermitent. Also, at one point both Iota4 and laptop could be reached from out cubes, but not Iota3, so we restarted it (that’s the restart at 14:35:39).
The new release in ALPHA has DST support. Also, it’s not necessary to restart to change time zone. Actually, except for the new PVoutput support, all of the internal timing is done with UTC except for the message log which is converted to local-time.
blinking green means it’s sampling, and also that it believes it is connected to WiFi. It would blink red if sampling but not connected to WiFi. In either case, the sampling is being recorded in the datalog, as appears to be the case here. All of the data backlog should have gone to influx when the connection was re-established.
HTTcode -4 means that a connection could not be established. It is possible that you have something interfering with your WiFi. I can see that both IoTa has problems at ~12:31 and again at ~13:14 where Iota4 was disconnected and Iota3 simply failed to connect.
I’d suggest you try using different Wifi channels. That is set in your router. Conventional wisdom recommends 1, 6 and 11. (you are on 9). It might be as simple as that.
I think we can’t use automatic DST since DST here is different than in the US.
We tried changing channels without success. The issue may be the router we’re using. We’ll try to change it. As you may recall the Iotas there are #3 and #4
Now we had brief issues with the Iotas 1 and 2 that we have in the original installation and the diesel power plant next to them. Anyway, those had been very stable (after October 2: 2 Exceptions, several power failures, 1 reset from “WiFi disconnected more than 60 minutes”) until recently.
First thing I’ve noticed, now happening in the 4 units, is the new message: “Incomplete HTTP request detected, id 7, restarting”. I don’t know if that’s just a new message for a previously known issue or if it may be an actual new issue, but it’s been happening on units 3,4 that have the disconnection issues. Here’s the logs just right after the upgrades for units 1 and 2. Notice how the influx issues occur during several days after the upgrade until the “Incomplete HTTP request” message on both units at very similar times. The behavior after the upgrade and is exactly the same in both units for almost 3 weeks.
On the morning of the 18, we had no access to these two units and later noticed our access point (IotaNetwork) was knocked from the desk where it was placed (but not disconnected). This happened before 9:00 in the morning and at about 9:00 the AP was moved back to the desk, but without resetting it. The units couldn’t be accessed.
Later in the day, at aroung 18:00 we went again to check the installation. The wifi signal of the AP was available from our laptop but we couldn’t actually access it. Iota1 was apparently in AP Mode (red-green-green) and Iota 2 was fast blinking green. From what we did and the logs this happened:
Apparently when the AP was moved back to the desk, Iota1 changed its status
12/18/18 06:34:50 Connecting with WiFiManager.
12/18/18 09:02:49 MDNS responder started for hostname Iota1
Note that during the day, there were several “Incomplete HTTP request detected, id 7, restarting.” cases
A few minutes after 18:00 we power cycled the AP and status changed to slow blinking red. The log shows the following, however, it seems as if the change to “slow blinking red” happened just after we tried to access the unit.
12/18/18 14:27:29 Connecting with WiFiManager.
12/18/18 18:13:02 No WiFi connection.
At 18:19 we powercycled Iota1, the Iota connected to the AP and it stayed there.
For Iota2 it looks much simpler.
12/18/18 18:09:54 WiFi disconnected.
12/18/18 18:11:07 WiFi connected. SSID=IoTaNetwork, IP=10.0.0.215, channel=4, RSSI -59db
A few notes and questions:
It seems as if the Iotas (or at least the logging) sometimes are getting stuck at the following point, sometimes for a few minutes, sometimes for days.
12/18/18 14:27:29 Connecting with WiFiManager
However, I’m not sure if this normal.
Apparently some of the connectivity issues have increased since version 02_03_18
Connectivity problems look very different for Iota1 and Iota2, even when they are very close together and only a few meters away from the access point.
Where are you and what is your DST rule? The DST rule is encoded in the config file when you activate DST. If your rule is different from US eastern time, it can be added to the rule tables. There are currently four rules available for different areas.
There was a problem where logging just stopped, and there was no indication why. It was always related to poor WiFi and was up to the user to notice and restart the IoTaWatt. The problem seemed to be related to HTTP requests not completing when WiFi drops. This all seems to happen at a lower level beyond my control. So this WDT was added to identify when a HTTP request gets completely lost. Since there doesn’t seem to be anything I can do about it, and I don’t know exactly what happened, the question is what to do when I detect it? I could ignore it and just retry the operation, but that is risky and could have lots of unintended consequences. So I just restart, which resets everything. There is no loss of data because sampling was ongoing.
Right. Lets be clear, both systems worked perfectly for two weeks after the upgrade to 02_03_18. From 11/28/18 to 12/13/18 at 14:23. The WiFi dropped for a minute on 12/10/18 at 13:59 but both systems reconnected uneventfully.
Then the wheels came off the wagon. There are a series of connection failures on 12/13 and 12/14 which effect both systems at about the same times. Then all quiet for a couple of days when it starts up again on 12/17 - 10:57, 12:13, 12/42, 13:01 - both systems, both recovered fine. This is clearly a router or interference issue external to the IoTaWatt. Then sometime around 13:46 or so, something happened that caused both systems to lose TCP context and this time the HTTP requests were completely lost, the WDT detected it, and both systems restarted.
Now after restart, neither system was able to connect to the router, so they hung waiting for the router from 14:02 (09) until 16:26 when the router apparently came back online and both systems connected. You lost data during that period. That issue is fixed in the latest release now in ALPHA.
Half an hour after the connection at 16:26, the WiFi problems started again for iota1. iota2 was unable to get started because it could not query for the last update, so it does not have subsequent connection problems because it is no longer trying to connect.
Bottom line is that everything worked great for two weeks after upgrade to 02_03_18 until something external happened to cause a lot of WiFi issues. When it does connect, the RSSI is great, so I have to wonder if there is something else in the environment that is causing the problem. What changed on 12/13?
Right, that’s a problem with WiFi Manager, which is out of my control. What I have done in 02_03_20 is only use WiFi manager after a power failure, so there should be no significant loss of sampling after an incomplete HTTP request restart.
As I said, it worked great for about two weeks after that. Something else happened on 12/13.
They look exactly the same to me. Where they seem different on 12/17-12/18 it’s because iota2 failed to get the influx service running and so wasn’t doing HTTP transactions. iota1 was and getting a lot of errors with the poor WiFi.
I’m in Mexico. There are several rules here, depending on the region (border) and states. The one that applies to us is
First Sunday in April clock moves from 2:00 to 3:00
Last Sunday in October clock moves from 2:00 to 1:00
Sounds reasonable.
Good. We’ll update the units.
We changed the time zone to account for the DST end (which was on OCT 28 but we forgot about). BTW, only Iota4 was manually reset after the change.
However, on the 12/14 , we changed the wifi channel from channel 9 to 6 based on your suggestion. Of course I’m not blaming you; we should have checked other channels and behavior. It’s weird however, that the change to channel 6 improved the RSSI from -72db to around -64db
OK. Is this 02_03_20 the current ALPHA version with the improvements?
I’ll code and test that rule. What is your standard time offset?
Couple of things. Something happened on 12/13 and restarting after a time zone change could not cause these WiFi issues, especially on two different systems. The log shows that you are on channel 4, not that it’s a problem, just different than what you said, and I don’t see any change between 11/28 and 12/17 in the log.
Those are informative messages indicating that an internet time server rejected time requests because it sensed that your devices were doing so at an excessive rate.
This is a problem that crops up from time to time in installations where there are multiple IoTaWatt. Apparently the servers are unable to discern that these are different devices.
I run into that here as well with sometimes as many as eight IoTaWatt running at one time. They will retry and eventually sync up to UTC, and the real time clocks typically don’t vary by more than a second or two per day anyway.
It does become a problem when trying to setup a new IoTaWatt where the RTC hasn’t been initialized. It can take an hour or more to get a timesync in that situation.
I hope you’re fine. Long time no “see”.
I need to ask a few questions but decided to get back in touch in this post since my first question is about the “Connecting with WiFiManager” issue I’m quoting above.
[First an update:
-User AG19 , who was working with me in our monitoring project with IotaWatt has left our company since the end of March. It’s just me in the project now.
-I was finishing another project that used most of the time but I’m back to this one since late April.
-I read about your call for reviewing IotaWatt. I sure will do it.
-The units have been doing great.
-I’m doing a summary and analysis about our findings on the monitoring we’ve been doing with the IotaWatts since September. Although we didn’t use as many units as we would have liked I think we’ve collected valuable information with them. Thanks a lot !!! ]
Well, and back to the technical questions… Three of out Iotas stopped monitoring for several days. I went on vacations for Easter Holidays on 4/17 and returned on 4/24 but I only noticed until 4/29 that the units had stopped collecting data since early on the morning of 4/17. Apparently our router was disconnected. So I reconnected it and power cycled the Iotas (or so I remember!)
The culprit seems to be the “Connecting with WiFiManager” issue I quoted above. I’m guessing this is still out of your control but I just want to double-check and add some comments.
The issue went like this: there were several restarts because WiFi was disconnected for more than 60 minutes then the “Connecting with WiFiManager” message. Here’s the log.
As you can see, time was 6:02:55 at the time of the last log entry at around the time data was no longer monitored. The time of this particular restart was at about 6:02:50
Questions:
What’s the meaning of the reset reason listed ?
4/17/19 12:02:51z Reset reason: External System
Although the reset occurred at about 6:02:50 the data collection (CSV output) stops at 07:02:50
-Maybe I asked you about this before but why the time difference? For reference, time configuration is:
Time zone: -6
Allow Daylight Time is unchecked
-One more question, once the data collection restarts on the 29, the units first detect and log 0V (in the voltage inputs of course) then the actual voltage. Wasn’t that fixed before? I believe we talked about it once.
It’s a power cycle. IoTaWatt detects power failure with the RTC which takes all the guesswork out of it. The ESP code will sometimes be power-on and sometimes external-system, but it’s a power cycle.
It’s actually 06:02:50. The local time conversion is being done in your browser which is using DST.
I did make some changes for that. Not sure when. I’ll make a deal: If you run the latest release I’ll look into old problems that seem to recur.
