Availability of MQTT?

When is it likely that MQTT support will be available ?

There are lots of third party IoT platforms like Ubidots and Thingspeak around that would provide an easy way to make dashboards available online. Most have free accounts available with some limitations but would be an easy way to create some simple dashboards.

Iā€™ve got PVoutput and MQTT in the queue but have been first doing some overdue rework of basic subsystems. Probably next would be PVoutput.

IMO MQTT has limited utility without a generalized method to timestamp the data being exported. One of the most distinguishing features of IoTaWatt is the ability to maintain 100% upload integrity in the face of communication and server interruptions. So any MQTT implementation would only be a real-time feed subject to any delivery delays and errors of the broker and client. Without timestamps, QoS 2 makes no sense.

Itā€™s great that there are IoT platforms available to provide dashboards, but I can only see real-time dashboards as being practical with MQTT. So my priority is to get the timestamped services working first, like PVoutput.

So probably some months away.

1 Like

Understand - most important things go to top of list.

Looks like Ubidots and Thingsboard take a timestamp (optionally) as a parameter from the device so would allow a client created time to be used. This may avoid problems of delivery delay and errors effecting data integrity and Q0S 2 would be useful. Thingsspeak however does not allow this so looks like there is no generalized solution. Also the datapacket structures differ for each service so while they use MQTT there would need to be some way to create a custom string structure.

Thatā€™s exactly right.

Quick look at their documentation, it appears that they accept a json formatted data packet with value, time stamp, and a collection of ā€œcontextā€ name/value pairs.

So MQTT is just a way to deliver that data packet, they also support delivery using RESTful HTTP. The data packet is the same, so itā€™s really not about supporting MQTT, itā€™s about supporting their data packet format.

Iotawatt general framework for HTTP uploads iwould probably be more appropriate here as it works off the datalog and provides historical integrity with time stamped data. Either way, itā€™s another custom protocol.

There are both practical and physical limits to how many custom device protocols can be supported by IoTaWatt. Iā€™ve seen what happens when a device tries to become a Swiss Army knife. As a power monitor, Iā€™m trying to concentrate on keeping the mission limited in scope and concentrating on making a limited few external databases and the internal datalog as useful as possible.

I didnā€™t understand that there was not a standard payload for MQTT. I was thinking it would solve the problem of having to implement a different interface for every IoT platform but thatā€™s not the case.

Here is an interesting explanation.

A generic MQTT or HTTP implementation would need to have a ā€˜payload builderā€™ which makes it more complex but maybe there are librarys available to do this. Something for the future.

It gets complicated real fast when you add QoS 2 to the mix, and that only guaranties delivery to the broker and any subscribers at the time the broker receives the packet. Because the timestamp is encoded in the payload, the broker has no idea if it is a historical upload or a current measurement.

Probably better to have a process running on a more capable platform that takes data from, say an influxDB database, and sends out MQTT packets using some kind of payload format specifier.

It all depends on what you are trying to achieve. I think of MQTT as a many to many communication transport layer. It is great for sensors and actuators and control elements. Since it supports many to many communications without the need for the sender to do anything special, it makes it easy to add listeners. Since a listener doesnā€™t have to do anything special to get messages from multiple senders, it also makes it easy to add senders. This means you can add control elements very easily.

I use NodeRED for control and have many sensors and control elements. They all speak their own commands/status, but NodeRED makes it easy to have them play well together.

While you can use it save stuff, you really need to timestamp at the source and then would need to do extra work to meet the consumerā€™s requirements. So, I would not use it for data where you need to get every single sample.

MQTT will be a great forward step to be added to IoTaWatt. It will allow many type of sensors to be integrated into one single system it also allows IoTaWatt to be controlled from multiple platforms and applications example HomeAssistant and openHub.
The Idea will be very interesting if done!

After exploring how MQTT integrates with IoTaWatt, Iā€™ve moved doing anything with it pretty far down the list. MQTT is a transport mechanism that is used to support higher level communications. While the simple use case of sending a value may seem a standard use, itā€™s of little value in the primary mission of IoTaWatt. HTTP is a much more robust protocol and more importantly, all of the intended clients and servers (Browsers, Emoncms, influxDB) have native support.

Beyond lack of utility, there are major issues with adding MQTT:

  • Increased code and heap space for the supporting code.
  • There is no embedded support in the ESP8266SDK so would rely on another package. The most popular has quite a few issues outstanding.
  • Without a higher level layer, there is no provision for time stamping.
  • Message delivery beyond the broker is not certain or acknowledged without a higher level protocol layered on.
  • Without TLS or SSL, there is no authorization mechanism.

It all goes to product philosophy. I realize that there are some energy monitors out there that try to do it all. Iā€™ve tried to keep the IoTaWatt function more narrowly focused and concentrate on ease of use, reliability, low-maintenance, and full-function within the capabilities that are supported.

If MQTT turns out to have a lot of value (by that I mean some very useful capability that cannot be supported without it), Iā€™d be happy to work with someone to provide a secure feed to an external (RPi, ESP32) capability.

1 Like

FYI a while ago I ended up writing a quick and dirty Ruby script which polls InfluxDB for the latest values from measurements I care about and posts them to MQTT if the value has changed. I decided to post an empty value if the latest measurement is more than 10 seconds old so I donā€™t act on stale data. I subscribe to these topics in Node-RED to do automations and also have these set up as MQTT sensors in Home Assistant for easy monitoring. Works well.

Sounds like an easy solution. One way to make that more general purpose would be to poll the IoTaWatt directly instead of polling influxDB. I realize that I havenā€™t published the query API details yet, but that is available from the code and Iā€™ll be working on improving the docs soon.

1 Like

Looks interesting, and it could make happy multiple people and give them the chance to implement all their wishes :wink:

Can you tell me on which area is of the implementation is this :wink: ?

Are the one one webserver.cpp, handleCommandā€¦ ?

The API that I had in mind is in getfeeddata.cpp. Itā€™s modeled after the EmonCMS request, but modified to allow requesting multiple ā€œfeedsā€ in one request and doesnā€™t return hundreds of millisecond time stamps because the time is known.

Someone did a similar thing with MQTT a few months ago using the API for the status request, which returns damped values but is otherwise a simpler interface. That handler is in webserver,cpp

There are some vey good points in this, and I tend to agree.
A point to note, is that in EMONS there is a process to allow an input to be sent to MQTT.
Maybe this is a good compromise for some of your customers.

I agree that keeping it as simple, to ensure reliability, is key for a device like this.
The MQTT package I use for ESP8266 has an issue with subscribing, and causes a massive overhead to overcome.

Hi, now 2021 Jun 1, is there a MQTT news? Does it work now? I want to buy a sensor that will allow me to receive the data over MQTT so it can be easily integrated to Node Red and then influx and grafana. I want to know the current status before to take a desition. Pleas let me know the current status. ESP32 have a very easy way to do it but I dont want to deal with the hardware side to connect many CTs.

There is no MQTT output option. However,you can output directly to influxDB (v1 or v2j using HTTP.

I know that MQTT is not planned but you asked a while ago about reasons for MQTT over other protocols.

I think the biggest thing which is missing from the current uploaders is the ability to export ā€œreal timeā€ data. The uploaders are concentrated around historical data and often have a minimum delay based on the polling period. (E.g. 5 seconds for InfluxDB)

Missing is the ability to output the current instantaneous values. An example would be using the IotaWatt to drive excess energy diversion, where you need the imported / exported energy as close to real time as possible. MQTT is ideal for this due to its pub/sub nature but itā€™s probably also worth adding the ability to output a ā€œlast updatedā€ timestamp alongside any published values in case thereā€™s any issues.

Solar diversion is a bit of an issue currently as there are many products which can do it, from EV chargers to immersion heating controllers, AC batteries and dedicated thermal stores. They all handle it separately which means you canā€™t easily configure a priority or assign a quota to each diverter.

In my case, Iā€™d like to use my IotaWatt to capture the raw import / export information. I can then subscribe to this, add machine learning along with tariff information and diversion priorities to then command various diverters to use the appropriate amounts. (Iā€™d publish the outputs back onto MQTT which puts everything in one place)

The diverters can then subscribe to this directly, a microcontroller can subscribe and interpret it for them or you could even have a dashboard output this live diversion data from the same source.

Anyway so thatā€™s my request:

  • The ability to publish selected inputs or outputs directly to MQTT with no polling delay. (Each to their own topic)
  • The ability to throttle this so that it will only publish a new value if it has changed by a certain amount from the previous value, has changed sign or a period of time has passed since the last publish.
  • The ability to publish a ā€œlast updatedā€ timestamp to its own topic as a health-check, to ensure that stale data is not used. (Last Will Testament | VerneMQ)

As an aside, this could also be a mechanism to allow expansion of IotaWatts. If an IotaWatt could subscribe to MQTT and consume topics as inputs, a single IotaWatt could collate, store and graph inputs from multiple physical devices.

Have you ever heard the acronym SMOP? Thatā€™s Simple Matter Of Programming. I used to hear that a lot working with hardware engineers. The high level description always seems simple but the devil is in the details.

I would suggest that there may be no commercial use of the 8266 that works harder than IoTaWatt. It spends 2/3 of its time exclusively sampling voltage and current in 16.6ms or 20ms slices with 8.3ms or 10ms intervals between where it analyses the sampling results and does everything else including writing the datalog, servicing queries and other web-server requests, uploading data.

The uploads require an asynchronous http client, which I had to write as there is none for the esp8266. Without that uploading would block and suspend sampling for the duration of each upload. Thatā€™s a show stopper. There is no async MQTT that I would trust or that can be integrated with the limited program memory or heap remaining.

But wait, IoTaWatt is an open source project. If you really want an MQTT feed there are ways you can do it. Iā€™d recommend that you investigate the status transaction that is used to update the browser app. You can easily see it in action using the browser debugger while the status display is active. It is fast and you can get slightly damped real-time values for both inputs and outputs. You could poll the IoTaWatt at whatever interval you want using something like NODEred or your own python program, then send whatever MQTT packets that you need.

I canā€™t say Iā€™ve ever heard that phrase, but I donā€™t work with hardware engineers so maybe thatā€™s why.

There was mention of a switch to the ESP32 and using a dedicated core for the time-sensitive sampling but I canā€™t find any other mentions of it, was that side-project dropped?

Thanks for the suggestion about the status transaction, Iā€™ll have a look into it. If the resources are really that constrained though, maybe it wouldnā€™t be such a good idea - I wouldnā€™t want to impact its functionality. If the ESP32 source code is available I might end up making something which is essentially a cut-down IotaWatt based on the ESP32 dedicated to this task, essentially just enough to read import/export of a 3 phase supply and nothing more - No uploaders, no data logging, no custom calculations. Youā€™ve done a lot of the heavy lifting already with what youā€™ve open sourced so thatā€™s always a possibility.

I agree MQTT shouldnā€™t be a top priority for you, but this is actually one thing that is hard to do with any of the current solutions, along with ā€œreal timeā€ automation. I can grab my data via the API to get what I am looking for, but that is an extra barrier for most.