Availability of MQTT?


#1

When is it likely that MQTT support will be available ?

There are lots of third party IoT platforms like Ubidots and Thingspeak around that would provide an easy way to make dashboards available online. Most have free accounts available with some limitations but would be an easy way to create some simple dashboards.


#2

I’ve got PVoutput and MQTT in the queue but have been first doing some overdue rework of basic subsystems. Probably next would be PVoutput.

IMO MQTT has limited utility without a generalized method to timestamp the data being exported. One of the most distinguishing features of IoTaWatt is the ability to maintain 100% upload integrity in the face of communication and server interruptions. So any MQTT implementation would only be a real-time feed subject to any delivery delays and errors of the broker and client. Without timestamps, QoS 2 makes no sense.

It’s great that there are IoT platforms available to provide dashboards, but I can only see real-time dashboards as being practical with MQTT. So my priority is to get the timestamped services working first, like PVoutput.

So probably some months away.


#3

Understand - most important things go to top of list.

Looks like Ubidots and Thingsboard take a timestamp (optionally) as a parameter from the device so would allow a client created time to be used. This may avoid problems of delivery delay and errors effecting data integrity and Q0S 2 would be useful. Thingsspeak however does not allow this so looks like there is no generalized solution. Also the datapacket structures differ for each service so while they use MQTT there would need to be some way to create a custom string structure.

https://ubidots.com/docs/api/mqtt.html#mqtt-api-reference


#4

That’s exactly right.

Quick look at their documentation, it appears that they accept a json formatted data packet with value, time stamp, and a collection of “context” name/value pairs.

So MQTT is just a way to deliver that data packet, they also support delivery using RESTful HTTP. The data packet is the same, so it’s really not about supporting MQTT, it’s about supporting their data packet format.

Iotawatt general framework for HTTP uploads iwould probably be more appropriate here as it works off the datalog and provides historical integrity with time stamped data. Either way, it’s another custom protocol.

There are both practical and physical limits to how many custom device protocols can be supported by IoTaWatt. I’ve seen what happens when a device tries to become a Swiss Army knife. As a power monitor, I’m trying to concentrate on keeping the mission limited in scope and concentrating on making a limited few external databases and the internal datalog as useful as possible.


#5

I didn’t understand that there was not a standard payload for MQTT. I was thinking it would solve the problem of having to implement a different interface for every IoT platform but that’s not the case.

Here is an interesting explanation.

A generic MQTT or HTTP implementation would need to have a ‘payload builder’ which makes it more complex but maybe there are librarys available to do this. Something for the future.


#6

It gets complicated real fast when you add QoS 2 to the mix, and that only guaranties delivery to the broker and any subscribers at the time the broker receives the packet. Because the timestamp is encoded in the payload, the broker has no idea if it is a historical upload or a current measurement.

Probably better to have a process running on a more capable platform that takes data from, say an influxDB database, and sends out MQTT packets using some kind of payload format specifier.


#7

It all depends on what you are trying to achieve. I think of MQTT as a many to many communication transport layer. It is great for sensors and actuators and control elements. Since it supports many to many communications without the need for the sender to do anything special, it makes it easy to add listeners. Since a listener doesn’t have to do anything special to get messages from multiple senders, it also makes it easy to add senders. This means you can add control elements very easily.

I use NodeRED for control and have many sensors and control elements. They all speak their own commands/status, but NodeRED makes it easy to have them play well together.

While you can use it save stuff, you really need to timestamp at the source and then would need to do extra work to meet the consumer’s requirements. So, I would not use it for data where you need to get every single sample.


#8

MQTT will be a great forward step to be added to IoTaWatt. It will allow many type of sensors to be integrated into one single system it also allows IoTaWatt to be controlled from multiple platforms and applications example HomeAssistant and openHub.
The Idea will be very interesting if done!


#9

After exploring how MQTT integrates with IoTaWatt, I’ve moved doing anything with it pretty far down the list. MQTT is a transport mechanism that is used to support higher level communications. While the simple use case of sending a value may seem a standard use, it’s of little value in the primary mission of IoTaWatt. HTTP is a much more robust protocol and more importantly, all of the intended clients and servers (Browsers, Emoncms, influxDB) have native support.

Beyond lack of utility, there are major issues with adding MQTT:

  • Increased code and heap space for the supporting code.
  • There is no embedded support in the ESP8266SDK so would rely on another package. The most popular has quite a few issues outstanding.
  • Without a higher level layer, there is no provision for time stamping.
  • Message delivery beyond the broker is not certain or acknowledged without a higher level protocol layered on.
  • Without TLS or SSL, there is no authorization mechanism.

It all goes to product philosophy. I realize that there are some energy monitors out there that try to do it all. I’ve tried to keep the IoTaWatt function more narrowly focused and concentrate on ease of use, reliability, low-maintenance, and full-function within the capabilities that are supported.

If MQTT turns out to have a lot of value (by that I mean some very useful capability that cannot be supported without it), I’d be happy to work with someone to provide a secure feed to an external (RPi, ESP32) capability.


Power Factor - Addition Output to Emoncms
#10

FYI a while ago I ended up writing a quick and dirty Ruby script which polls InfluxDB for the latest values from measurements I care about and posts them to MQTT if the value has changed. I decided to post an empty value if the latest measurement is more than 10 seconds old so I don’t act on stale data. I subscribe to these topics in Node-RED to do automations and also have these set up as MQTT sensors in Home Assistant for easy monitoring. Works well.


#11

Sounds like an easy solution. One way to make that more general purpose would be to poll the IoTaWatt directly instead of polling influxDB. I realize that I haven’t published the query API details yet, but that is available from the code and I’ll be working on improving the docs soon.


#12

Looks interesting, and it could make happy multiple people and give them the chance to implement all their wishes :wink:

Can you tell me on which area is of the implementation is this :wink: ?

Are the one one webserver.cpp, handleCommand… ?


#13

The API that I had in mind is in getfeeddata.cpp. It’s modeled after the EmonCMS request, but modified to allow requesting multiple “feeds” in one request and doesn’t return hundreds of millisecond time stamps because the time is known.

Someone did a similar thing with MQTT a few months ago using the API for the status request, which returns damped values but is otherwise a simpler interface. That handler is in webserver,cpp


#14

There are some vey good points in this, and I tend to agree.
A point to note, is that in EMONS there is a process to allow an input to be sent to MQTT.
Maybe this is a good compromise for some of your customers.

I agree that keeping it as simple, to ensure reliability, is key for a device like this.
The MQTT package I use for ESP8266 has an issue with subscribing, and causes a massive overhead to overcome.