Skip to content
mbedded.ninja

MQTT Protocol

Authors:
Geoffrey Hunter
Geoffrey Hunter
Published On:
Feb 16, 2014
Last Updated:
Sep 24, 2024

MQTT is a lightweight publish/subscribe communications protocol that uses TCP/IP (and other secondary transport mechanisms) that is a popular choice for IoT communications. It is designed for remote sensors to communicate with controlling devices. The specification is under a royalty-free licence. Interesting features of the protocol include the one-to-many messaging service (publish/subscribe), the Last Will/Testament feature, and the three qualities of service. You can read more about it here.

The MQTT logo1.

MQTT-SN (MQTT for Sensor Networks) is an optimized version of the MQTT protocol designed to be used over non-TCP/IP connections.

History

Andy Stanford-Clark (working at IBM) and Arlen Nipper (working for Eurotech) published the first version of the MQTT protocol in 1999. It was used to monitor oil pipelines that used the SCADA (Supervisory Control and Data Acquisition) control system2, with connections to the internet primarily through expensive satellite connections3.

The MQ in MQTT stood for “Message Queue” in the IBM MQSeries, a family of messaging software products. Version 3.1 of the protocol referred to it as MQ Telemetry Transport, and hence the name “MQTT” was born2.

In 2013, IBM submitted MQTT v3.1 to the OASIS specification body2.

On March 7, 2019, MQTT Version 5 was released, and as of 2022-06 is the most recent major version.

Topics

MQTT uses a topic-based system for sending and receiving messages. A topic is a UTF-8 encoded string4 that clients and publish on and subscribe to. Topics are hierarchical and can contain multiple levels separated by forward slashes (/). For example, home/bedroom/temperature is a topic with three levels.

Generic MQTT viewers support this hierarchical structure and allow you to “drill down” in large datasets to view data as needed.

MQTT Brokers

The MQTT broker is the central server which “brokers” messages between clients. Clients connects to the broker and publish messages to specific topics. Other clients can then subscribe to those topics and receive the messages. Brokers are typically run on a cloud-based server, while clients run on embedded devices and inside software applications.

Mosquitto is a very popular open-source MQTT broker.

Online test/sandbox brokers:

  • mqtt.eclipse.org

Mosquitto

Installation

Windows:

Mosquitto can be installed on Windows by downloading the pre-compiled binaries. The x64 version will be installed in a path similar to C:\Program Files\Mosquitto. You will likely want to add that directory path to your system path so that you can call mosquitto from the command line.

Running

If you have added the installation directory to your system path, you can invoke mosquitto with:

Terminal window
mosquitto

By default the above command will start a MQTT broker listening on port 1883 (the default port for non-encrypted traffic). The broker will not run as a daemon, so you can stop the broker with Ctrl-C (or equivalent). It also uses the following defaults:

  • No authentication

You can listen to all system topics by subscribing to $SYS/#.

If you are experiencing problems when running mosquitto, you might want to run in verbose mode -v to print debug information, as by default very little information is printed to the command-line:

Terminal window
$ mosquitto -v

1883 is the default port for unencrypted traffic. 8883 is the default port for traffic encrypted with SSL/TLS.

Common Errors

Client <client-name> disconnected due to protocol error.: Can be due to the client requesting TLS/SSL encryption on the non-encrypted 1883 port.

ssl3_read_bytes:tlsv1 alert unknown ca: Broker does not recognize the CA sent by the connecting client.

Creating Password Files

mosquitto_passwd is a utility provided alongside mosquitto which can generate password files for mosquitto. These password files control user access to the broker.

MQTT Clients

Paho is a popular MQTT client for Desktop and server machines running Windows, Linux or macOS. It’s API is reasonably well documented at https://pypi.org/project/paho-mqtt/.

You can install Paho the standard way using pip:

pip install paho-mqtt

The Paho library can then be imported into your Python files with:

import paho.mqtt.client as mqtt

To request that the client creates the connection by supplying a username and password, call username_pw_set() on the client before calling connect():

client.username_pw_set('admin', 'password')
client.connect_async(self.broker_address, 1883, 60)

MQTT Clients For Embedded Devices

There are a number of MQTT clients designed for embedded devices, these include:

  • The embedded Paho MQTT client: Embedded version of the popular Paho MQTT client. It is released as two separate APIs:
    • MQTTPacket: Low-level C library that deals with the serialization/deserialization of MQTT packets.
    • MQTTClient: Higher-level C++ library first written for the mbed platform. Depends on MQTTPacket.
  • arduino-mqtt: Arduino wrapper around the lwmqtt MQTT client. Also available for PlatformIO. This one is pretty popular and is recommended in the AWS IoT tutorials.
  • coreMQTT: MQTT client maintained by the FreeRTOS group (however, the library does not depend on FreeRTOS to operate).

MQTT-SN

MQTT-SN (MQTT for Sensor Networks, previously known as just MQTT-S5) is a version of the MQTT protocol which has been optimized use on low-power, low-performance, wireless IoT devices. It is designed to work on non-TCP/IP networks, such as Zigbee5.

This is a placeholder for the reference: mqtt-sn-architecture shows the high-level MQTT-SN architecture, and how MQTT-SN clients talk to a standard MQTT broker.

High-level diagram of the MQTT-SN architecture6.

Some of the changes in MQTT-SN include (compared to standard MQTT):

  1. Shortened two-octet (byte) topic IDs are used instead of full topic names (strings)7. This can cut down on the amount of bytes sent for topic identification considerably, e.g. the topic name device1/sensor5/temperature now just becomes topic ID 54.
  2. Ability to pre-assign topic IDs on both the client and Gateways (GW) so they don’t have to be registered across the network (with the REGISTER message).

The complete specification for MQTT-SN is called MQTT For Sensor Networks (MQTT-SN) Protocol Specification: Version 1.2 and can be downloaded for free from https://www.oasis-open.org/committees/download.php/66091/MQTT-SN_spec_v1.2.pdf.

MQTT-SN is designed in such a way that it is agnostic of the underlying networking services. Any network which provides a bi-directional data transfer service between any node and a particular one (a gateway) should be able to support MQTT-SN6.

Message Ordering

The ordering of messages can sometimes be important, especially when a device is sending control messages or state information across MQTT. For example, if a switch published a message saying it was on, and then quickly published a message saying it was off, you need the order of these messages to be preserved if you want to know the current state of the switch.

MQTT makes the following guarantees about message ordering for messages sent on the same topic (no guarantee of ordering is made across messages on different topics)8:

  • No guarantees are made about the relative ordering between messages published on a topic with different QoS values. For example, a message published by one client with a QoS of 0 after a message with a QoS of 2 may be received first by a different client.
  • QoS 0 messages will be delivered in order (but messages may get lost)
  • QoS 1 messages will be delivered in order, but only for the first attempt. Duplicates may be sent, and a duplicate may arrive after the next message was received.
  • QoS 2 messages will be delivered in order.

Per Section 4.6: Message ordering of the MQTT v3.1.1 documentation9:

A Client MUST follow these rules when implementing the protocol flows defined elsewhere in this chapter:

When it re-sends any PUBLISH packets, it MUST re-send them in the order in which the original PUBLISH packets were sent (this applies to QoS 1 and QoS 2 messages) [MQTT-4.6.0-1]

  • It MUST send PUBACK packets in the order in which the corresponding PUBLISH packets were received (QoS 1 messages) [MQTT-4.6.0-2]
  • It MUST send PUBREC packets in the order in which the corresponding PUBLISH packets were received (QoS 2 messages) [MQTT-4.6.0-3]
  • It MUST send PUBREL packets in the order in which the corresponding PUBREC packets were received (QoS 2 messages) [MQTT-4.6.0-4]

A Server MUST by default treat each Topic as an “Ordered Topic”. It MAY provide an administrative or other mechanism to allow one or more Topics to be treated as an “Unordered Topic” [MQTT-4.6.0-5].

When a Server processes a message that has been published to an Ordered Topic, it MUST follow the rules listed above when delivering messages to each of its subscribers. In addition it MUST send PUBLISH packets to consumers (for the same Topic and QoS) in the order that they were received from any given Client

Footnotes

  1. Eclipse. Paho. Retrieved 2021-03-30, from https://www.eclipse.org/paho/. ↩

  2. Wikipedia (2022, June 21). MQTT. Retrieved 2022-06-22, from https://en.wikipedia.org/wiki/MQTT. ↩ ↩2 ↩3

  3. Shaun Behrens (2019, Aug 09). From Oil Pipelines to the IoT: A Brief History of MQTT. Paessler. Retrieved 2022-06-22, from https://blog.paessler.com/a-brief-history-of-mqtt. ↩

  4. Influx. A Guide to MQTT - MQTT Guide - Definition, Use Cases, Applications, and Resources. Retrieved 2024-09-25, from https://www.influxdata.com/mqtt/. ↩

  5. MQTT. MQTT Specifications. Retrieved 2022-06-20, from https://mqtt.org/mqtt-specification/. ↩ ↩2

  6. Andy Stanford-Clark and Hong Linh Truong (2013, Nov 14). MQTT For Sensor Networks (MQTT-SN) Protocol Specification: Version 1.2. Retrieved 2022-06-21, from https://www.oasis-open.org/committees/download.php/66091/MQTT-SN_spec_v1.2.pdf. ↩ ↩2

  7. ublox (2020, Jun 16). MQTT-SN – lowering the cost of IoT at scale. Retrieved 2022-06-20, from https://www.u-blox.com/en/blogs/insights/mqtt-sn. ↩

  8. StackOverflow. Is message order preserved in MQTT messages?. Retrieved 2022-10-03, from https://stackoverflow.com/questions/30955110/is-message-order-preserved-in-mqtt-messages. ↩

  9. Oasis. MQTT Version 3.1.1 Specifications. Retrieved 2022-10-03, from http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html. ↩