Internal High-Speed Bus (PMQ)
The Gateway Agent uses a local MQTT broker as the main Internal Bus for a message exchange between the Gateway Agent components. In certain use cases, when a platform needs to acquire and transport high-frequency data like raw vibration or precise movement of an asset, the MQTT communication becomes a bottleneck and introduces a significant load to the host system.
To overcome the MQTT bandwidth limitations, a new Internal High-Speed Bus (IHSB) was introduced. The IHSB uses PMQ (POSIX Message Queues) as an underlying message exchange method and is about 200 times more efficient than MQTT.
You can use the IHSB when you need to transfer a large number of messages from the Gateway Agent data acquisition layer to the data processing and/or storage layers, i.e. from Protocol Adapters to the Rule Engine or the Storage Service.
Consider using the IHSB when the amount of data on the Internal Bus/MQTT exceeds a few hundreds of msg/s.
A number of messages that can be handled on MQTT depends on the hardware platform capabilities.
Even if you use the IHSB, MQTT remains a required communication channel for the internal message exchange between the Gateway Agent components, e.g. for the Health Monitoring.
Components Supporting the High Speed Bus
You can use the Internal High Speed Bus to send data from Protocol Adapters to the following Gateway Agent components:
The diagram below shows the Gateway Agent components that can communicate using the Internal High Speed Bus (PMQ).
Each component that supports the High Speed Bus can have many input and/or output PMQ queues defined.
The relayr Cloud Adapter does not support PMQ communication. To report data to the northbound interface, always use MQTT.
Configuring the High Speed Bus
To use the High Speed Bus communication channel, you need to configure:
Configuring PMQ in the Gateway Agent Components
To use the Internal High Speed Bus, configure the PMQ settings in the deployment configuration files of the Gateway Agent components:
Example: For a Protocol Adapter, set
pmq
in thetransport
option and provide details of the High Speed Bus connection in thepmq_connection
configuration section (available settings are:output_queues
,input_queues
,message_size
andmessage_count
).
For more information, read the component-specific documentation.
Configuring PMQ in the Operating System
To use the Internal High Speed Bus, you need to configure your operating system to support PMQ (POSIX Message Queues).
About POSIX Message Queues
POSIX interprocess communication was introduced in the POSIX.1b standard for real-time extensions. POSIX Message Queues have been available in Linux since kernel version 2.6.6.
POSIX Message Queues are created in /dev/mqueue
, by default. They are identified by name strings, starting with a forward slash, followed by one or more characters.
Any process knowing the queue name can send and receive messages from the queue, assuming that proper permissions were granted.
Every queue has its own permissions (owner, group, access rights).
Example:
$ ls -al /dev/mqueue
drwxrwxrwt 2 root root 140 mar 13 11:19 .
drwxr-xr-x 19 root root 3640 mar 11 07:48 ..
-rw-r--r-- 1 relayr relayr 80 mar 13 11:19 gwa-rule-engine
-rw-r--r-- 1 root root 80 mar 9 11:14 xyz_lsh3_1
Configuring PMQ in the Linux Kernel Interfaces
You can configure PMQ using the /proc
interfaces provided by the Linux kernel.
Here is the list of all the /proc
interfaces:
Interface | Description |
---|---|
/proc/sys/fs/mqueue/msg_default | Defines the default number of messages in a newly created queue. By default, it is set to 10 and may not exceed the /proc/sys/fs/mqueue/msg_max value. |
/proc/sys/fs/mqueue/msg_max | Defines a maximum number of messages in a queue. By default, 10 . The upper limit depends on the kernel version and varies from 65536 to 131072 . |
/proc/sys/fs/mqueue/msgsize_default | Defines the default size of a message in a queue. By default, 8192 (bytes). It may not be greater than the /proc/sys/fs/mqueue/msgsize_max value. |
/proc/sys/fs/mqueue/msgsize_max | Defines a maximum message size in a queue. By default, 8192 (bytes). The maximum value depends on the kernel version and can be between INT_MAX and HARD_MSGSIZEMAX (16777216 bytes). |
/proc/sys/fs/mqueue/queues_max | Sets a system-wide limit of message queues that can be created. By default, 256 . |
You can modify all the files listed above manually or during the system startup.
You can also use the /etc/sysctl.conf
configuration file to trigger an automated modification of these files.
Example: Adding the
fs.mqueue.msgsize_max = 32768
line to the/etc/sysctl.conf
file results in setting the/proc/sys/fs/mqueue/msgsize_max
to32768
at every system startup.
Configuring the Maximum Memory of PMQ
For kernel 2.6 and newer, you must define the maximum memory used by POSIX Message Queues. You can specify it in the /etc/security/limits.conf
file.
Example: Adding the
relayr - msgqueue 20971520
line to the/etc/security/limits.conf
file sets20971520
bytes as the maximum memory used by POSIX Message Queues, created and used by the relayr user.
Alternatively, if you use systemd, you can specify the limits in the systemd service files.
Example:
[Service]
LimitNOFILE=4096
LimitMSGQUEUE=20971520
This sets the maximum number of opened files to 4096
and the maximum memory used by PMQ queues to 20971520
.
To check the current memory limit for PMQ queues (for a specific user), run the
"ulimit -q"
command.