Azure IoT Edge Adapter - Work in Progress
The Azure IoT Edge Adapter is a Gateway Agent component, developed as Azure IoT Edge module. It is responsible for collecting messages from other components and transmitting them to IoT Edge Hub, using the MQTT protocol. It enables sending messages in compressed batches to minimize the amount of data transmitted e.g. through a billed connection.
For more information on Azure IoT Edge, see here.
Supported Features
The Azure IoT Edge Adapter supports the following features:
- Reporting of:
- measurements
- alerts
- configuration
- logs
- locations
- metadata
- analytic events
- Message batching
- Message compression
- Configurable prioritization of messages
- Local storage for complex measurements
- Adapter configuration through a module twin
- Health monitoring
The adapter's implementation is currently in progress. The following features are not supported yet:
- Reporting of complex measurements
- Commands (Tasks)
- Installations
- Business parameters (Device Configuration service)
Installing the Azure IoT Edge Adapter
To run the Azure IoT Edge Adapter, you first need to install and run Azure IoT Edge. For more information, see the IoT Edge documentation.
Creating a Deployment Manifest
To install the Azure IoT Edge Adapter as one of Azure IoT Edge modules, you need to include its configuration in a deployment manifest. See here for more information.
The deployment manifest must contain settings related to mqttBroker
authorizations, routes, and modules definition. For each module, define a Docker registry with the module image, for example:
"modules": {
"gwa-azure-iot-edge-adapter": {
"settings": {
"image": "gwadevregistry.azurecr.io/gwa_azure_iot_edge_adapter:0.0.29-amd64",
"createOptions": ""
},
"type": "docker",
"version": "1.0",
"status": "running",
"restartPolicy": "always"
}
}
For an example of the IoT Edge deployment manifest configuration, see the
azure/deployment/deployment.amd64.json
file.
Configuring the Azure IoT Edge Adapter
To configure the Azure IoT Edge Adapter, follow these steps:
- Pack its configuration file, named deployment-config.json, as a Zip archive.
- Provide the archive's URL address and MD5 checksum in the
configuration
section of the module twin, in theurl
andmd5
settings.
Example:
{
"configuration": {
"url": "https://gwadevstorage.blob.core.windows.net/prototyping-configs/am-iot-edge-adapter-configs-test.tar.gz?se=2031-01-01T00%3A00%3A00Z&sp=r&sv=2018-11-09&sr=b&sig=iTwxooe52lm1wlFxqzt9TZ2alNrqFKIQJ6J21P1pYBI%3D",
"md5": "4f59cd0109e91303aab0313ac4647ab5"
}
}
See the azure/module-twin-configuration-section.json
file for an example of the configuration object. See the azure/am-iot-edge-adapter-configs-test/upload.sh
file for an example of the bash script that uploads the configuration files from the directory.
Deployment Configuration File
In the deployment-config.json configuration file, include the following settings:
database
:
Configuration Option | Description | Default Value |
---|---|---|
path | Path to the adapter's database file. You can configure the database to be persistent. To do so, link the module storage to the device storage in Azure IoT Edge deployment configuration. | gwa-iot-edge-adapter-db.sqlite3 |
blobs_limit | Limit of blobs which can be stored in the database. If exceeded, the older blobs are removed and aren't delivered. | 100 |
health_monitor
:
Configuration Option | Description | Default Value |
---|---|---|
heartbeat_interval | Health monitor heartbeat message sending interval, given in seconds. | 60 |
message_batching
- see here for more information.
Configuration Option | Description | Default Value |
---|---|---|
log_sending_timeout_ms | How long a log message can wait to be sent when there's no data for a full batch. -1 means indefinitely. | 10000 ms |
alert_sending_timeout_ms | How long an alert message can wait to be sent when there's no data for a full batch. -1 means indefinitely. | 100 ms |
measurement_sending_timeout_ms | How long a measurement message can wait to be sent when there's no data for a full batch. -1 means indefinitely. | 10000 ms |
task_update_sending_timeout_ms | How long a task update message can wait to be sent when there's no data for a full batch .-1 means indefinitely. | 10000 ms |
config_response_sending_timeout_ms | How long a configuration response message can wait to be sent when there's no data for a full batch. -1 means indefinitely. | 10000 ms |
config_request_sending_timeout_ms | How long a configuration request message can wait to be sent when there's no data for a full batch. -1 means indefinitely. | 10000 ms |
locations_sending_timeout_ms | How long a locations message can wait to be sent when there's no data for a full batch. -1 means indefinitely. | 10000 ms |
analytics_event_sending_timeout_ms | How long an analytics event message can wait to be sent when there's no data for a full batch. -1 means indefinitely. | 100 ms |
blob_request_sending_timeout_ms | How long a blob token request message can wait to be sent when there's no data for a full batch. -1 means indefinitely. | 10000 ms |
compression
:
Configuration Option | Description |
---|---|
use_compression | If set to true , transmitted messages will be compressed with the zstd algorithm. By default, true . |
compression_level | How strong should the compression be? (levels from 0 to 22. Please note that level 0 still compresses somewhat. Set use_compression:false to disable the compression entirely. |
compression_ratio_a | Heuristic estimation of compression ratio as output_size = compression_ratio_a * input_size + compression_ratio_b * log(input_size) + compression_ratio_c |
compression_ratio_b | Heuristic estimation of compression ratio as output_size = compression_ratio_a * input_size + compression_ratio_b * log(input_size) + compression_ratio_c |
compression_ratio_c | Heuristic estimation of compression ratio as output_size = compression_ratio_a * input_size + compression_ratio_b * log(input_size) + compression_ratio_c |
message_priorities
: Configuration of priorities for Azure IoT Edge routing. See here for more information.
WARNING: Configured
message_priorities
increase the amount of data transmitted through the network, because different message types aren't compressed together.
Configuration Option | Description | Default Value |
---|---|---|
alerts | Priority of alert messages. Possible value: <0, 15> . 0 means the highest priority. | 1 |
logs | Priority of log messages. Possible value: <0, 15> . 0 means the highest priority. | 1 |
measurements | Priority of measurement messages. Possible value: <0, 15> . 0 means the highest priority. | 1 |
tasks | Priority of task response messages. Possible value: <0, 15> . 0 means the highest priority. | 1 |
config_responses | Priority of configuration (business parameters) responses. Possible value: <0, 15> . 0 means the highest priority. | 1 |
locations | Priority of location messages. Possible value: <0, 15> . 0 means the highest priority. | 1 |
analytics_events | Priority of analytic event messages. Possible value: <0, 15> . 0 means the highest priority. | 1 |
blob_requests | Priority of blob request messages. Possible value: <0, 15> . 0 means the highest priority. | 1 |
message_processing
:
Configuration Option | Description | Default Value |
---|---|---|
queue_size | Maximum number of messages in processing queue, it affects the maximum amount of memory used by adapter. If adapters receives data faster than it can (or is configured to, see: batching timeouts) process. | 1000 |
See below for a configuration file example.
Message Batching and Priorities
You can optimize the transferred amount of data and ensure a reliable data exchange by configuring the message batching and priorities.
Message Batching
The adapter always sends data in batches. The minimum data batch size is 4 KB and the maximum size is 256 KB. Before sending data to the cloud, the adapter waits until it collects at least the minimum batch.
For each message type, e.g. measurements, alerts, or task updates, you can configure a timeout. It is a number of milliseconds the adapter waits if there's not enough data for a full batch. If the timeout expires, the adapter sends the collected data, no matter how much data it has gathered. The data is batched according to the configured message priorities.
You can configure the batching timeouts in the message_batching
section of the deployment configuration file. If you don't want to set any timeout, set the setting value to -1
. It means that the adapter always waits for a full batch before sending the data.
It's not recommended to set a timeout to
-1
. When you disable the timeout and the adapter stops receiving data due to some problems, the data collected so far may never reach the cloud.
Message Priorities
When a batching timeout expires, the adapter sends data according to message priorities you have configured for each message type. By default, all priorities are set to 1
. It means that the adapter batches all message types and sends them together when the timeout has expired for any message type. To define different priorities, configure the message_priorities
section of the deployment configuration file.
Setting different message priorities can be useful if you're planning to send big data batches after a long period without a network connection and you want to ensure that critical data, e.g. alerts, are transmitted first. In this case, the adapter compresses together only message types that have the same priority.
WARNING: Configured
message_priorities
increase the amount of data transmitted through the network, because different message types aren't compressed together.
Example:
"message_priorities": {
"alerts": 1,
"logs": 2,
"measurements": 2,
"tasks": 1,
"config_responses": 2,
"locations": 2,
"analytics_events": 2,
"blob_requests": 2
},
In this example,
alerts
andtasks
have the highest priority and they are compressed together when the batching timeout expires for any of them. The remaining message types have lower priority, and they are sent in a separate batch from types with priority1
.
Configuration File Example
{
"compression": {
"use_compression": true,
"compression_level": 15,
"compression_ratio_a": 0.05,
"compression_ratio_b": 1.5,
"compression_ratio_c": 12
},
"message_processing": {
"queue_size": 1000
},
"message_batching": {
"alert_sending_timeout_ms": 100,
"log_sending_timeout_ms": 10000,
"measurement_sending_timeout_ms": 120000,
"task_update_sending_timeout_ms": 10000,
"config_response_sending_timeout_ms": 100,
"config_request_sending_timeout_ms": 10000,
"locations_sending_timeout_ms": 10000,
"analytics_event_sending_timeout_ms": 100,
"blob_request_sending_timeout_ms": 10000
},
"message_priorities": {
"alerts": 1,
"logs": 1,
"measurements": 1,
"tasks": 1,
"config_responses": 1,
"locations": 1,
"analytics_events": 1,
"blob_requests": 1
},
"database": {
"path": "cloud.sqlite3",
"blobs_limit": 5
},
"health_monitor": {
"heartbeat_interval": 60
}
}