Storage Service
The Storage Service enables storing and backing up acquired data to avoid data losses. Such backup copies may be useful if a connection to the relayr Cloud is interrupted or if you need access to the stored data for the Rule Engine or the GWA Analytics components.
The Storage Service stores data persistently in a database. For standard data types such as measurements and alerts, it stores structured data. For custom data types, it stores raw data. By supporting the data rotation, the Storage Service lets you manage the amount of data you want to store persistently. You can configure all settings related to the data storage in the application configuration file.
To download or delete the stored data, use the Storage Service's REST interface. You can also perform basic calculations on the stored data, such as calculate an average value or a maximum value from a time period you select. For more information on available actions, see the API documentation.
For a step-by-step guide on how to install the Storage Service, see the Installation Guide.
Configuring the Storage Service
There are two configuration files of the Storage Service:
Deployment configuration file - Configuration of the database, MQTT connection, PMQ connection, REST interface, and health monitoring.
Application configuration file - Configuration of data streams to store and rotation limits of the persistent storage.
When you modify the Storage Service configuration, restart the component to apply the changes by means of the following command:
systemctl restart gwa-storage-service-cpp
For an example of the Storage Service configuration files, see here.
Deployment Configuration File
The configuration file of the Storage Service is located as:
/etc/relayr/gwa-storage-service-cpp/gwa-storage-service-config.json
The following configuration settings are available:
database_path
- Path to the database to store data. By default,/var/cache/relayr/gwa-storage-service-cpp/database.sqlite3
.transport
- Type of transport. Available options:mqtt
orpmq
. By default,mqtt
.mqtt_connection
- Connection settings of the MQTT message broker:
Configuration Setting | Description | Default Value |
---|---|---|
host | IP address. | localhost |
port | Port number for connecting to the MQTT broker. | 1883 |
keepalive | Keepalive value of the MQTT connection. | 20 |
max_inflight_messages | Maximum number of QoS 1 or 2 messages that can be in the process of being transmitted simultaneously. This includes messages currently going through handshakes and messages that are being retried. If set to 1 , this will guarantee in-order delivery of messages. | 100 |
pmq_connection
- Configuration of the High Speed Bus (PMQ) connection. Configure this section to use thepmq
transport
:
Configuration Option | Description | Default Value |
---|---|---|
input_queues | List of PMQ queues to monitor for inbound messages. | [] |
message_size | Maximum size of a PMQ message, given in bytes. | 8192 |
message_count | Maximum number of messages in a PMQ queue. | 10 |
rest
- Connection settings of the Storage Service's REST interface that allows you to access and manage the stored data. See the API documentation for more information.
Configuration Setting | Description | Default Value |
---|---|---|
host | Host name or IP address of the REST interface. | 127.0.0.1 |
port | Port number of the REST interface. | 26080 |
health_monitor
- Health monitoring settings:
Configuration Setting | Description | Default Value |
---|---|---|
heartbeat_interval | Time interval at which the periodic heartbeat messages are sent, given in seconds. | 60 |
For an example of the deployment configuration file, see here.
Application Configuration File
The application configuration file of the Storage Service is located as:
/etc/relayr/gwa-storage-service-cpp/gwa-storage-service-application-config.json
The application configuration file lets you configure the persistent storage settings:
custom_prefix
- Prefix from which the Storage Service receives data. The default value isv1/nb
, which means that it receives all data from the northbound interface. This is a global setting for all the received data streams. You can also configure a different custom prefix for each object in the streams configuration. If configured, the local custom prefix overrides the global one.rotation_limits
- Settings of the circular buffer. For each message type, configure a maximum number of messages to be stored in the database. If the Storage Service receives more messages, it deletes the oldest ones. You can also configure more specific limits for each data stream in the streams configuration.
Configuration Setting | Description | Default Value |
---|---|---|
alerts | Maximum number of messages of the alerts type to be stored persistently. | 1000 |
complex_measurements | Maximum number of messages of the complex_measurements type that is stored persistently. | 1000 |
measurements | Maximum number of messages of the measurements type to be stored persistently. | 1000 |
metadata | Maximum number of messages of the metadata type to be stored persistently. | 1000 |
analytics | Maximum number of messages of the analytics type to be stored persistently. | 1000 |
locations | Maximum number of messages of the locations type to be stored persistently. | 1000 |
logs | Maximum number of messages of the logs type to be stored persistently. | 1000 |
custom | Maximum number of messages of the custom type to be stored persistently. | 1000 |
streams
- JSON array of data streams to subscribe to. See the section below for more information.
For an example of the application configuration file, see here.
Configuring Data Streams
The streams
section of the application configuration file lets you configure an array of message types and topics that the Storage Service subscribes to.
To store all data of a given type, without any additional restrictions, configure an object with the type
field only, e.g.:
"streams": [
{
"type": "complex_measurements"
}
Apart from the message type
, you can configure additional fields that define which data to store for each object in the streams
section. You can use these settings in any combination. See below for the streams
configuration examples.
For the
custom
data type, you must specify atopic
and aname
of the data you want to store.
Configuration Setting | Description | Example of Value |
---|---|---|
device_id | UUID of the device for which the Storage Service stores data. | 12345678-1234-5678-abcd-12345678 |
custom_prefix | Prefix of topics which the Storage Service subscribes to. This setting overrides the global custom prefix you configure in the application configuration file. | v1/custom_ms |
topic | Topic to subscribe to. You can configure both full topics and wildcards. For the custom data type, the topic field is mandatory. | v1/custom/+/measurements/temperature , just/a/topic/# |
msg_id | Identifier of the messages to store. In the standard Gateway Agent's message format, an ID is the last part of the message topic. By means of the msg_id value, you can retrieve messages using REST API calls. | xyz |
name | Name of the messages to store. The name field is mandatory for the custom message type and you can use it to retrieve custom messages by means of REST API calls. | temperature_c |
limit | Maximum number of messages of a given stream to store. If the limit is crossed, the Storage Service deletes the oldest messages from the database. This setting lets you divide the global limit from the rotation_limits section among various data streams of a given type, to make sure that the Storage Service saves data from each stream. The value of this local limit can't be greater than the global limit for a given data type. | 300 |
You can't define the same topic twice, no matter if in standard or custom message types. If you do so, the Storage Service doesn't start correctly. For example, you can't configure a measurement data stream with a given
msg_id
and, at the same time, a stream only with thetype
specified asmeasurements
, with no extra settings. You also can't configure thev1/nb/aaa/bbb
topic in one data stream andv1/nb/+/bbb
in another one.
Streams Configuration Examples
Example 1:
"streams": [
{
"type": "measurements",
"limit": 500,
"msg_id": "test",
"custom_prefix": "v1/custom_ms",
"device_id": "12345678-1234-5678-abcd-12345678"
},
In the example above, the Storage Service stores measurements only for a device with UUID
12345678-1234-5678-abcd-12345678
, received from a custom prefixv1/custom_ms
with message IDtest
. It stores up to 500 of such messages.
Example 2:
"streams": [
{
"type": "custom",
"topic": "v1/custom/+/measurements/temperature",
"name": "temperature_c",
"limit": 300
},
In the example above, the Storage Service stores messages of the
custom
type, received to topicv1/custom/+/measurements/temperature
, with a nametemperature_c
. It stores up to 300 of such messages.
Example 3:
"streams": [
{
"type": "alerts",
"limit": 500
},
In the example above, the Storage Service stores up to 500 alerts with no additional specific settings.
Getting the Stored Data
You can manage the stored data, using the Storage Service's REST interface. You can configure the interface settings in the rest
section of the deployment configuration file. Its default host name is 127.0.0.1
and the port number is 26080
.
Using the REST interface, you can perform the following actions:
- Get all messages of a given type from a selected time period.
- Get messages with a specific ID or name from a selected time period.
- Perform basic calculations on the obtained data, e.g. calculate an average value or a maximum value from a selected time period.
- Delete all messages of a given type from a selected time period.
- Delete messages with a specific ID or name from a selected time period.
For detailed information on available actions, see the Storage Service's API documentation.
Examples of Configuration Files
Deployment Configuration File
{
"database_path": "/var/cache/relayr/gwa-storage-service-cpp/database.sqlite3",
"transport": "mqtt",
"mqtt_connection":
{
"host": "localhost",
"port": 1883,
"keepalive": 20,
"max_inflight_messages": 100
},
"pmq_connection":
{
"input_queues": [],
"message_size": 8192,
"message_count": 10
},
"rest":
{
"host": "127.0.0.1",
"port": 26080
},
"health_monitor":
{
"heartbeat_interval": 60
}
}
Application Configuration File
{
"custom_prefix": "v1/nb",
"rotation_limits": {
"alerts": 1000,
"complex_measurements": 1000,
"measurements": 1000,
"metadata": 1000,
"analytics": 1000,
"locations": 1000,
"logs": 1000,
"custom": 500
},
"streams": [
{
"type": "measurements",
"limit": 500,
"msg_id": "test",
"custom_prefix": "v1/custom_ms",
"device_id": "12345678-1234-5678-abcd-12345678"
},
{
"type": "alerts",
"limit": 500
},
{
"type": "complex_measurements"
},
{
"type": "custom",
"topic": "v1/custom/+/measurements/temperature",
"name": "temperature_c",
"limit": 300
},
{
"type": "custom",
"topic": "just/a/topic/#",
"name": "just_a_message",
"limit": 300
},
{
"type": "metadata"
},
{
"type": "analytics"
},
{
"type": "locations"
},
{
"type": "logs"
}
]
}