Using InfluxDB to store performance metrics

InfluxDB it is a database oriented on storing time series. It was designed to handle large number of queries and write requests. It is perfect for storing a high volume of timestamped data – data we would like to query by timestamp rather than index such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics. 

How to get started

Installing InfluxDB

In order to install InfluxDb  on your local machine go to https://docs.influxdata.com/influxdb/v2/install/#download-and-install-influxdb-v2 and follow instructions.

Once InfluxDb is downloaded you can start database engine with following command:

.\influxd

And you should see following message:

 Welcome to InfluxDB {“log_id”: “0tXBjs9W000”, “version”: “v2.7.11”, “commit”: “fbf5d4ab5e”, “build_date”: “2024-12-02T17:48:13Z”, “log_level”: “info”}

By default, influx will start on port 8086

Initialization process

Initialization with Influx CLI

Start interactive initialization process with following command

influx setup

This will lead you through setting up username, password, bucket, organization and API token.

Setup can be also done non-interactively. For that pass all required values as a parameter to influx setup command:

influx setup \
  --username USERNAME \
  --password PASSWORD \
  --token TOKEN \
  --org ORG_NAME \
  --bucket BUCKET_NAME \
  --force

Initialization with Influx UI

Go to localhost:8086/onboarding to run initialization process.

Once initialization is done don’t forget to save API token:

This will be needed in the next step to configure influx CLI

Installing Influx CLI

To install influx CLI go to https://docs.influxdata.com/influxdb/v2/tools/influx-cli/ and follow instructions

To avoid authorizing CLI every time we need to create and store configuration in influx CLI:

influx config create --config-name CONFIG_NAME \
  --host-url http://localhost:8086 \
  --org ORG \
  --token API_TOKEN \
  --active

Replace CONFIG_NAME, ORG and API_TOKEN with proper values.

Influx data model

Influx stores data in buckets and measurements. Where one bucket can contain multiple measurements and one measurement stores mutliple tags and fields. You can think about bucket as a database and measurement as a table.

We can have multiple buckets within single influx instance, so each type of metric can be stored in different database.

In InfluxDB measurements, tag keys, tag values and field keys are always strings. This is the reason why any math operation cannot be performed on tag values. Different from this are field values where different data types can be stored:

•  Float – default numerical value, when we request value=2990.0 or value=2990 it will be stored as float
• Integer – to force value be saved as integer it must have an ‘i’ at the end: value=2990i
• String – value put in quotes will be treated as string: value=”2990”
• Boolean – TRUE is represented by t, T, true, True, or TRUE; FALSE is represented by f, F, false, False, or FALSE

Type of filed value is set during first write request.

Details about Influx data model can be found here InfluxDB data elements

Writing data

To write data you can use Influx CLI, Influx API or Influx UI.

Writing using Influx CLI

For writing data, we can use “influx write” command. We need to specify bucket name and timestamp precision in the parameters list.

influx write \
  –bucket weather-db \
  –precision s “
weather,city=Krakow temp=11.0,hum=31.9,co=0i 1641024000

weather,city=Warsaw temp=11.4,hum=31.9,co=0i 1641027600

weather,city=Krakow temp=13.0,hum=32.2,co=0i 1641027600

weather,city=Warsaw temp=11.8,hum=32.0,co=0i 1641031200

weather,city=Krakow temp=12.7,hum=32.1,co=0i 1641031200

weather,city=Warsaw temp=12.2,hum=32.0,co=0i 1641034800

weather,city=Krakow temp=12.4,hum=32.0,co=0i 1641034800”

Measurements are created dynamically upon write request – when where is a request with data to non-existing measurement it will be created automatically with all the fields and tags (also write request with data with non-existing tags or fields will extend this measurement).

Writing using API

Another option is to use Influx API 

To write data into InfluxDB send a POST request to http://localhost:8086/api/v2/write endpoint. Here is sample request:

curl –request POST \
“http://localhost:8086/api/v2/write?org=YOUR_ORG&bucket=YOUR_BUCKET&precision=ns” \
  –header “Authorization: Token YOUR_API_TOKEN” \
  –header “Content-Type: text/plain; charset=utf-8” \
  –header “Accept: application/json” \
  –data-binary ‘
weather,city=Warsaw temp=14.2,humidity=16.0,air\ pressure=1018i 1734511033

weather,city=Krakow temp=15.4,humidity=14.3,air\ pressure=1015i 1734511038
    ‘

Writing programmatically using NodeJS

Here is NodeJS example how to write data to InfluxDB using HTTP API:

import influxdb_client
from influxdb_client.client.write_api import SYNCHRONOUS

bucket = “<my-bucket>”
org = “<my-org>”
token = “<my-token>”
# Store the URL of your InfluxDB instance
url=”http://localhost:8086″

client = influxdb_client.InfluxDBClient(
    url=url,
    token=token,
    org=org
)

# Write script
write_api = client.write_api(write_options=SYNCHRONOUS)

p = influxdb_client.Point(“weather”).tag(“city”, “Krakow”).field(“temp”, 25.3)
write_api.write(bucket=bucket, org=org, record=p)

Reading data

To query data, we can use Flux – a scripting language for influxDB or InfluxQL – a SQL-like query language designed to query time series data.

Since Flux will not be supported in Influx v3 it is recommended to use InfluxQL

InfluxQL was designed for InfluxDB v1 so we have to use influx v1 shell:

influx v1 shell

And then to set bucket (in v1 called “database”) we will use:

use weather-db 

To know the structure of a measurement we can use following commands:

• show tag keys from <measurement> – to list all tags within measurement
• show field keys from <measurement> – to list all fields and its type within measurement

Having this information, we can create a select query to grab information. The simples one is:

 select * from “weather” (note that measurement’s name must be in quotes):

 Index  ┃            time                ┃  air pressure   ┃  city  ┃   humidity    ┃     temp            |       1┃  1734511000000000000.0000000000┃  1024.0000000000┃Warsaw  ┃  32.0000000000┃  11.8000000000┃

┃      2┃  1734511010000000000.0000000000┃  1023.0000000000┃Krakow  ┃  32.1000000000┃  12.7000000000┃

┃      3┃  1734511023000000000.0000000000┃  1021.0000000000┃Warsaw  ┃  32.0000000000┃  12.2000000000┃

┃      4┃  1734511028000000000.0000000000┃  1025.0000000000┃Krakow  ┃  32.0000000000┃  12.4000000000┃

┃      5┃  1734519900000000000.0000000000┃  1024.0000000000┃Krakow  ┃  31.9000000000┃  11.0000000000┃

┃      6┃  1734519970000000000.0000000000┃  1022.0000000000┃Warsaw  ┃  31.9000000000┃  11.4000000000┃

┃      7┃  1734519970000000000.0000000000┃  1019.0000000000┃Krakow  ┃  32.2000000000┃  13.0000000000

And the more complicated one using fields and tags will be like:

select temp, humidity from “weather” where city = ‘Krakow’ 

Finally, we can query based on timestamp what is a key feature here:

select * from “weather” where time <  1734519970000000000 – based on given timestamp

select * from “weather” where time < now() – 10d – list all entries from the last 10 days

Influx has also several aggregation functions especially useful for data analysis, like:

• COUNT()
• MEAN()
• STDDEV()
• SUM()
• DISTINCT()

Like regular SQL it supports data selectors like:

• FIRST()
• LAST()
• MAX()
• PERCENTILE()

And transformations:

• EXP()
• COS()
• ROUND()

select mean(“temp”) from “weather” where “city” = ‘Warsaw’
| index      ┃     time     ┃     mean       ┃

|      1     ┃  0.0000000000┃  11.8000000000 |

 Summary

Influx seems to be a very decent DB for storing time-oriented large amount of data.  According to external benchmarks it is much more efficient compared to i.e. Cassandra in terms of write throughput or response times. Additionally, many visualizing data tools like Grafana have a built-in support for pulling out data which makes charts creation very easy.