Datamigration from InfluxDB to QuestDB with Telegraf

Community
1

@javier
I’m fighting around with data migration from InfluxDB to QuestDB and no way I was discussing with Nick Woolmer was working really well. So he has the idea to contact you for transferring data with Telegraf vom InfluxDB to QuestDB to get the data without errors and in dens format to QuestDB.

2

Hi,

Sorry I didn’t reply earlier. I was travelling last week for a conference and didn’t notice this.

I have successfully converted from non-dense to dense data using telegraf using an aggregator on telegraf, so metrics with the same timestamp get merged into a single dense row.

I have an example on this repository GitHub - questdb/opcua-questdb-telegraf: Playground for OPCUA data ingestion into QuestDB using telegraf and on this one GitHub - javier/questdb-basic-monitoring: Basic questdb monitoring using grafana and collecting questdb metrics via telegraf

In both I am using the same approach.

In your case, you want to migrate from influx into questdb, so one option is to use telegraf to read from influx and write into questdb. I did that once like this.

Telegraf.conf file

[agent]
  omit_hostname = true

# Accept metrics over InfluxDB 2.x HTTP API
[[inputs.influxdb_v2_listener]]
  ## Address and port to host InfluxDB listener on
  ## (Double check the port. Needs to be available)
  service_address = ":8099"

 # Merge metrics into multifield metrics by series key.
[[aggregators.merge]]
 period = "876000h"  # Set the period to 100 years (876000 hours)
  grace = "876000h"  # Set the grace period to 100 years (876000 hours)
  drop_original = true

# this is the questdb destination
 [[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:9000"]
  content_encoding = "identity"  # Important to ensuring no gzip encoding

Start Telegraf

telegraf --config sparse_to_dense.conf --debug

Use influxdb2 to export data into the telegraf proxy

In my case, I have a dockerized influx, so I am running this directly from docker.

docker exec -it  influxdb2 influx write --bucket javier --file output.lp  --host http://host.docker.internal:8099

Hopefully this helps!

1 Like
3

I have tried to migrate my data from influxdb2 using telegraf today. I have more than 1 mio values per month so i had to fiddle with limits and timeouts. Finally it worked by exporting the data per month using a simple script.

my telegraf config

[agent]
  omit_hostname = true

[[inputs.influxdb_v2_listener]]
  ## Address and port to host InfluxDB listener on
  ## (Double check the port. Needs to be available)
  service_address = ":8099"

[[aggregators.merge]]
 period = "876000h"  # Set the period to 100 years (876000 hours)
  grace = "876000h"  # Set the grace period to 100 years (876000 hours)
  drop_original = true
  metric_buffer_limit = 2000000

 [[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:9000"]
  metric_batch_size = 10000
  metric_buffer_limit = 2000000
   timeout = "60s"
  ping_timeout = "30s"
  flush_interval = "99999s"
  content_encoding = "identity"  # Important to ensuring no gzip encoding
~

and the script i used

!/bin/bash
# copy all data from influxdb2 to questdb using telegraf to transform data in rows
# we have to do it month by month as we have > 140000 values per month and telegraf
# has to perform the transformation in memory
#
# AD 10/2024
# !one time script, no error checking!

if [ $"$UID" != "0" ]; then
        echo "Influx export requires root"
        exit 1
fi

EP=/mpeg/data/influxdb-2/engine
BID=e1cffc49bb328fea
FN=influx_export

INV="\033[7m"
NOR="\033[0m"
BLD="\033[1m"

for YEAR in 2019 2020 2021 2022 2023 2024; do
        for MONTH in 01 02 03 04 05 06 07 08 09 10 11 12; do
                LASTDAY=$(cal --months=1 1 $MONTH $YEAR | awk 'NF {DAYS = $NF}; END {print DAYS}')
                START=$YEAR-$MONTH-01T00:00:00Z
                END=$YEAR-$MONTH-${LASTDAY}T23:59:59Z
                printf "\n${INV}Exporting $MONTH/$YEAR ($START to $END)${NOR}\n"
                influxd inspect export-lp --bucket-id $BID --engine-path $EP --output-path $FN --start $START --end $END 2>&1
                FILESIZE=$(stat -c%s "$FN")
                #echo "Size of $FN = $FILESIZE bytes."
                if [ $FILESIZE -le 1 ]; then
                        echo "No data for $MONTH/$YEAR"
                else
                        # start telegraf
                        telegraf --config sparse_to_dense.conf --debug &
                        TPID=$!
                        echo "Started telegraf with PID $TPID"
                        sleep 3
                        printf "${BLD}Sending data to telegraf${NORM}\n"
                        influx write --bucket ad --file $FN --host http://localhost:8099
                        printf "${BLD}waiting until telegraf has finished sending data to questdb, pid:${TPID}${NOR}\n"
                        kill $TPID
                        #killall telegraf
                        wait $TPID
                        printf "${BLD}-- end of processing for $MONTH/$YEAR --${NOR}\n"
                fi
        done
done
4

Btw, the size of the database directory is nearly the same as for InfluxDB V2:
influxdb-2 4686M
questdb 4926M

5

Very cool! Thank you for sharing!!!