Configuring Vector

Vector is configured using a configuration file. This section contains a comprehensive reference of all Vector configuration options.

Example

The following is an example of a popular Vector configuration that ingests logs from a file and routes them to both Elasticsearch and AWS S3. Your configuration will differ based on your needs.

{
   "data_dir": "/var/lib/vector",
   "sources": {
      "apache_logs": {
         "type": "file",
         "include": [
            "/var/log/apache2/*.log"
         ],
         "ignore_older": 86400
      }
   },
   "transforms": {
      "remap": {
         "inputs": [
            "apache_logs"
         ],
         "type": "remap",
         "source": ". = parse_apache_log(.message)"
      },
      "apache_sampler": {
         "inputs": [
            "apache_parser"
         ],
         "type": "sampler",
         "rate": 50
      }
   },
   "sinks": {
      "es_cluster": {
         "inputs": [
            "apache_sampler"
         ],
         "type": "elasticsearch",
         "host": "http://79.12.221.222:9200",
         "index": "vector-%Y-%m-%d"
      },
      "s3_archives": {
         "inputs": [
            "apache_parser"
         ],
         "type": "aws_s3",
         "region": "us-east-1",
         "bucket": "my-log-archives",
         "key_prefix": "date=%Y-%m-%d",
         "compression": "gzip",
         "encoding": "ndjson",
         "batch": {
            "max_size": 10000000
         }
      }
   }
}
# Set global options
data_dir = "/var/lib/vector"

# Vector's API (disabled by default)
# Enable and try it out with the `vector top` command
[api]
enabled = false
# address = "127.0.0.1:8686"

# Ingest data by tailing one or more files
[sources.apache_logs]
type         = "file"
include      = ["/var/log/apache2/*.log"]    # supports globbing
ignore_older = 86400                         # 1 day

# Structure and parse via Timber's Remap Language
[transforms.remap]
inputs       = ["apache_logs"]
type         = "remap"
source       = '''
. = parse_apache_log(.message)
'''

# Sample the data to save on cost
[transforms.apache_sampler]
inputs       = ["apache_parser"]
type         = "sampler"
rate         = 50                            # only keep 50%

# Send structured data to a short-term storage
[sinks.es_cluster]
inputs       = ["apache_sampler"]            # only take sampled data
type         = "elasticsearch"
host         = "http://79.12.221.222:9200"   # local or external host
index        = "vector-%Y-%m-%d"             # daily indices

# Send structured data to a cost-effective long-term storage
[sinks.s3_archives]
inputs       = ["apache_parser"]             # don't sample for S3
type         = "aws_s3"
region       = "us-east-1"
bucket       = "my-log-archives"
key_prefix   = "date=%Y-%m-%d"               # daily partitions, hive friendly format
compression  = "gzip"                        # compress final objects
encoding     = "ndjson"                      # new line delimited JSON
batch.max_size   = 10000000                  # 10mb uncompressed
data_dir: /var/lib/vector
sources:
  apache_logs:
    type: file
    include:
      - /var/log/apache2/*.log
    ignore_older: 86400
transforms:
  remap:
    inputs:
      - apache_logs
    type: remap
    source: |
            . = parse_apache_log(.message)
  apache_sampler:
    inputs:
      - apache_parser
    type: sampler
    rate: 50
sinks:
  es_cluster:
    inputs:
      - apache_sampler
    type: elasticsearch
    host: 'http://79.12.221.222:9200'
    index: vector-%Y-%m-%d
  s3_archives:
    inputs:
      - apache_parser
    type: aws_s3
    region: us-east-1
    bucket: my-log-archives
    key_prefix: date=%Y-%m-%d
    compression: gzip
    encoding: ndjson
    batch:
      max_size: 10000000

To use this configuration file, specify it with the --config flag when starting Vector:

vector --config /etc/vector/vector.json
vector --config /etc/vector/vector.toml
vector --config /etc/vector/vector.yaml

Reference

Components

Advanced

How it works

Environment variables

Vector interpolates environment variables within your configuration file with the following syntax:

[transforms.add_host]
type = "add_fields"

[transforms.add_host.fields]
host = "${HOSTNAME}"
environment = "${ENV:-development}" # default value when not present

Default values

Default values can be supplied using :- syntax:

option = "${ENV_VAR:-default}"

Escaping

You can escape environment variables by prefacing them with a $ character. For example $${HOSTNAME} is treated literally in the above environment variable example.

Formats

Vector supports TOML, YAML, and JSON to ensure that Vector fits into your workflow. A side benefit of supporting JSON is that it enables you to use JSON-outputting data templating languages like Jsonnet and Cue.

Location

The location of your Vector configuration file depends on your installation method. For most Linux-based systems, the file can be found at /etc/vector/vector.toml.

Multiple files

You can pass multiple configuration files when starting Vector:

vector --config vector1.toml --config vector2.toml

Or using a globbing syntax:

vector --config /etc/vector/*.toml

Wilcards in component names

Vector supports wildcards (*) in component names when building your topology, but only supports them as the last character. For example:

[sources.app1_logs]
type = "file"
includes = ["/var/log/app1.log"]

[sources.app2_logs]
type = "file"
includes = ["/var/log/app.log"]

[sources.system_logs]
type = "file"
includes = ["/var/log/system.log"]

[sinks.app_logs]
type = "datadog_logs"
inputs = ["app*"]

[sinks.archive]
type = "aws_s3"
inputs = ["app*", "system_logs"]

Sections

Pages