coercer transform

Accepts `log` events and allows you to coerce event fields into fixed types.

The coercer transform accepts log events and allows you to coerce event fields into fixed types.

Config File

vector.toml (example)
vector.toml (schema)
vector.toml (specification)
[transforms.my_transform_id]
# REQUIRED - General
type = "coercer" # must be: "coercer"
inputs = ["my-source-id"]
# OPTIONAL - Types
[transforms.my_transform_id.types]
status = "int"
duration = "float"
success = "bool"
timestamp = "timestamp|%s"
timestamp = "timestamp|%+"
timestamp = "timestamp|%F"
timestamp = "timestamp|%a %b %e %T %Y"

Options

Key

Type

Description

REQUIRED - General

type

string

The component type required must be: "coercer"

inputs

[string]

A list of upstream source or transform IDs. See Config Composition for more info. required example: ["my-source-id"]

OPTIONAL - Types

types.*

string

A definition of field type conversions. They key is the field name and the value is the type. strftime specifiers are supported for the timestamp type. required enum: "string", "int", "float", "bool", and "timestamp\|strftime"

Examples

Given the following input event:

log
{
// ... existing fields
"bytes_in": "5667",
"bytes_out": "20574",
"host": "5.86.210.12",
"message": "GET /embrace/supply-chains/dynamic/vertical",
"status": "201",
"timestamp": "19/06/2019:17:20:49 -0400",
"user_id": "zieme4647"
}

And the following configuration:

vector.toml
[transforms.<transform-id>]
type = "coercer"
[transforms.<transform-id>.types]
bytes_in = "int"
bytes_out = "int"
timestamp = "timestamp|%m/%d/%Y:%H:%M:%S %z"
status = "int"

A log event will be emitted with the following structure:

{
// ... existing fields
"bytes_in": 5667,
"bytes_out": 20574,
"host": "5.86.210.12",
"message": "GET /embrace/supply-chains/dynamic/vertical",
"status": 201,
"timestamp": <19/06/2019:17:20:49 -0400>,
"user_id": "zieme4647"
}

How It Works

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Types

By default, extracted (parsed) fields all contain string values. You can coerce these values into types via the types table as shown in the Config File example above. For example:

[transforms.my_transform_id]
# ...
# OPTIONAL - Types
[transforms.my_transform_id.types]
status = "int"
duration = "float"
success = "bool"
timestamp = "timestamp|%s"
timestamp = "timestamp|%+"
timestamp = "timestamp|%F"
timestamp = "timestamp|%a %b %e %T %Y"

The available types are:

Type

Desription

bool

Coerces to a true/false boolean. The 1/0 and t/f values are also coerced.

float

Coerce to 64 bit floats.

int

Coerce to a 64 bit integer.

string

Coerces to a string. Generally not necessary since values are extracted as strings.

timestamp

Coerces to a Vector timestamp. strftime specificiers must be used to parse the string.

Troubleshooting

The best place to start with troubleshooting is to check the Vector logs. This is typically located at /var/log/vector.log, then proceed to follow the Troubleshooting Guide.

If the Troubleshooting Guide does not resolve your issue, please:

  1. If encountered a bug, please file a bug report.

  2. If encountered a missing feature, please file a feature request.

  3. If you need help, join our chat/forum community. You can post a question and search previous questions.

Alternatives

Finally, consider the following alternatives:

Resources