grok_parser transform

Accepts `log` events and allows you to parse a field value with Grok.

The grok_parser transform accepts log events and allows you to parse a field value with Grok.

Config File

vector.toml (example)
vector.toml (schema)
vector.toml (specification)
[transforms.my_transform_id]
# REQUIRED - General
type = "grok_parser" # must be: "grok_parser"
inputs = ["my-source-id"]
pattern = "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
# OPTIONAL - General
drop_field = true # default
field = "message" # default
# OPTIONAL - Types
[transforms.my_transform_id.types]
status = "int"
duration = "float"
success = "bool"
timestamp = "timestamp|%s"
timestamp = "timestamp|%+"
timestamp = "timestamp|%F"
timestamp = "timestamp|%a %b %e %T %Y"

Options

Key

Type

Description

REQUIRED - General

type

string

The component type required must be: "grok_parser"

inputs

[string]

A list of upstream source or transform IDs. See Config Composition for more info. required example: ["my-source-id"]

pattern

string

The Grok pattern required example: (see above)

OPTIONAL - General

drop_field

bool

If true will drop the field after parsing. default: true

field

string

The field to execute the pattern against. Must be a string value. default: "message"

OPTIONAL - Types

types.*

string

A definition of mapped field types. They key is the field name and the value is the type. strftime specifiers are supported for the timestamp type. required enum: "string", "int", "float", "bool", and "timestamp\|strftime"

How It Works

Available Patterns

Vector uses the Rust grok library. All patterns listed here are supported. It is recommended to use maintained patterns when possible since they can be improved over time by the community.

Debugging

We recommend the Grok debugger for Grok testing.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Performance

Grok is approximately 50% slower than the regex_parser transform. We plan to add a performance test for this in the future. While this is still plenty fast for most use cases we recommend using the regex_parser transform if you are experiencing performance issues.

Types

By default, extracted (parsed) fields all contain string values. You can coerce these values into types via the types table as shown in the Config File example above. For example:

[transforms.my_transform_id]
# ...
# OPTIONAL - Types
[transforms.my_transform_id.types]
status = "int"
duration = "float"
success = "bool"
timestamp = "timestamp|%s"
timestamp = "timestamp|%+"
timestamp = "timestamp|%F"
timestamp = "timestamp|%a %b %e %T %Y"

The available types are:

Type

Desription

bool

Coerces to a true/false boolean. The 1/0 and t/f values are also coerced.

float

Coerce to 64 bit floats.

int

Coerce to a 64 bit integer.

string

Coerces to a string. Generally not necessary since values are extracted as strings.

timestamp

Coerces to a Vector timestamp. strftime specificiers must be used to parse the string.

Troubleshooting

The best place to start with troubleshooting is to check the Vector logs. This is typically located at /var/log/vector.log, then proceed to follow the Troubleshooting Guide.

If the Troubleshooting Guide does not resolve your issue, please:

  1. If encountered a bug, please file a bug report.

  2. If encountered a missing feature, please file a feature request.

  3. If you need help, join our chat/forum community. You can post a question and search previous questions.

Alternatives

Finally, consider the following alternatives:

Resources