Lua

Modify event data using the Lua programming language

status: stable egress: stream state: stateful

Warnings

The lua transform is ~60% slower than the remap transform; we recommend that you use the remap transform whenever possible. The lua transform is designed solely for edge cases not covered by the remap transform and not as a go-to option. If the remap transform doesn’t cover your use case, please open an issue and let us know.

Configuration

hooks

required object
Configures hooks handlers.

hooks.init

optional string literal
A function which is called when the first event comes, before calling hooks.process

hooks.process

required string literal
A function which is called for each incoming event. It can produce new events using emit function.

hooks.shutdown

optional string literal
A function which is called when Vector is stopped. It can produce new events using emit function.

inputs

required [string]

A list of upstream source or transform IDs. Wildcards (*) are supported but must be the last character in the ID.

See configuration for more info.

Array string literal
Examples
[
  "my-source-or-transform-id",
  "prefix-*"
]

search_dirs

optional [string]
A list of directories to search when loading a Lua file via the require function. If not specified, the modules are looked up in the directories of Vector’s configs.
Array string literal
Examples
[
  "/etc/vector/lua"
]

source

optional string
The source which is evaluated when the transform is created.

timers

optional [object]
Configures timers which are executed periodically at given interval.
Array object

version

required string enum
Transform API version. Specifying this version ensures that Vector does not break backward compatibility.
Enum options string literal
OptionDescription
1Lua transform API version 1
2Lua transform API version 2

Telemetry

Metrics

link

events_in_total

counter
The number of events accepted by this component either from tagged origin like file and uri, or cumulatively from other origins.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
container_name optional
The name of the container from which the event originates.
file optional
The file from which the event originates.
mode optional
The connection mode used by the component.
peer_addr optional
The IP from which the event originates.
peer_path optional
The pathname from which the event originates.
pod_name optional
The name of the pod from which the event originates.
uri optional
The sanitized URI from which the event originates.

events_out_total

counter
The total number of events emitted by this component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

memory_used_bytes

gauge
The total memory currently being used by Vector (in bytes).

processed_bytes_total

counter
The number of bytes processed by the component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
container_name optional
The name of the container from which the bytes originate.
file optional
The file from which the bytes originate.
mode optional
The connection mode used by the component.
peer_addr optional
The IP from which the bytes originate.
peer_path optional
The pathname from which the bytes originate.
pod_name optional
The name of the pod from which the bytes originate.
uri optional
The sanitized URI from which the bytes originate.

processed_events_total

counter
The total number of events processed by this component. This metric is deprecated in place of using events_in_total and events_out_total metrics.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.

processing_errors_total

counter
The total number of processing errors encountered by this component.
component_kind required
The Vector component kind.
component_name required
The Vector component name.
component_type required
The Vector component type.
error_type required
The type of the error

Examples

Given this event...
{
  "log": {
    "field_to_remove": "remove me",
    "field_to_rename": "old value"
  }
}
...and this Vector configuration...
{
  "hooks": {
    "process": "function (event, emit)\n\t-- Add root level field\n\tevent.log.field = \"new value\"\n\t-- Add nested field\n\tevent.log.nested.field = \"nested value\"\n\t-- Rename field\n\tevent.log.renamed_field = event.log.field_to_rename\n\tevent.log.field_to_rename = nil\n\t-- Remove fields\n\tevent.log.field_to_remove = nil\n\temit(event)\nend"
  },
  "inputs": null,
  "search_dirs": null,
  "source": null,
  "timers": null,
  "type": null,
  "version": "2"
}
...this Vector log event is produced:
{
  "field": "new value",
  "nested": {
    "field": "nested value"
  },
  "renamed_field": "old value"
}
Given this event...
{
  "metric": {
    "counter": {
      "value": 2
    },
    "kind": "incremental",
    "name": "logins",
    "tags": {
      "tag_to_remove": "remove me",
      "tag_to_rename": "old value"
    }
  }
}
...and this Vector configuration...
{
  "hooks": {
    "process": "function (event, emit)\n\t-- Add tag\n\tevent.metric.tags.tag = \"new value\"\n\t-- Rename tag\n\tevent.metric.tags.renamed_tag = event.log.tag_to_rename\n\tevent.metric.tags.tag_to_rename = nil\n\t-- Remove tag\n\tevent.metric.tags.tag_to_remove = nil\n\temit(event)\nend"
  },
  "inputs": null,
  "search_dirs": null,
  "source": null,
  "timers": null,
  "type": null,
  "version": "2"
}
...this Vector metric event is produced:
{
  "counter": {
    "value": 2
  },
  "kind": "incremental",
  "name": "logins",
  "tags": {
    "renamed_tag": "old value",
    "tag": "new value"
  }
}
Given this event...
{
  "log": {
    "field_to_remove": "remove me",
    "field_to_rename": "old value"
  }
}
...and this Vector configuration...
{
  "hooks": {
    "process": "function (event, emit)\n\t-- Drop event entirely by not calling the `emit` function\nend"
  },
  "inputs": null,
  "search_dirs": null,
  "source": null,
  "timers": null,
  "type": null,
  "version": "2"
}
Given this event...
{
  "log": {
    "value_to_keep": "keep",
    "value_to_remove": "-"
  }
}
...and this Vector configuration...
{
  "hooks": {
    "process": "function (event, emit)\n\t-- Remove all fields where the value is \"-\"\n\tfor f, v in pairs(event) do\n\t\tif v == \"-\" then\n\t\t\tevent[f] = nil\n\t\tend\n\tend\n\temit(event)\nend"
  },
  "inputs": null,
  "search_dirs": null,
  "source": null,
  "timers": null,
  "type": null,
  "version": "2"
}
...this Vector log event is produced:
{
  "value_to_keep": "keep"
}
Given this event...
{
  "log": {
    "timestamp_string": "2020-04-07 06:26:02.643"
  }
}
...and this Vector configuration...
{
  "hooks": {
    "process": "process",
    "source": "  timestamp_pattern = \"(%d%d%d%d)[-](%d%d)[-](%d%d) (%d%d):(%d%d):(%d%d).?(%d*)\"\n  function parse_timestamp(str)\n\tlocal year, month, day, hour, min, sec, millis = string.match(str, timestamp_pattern)\n\tlocal ms = 0\n\tif millis and millis ~= \"\" then\n\t\tms = tonumber(millis)\n\tend\n\treturn {\n\t\tyear    = tonumber(year),\n\t\tmonth   = tonumber(month),\n\t\tday     = tonumber(day),\n\t\thour    = tonumber(hour),\n\t\tmin     = tonumber(min),\n\t\tsec     = tonumber(sec),\n\t\tnanosec = ms * 1000000\n\t}\n  end\n  function process(event, emit)\n\tevent.log.timestamp = parse_timestamp(event.log.timestamp_string)\n\temit(event)\n  end"
  },
  "inputs": null,
  "search_dirs": null,
  "source": null,
  "timers": null,
  "type": null,
  "version": "2"
}
...this Vector log event is produced:
{
  "timestamp": "2020-04-07 06:26:02.643",
  "timestamp_string": "2020-04-07 06:26:02.643"
}
Given this event...
{
  "log": {}
}
...and this Vector configuration...
{
  "hooks": {
    "init": "init",
    "process": "process",
    "shutdown": "shutdown"
  },
  "inputs": null,
  "search_dirs": null,
  "source": "function init()\n\tcount = 0\nend\nfunction process()\n\tcount = count + 1\nend\nfunction timer_handler(emit)\n\temit(make_counter(count))\n\tcount = 0\nend\nfunction shutdown(emit)\n\temit(make_counter(count))\nend\nfunction make_counter(value)\n\treturn metric = {\n\t\tname = \"event_counter\",\n\t\tkind = \"incremental\",\n\t\ttimestamp = os.date(\"!*t\"),\n\t\tcounter = {\n\t\t\tvalue = value\n\t\t}\n\t}\nend",
  "timers": [
    {
      "handler": "timer_handler",
      "interval_seconds": 5
    }
  ],
  "type": null,
  "version": "2"
}
...this Vector metric event is produced:
{
  "counter": {
    "value": 1
  },
  "kind": "incremental",
  "name": "event_counter",
  "tags": {
    "renamed_tag": "old value",
    "tag": "new value"
  }
}

How it works

Event Data Model

The process hook takes an event as its first argument. Events are represented as tables in Lua and follow Vector’s data model exactly. Please refer to Vector’s data model reference for the event schema. How Vector’s types map to Lua’s type are covered below.

Type Mappings

The correspondence between Vector’s data types and Lua data type is summarized by the following table:

Vector TypeLua TypeComment
Stringstring
Integerinteger
Floatnumber
Booleanboolean
TimestamptableThere is no dedicated timestamp type in Lua. Timestamps are represented as tables using the convention defined by os.date and os.time. The table representation of a timestamp contains the fields year, month, day, hour, min, sec, nanosec, yday, wday, and isdst. If such a table is passed from Lua to Vector, the fields yday, wday, and isdst can be omitted. In addition to the os.time representation, Vector supports sub-second resolution with a nanosec field in the table.
Nullempty stringIn Lua setting the value of a table field to nil means deletion of this field. In addition, the length operator # does not work in the expected way with sequences containing nulls. Because of that Null values are encoded as empty strings.
Maptable
ArraysequenceSequences are a special case of tables. Indexes start from 1, following the Lua convention.

Learning Lua

In order to write non-trivial transforms in Lua, one has to have basic understanding of Lua. Because Lua is an easy to learn language, reading a few first chapters of the official book or consulting the manual would suffice.

Search Directories

Vector provides a search_dirs option that allows you to specify absolute paths that will be searched when using the Lua require function. If this option is not set, the directories of the configuration files will be used instead.

State

This component is stateful, meaning its behavior changes based on previous inputs (events). State is not preserved across restarts, therefore state-dependent behavior will reset between restarts and depend on the inputs (events) received since the most recent restart.