Sinks

Push or expose events in Vector to external systems

Sinks are last in the pipeline, responsible for sending events downstream. These can be service specific sinks, such as vector, elasticsearch, and s3, or generic protocol sinks like http or tcp.

Name

Description

aws_cloudwatch_logs

Batches log events to AWS CloudWatch Logs via the PutLogEvents API endpoint.

aws_kinesis_streams

Batches log events to AWS Kinesis Data Stream via the PutRecords API endpoint.

aws_s3

Batches log events to AWS S3 via the PutObject API endpoint.

blackhole

Streams log and metric events to a blackhole that simply discards data, designed for testing and benchmarking purposes.

clickhouse

Batches log events to Clickhouse via the HTTP Interface.

console

Streams log and metric events to the console, STDOUT or STDERR.

elasticsearch

Batches log events to Elasticsearch via the _bulk API endpoint.

http

Batches log events to a generic HTTP endpoint.

kafka

Streams log events to Apache Kafka via the Kafka protocol.

prometheus

Exposes metric events to Prometheus metrics service.

splunk_hec

Batches log events to a Splunk HTTP Event Collector.

tcp

Streams log events to a TCP connection.

vector

Streams log events to another downstream Vector instance.

+ request a new sink

Healthchecks

Sinks may implement a healthcheck as a means for validating their configuration against the envionment and external systems. Ideally, this allows the system to inform users of problems such as insufficient credentials, unreachable endpoints, non-existant tables, etc. They're not perfect, however, since it's impossible to exhaustively check for issues that may happen at runtime.

Guidelines for writing healthchecks

When implementing healthchecks, we prefer false positives to false negatives. This means we would prefer that a healthcheck pass and the sink then fail than to have the healthcheck fail when the sink would have been able to run successfully.

A common cause of false negatives in healthchecks is performing an operation that the sink itself does not need. For example, listing all of the available S3 buckets and checking that the configured bucket is in that list. The S3 sink doesn't need the ability to list all buckets, and a user that knows that may not have given it permission to do so. In that case, the healthcheck will fail due to bad credentials even through its credentials are sufficient for normal operation.

This leads to a general strategy of mimicking what the sink itself does. Unfortunately, the fact that healthchecks don't have real events available to them leads to some limitations here. The most obvious example of this is with sinks where the exact target of a write depends on the value of some field in the event (e.g. an interpolated Kinesis stream name). It also pops up for sinks where incoming events are expected to conform to a specific schema. In both cases, random test data is reasonably likely to trigger a potentially false negative result. Even in simpler cases, we need to think about the effects of writing test data and whether the user would find that surprising or invasive. The answer usually depends on the system we're interfacing with.

In some cases, like the Kinesis example above, the right thing to do might be nothing at all. If we require dynamic information to figure out what entity (i.e. Kinesis stream in this case) that we're even dealing with, odds are very low that we'll be able to come up with a way to meaningfully validate that it's in working order. It's perfectly valid to have a healthcheck that falls back to doing nothing when there is a data dependency like this.

With all that in mind, here is a simple checklist to go over when writing a new healthcheck:

  • Does this check perform different fallible operations from the sink itself?

  • Does this check have side effects the user would consider undesirable (e.g. data pollution)?

  • Are there situations where this check would fail but the sink would operate normally?

Not all of the answers need to be a hard "no", but we should think about the likelihood that any "yes" would lead to false negatives and balance that against the usefulness of the check as a whole for finding problems. Because we have the option to disable individual healthchecks, there's an escape hatch for users that fall into a false negative circumstance. Our goal should be to minimize the likelihood of users needing to pull that lever while still making a good effort to detect common problems.