AWS Kinesis Streams logs
Publish logs to AWS Kinesis Streams topics
type = "memory"
type = "disk"
|Stores the sink’s buffer on disk. This is less performant, but durable. Data will not be lost between restarts. Will also hold data in memory to enhance performance. WARNING: This may stall the sink if disk performance isn’t on par with the throughput. For comparison, AWS gp2 volumes are usually too slow for common cases.|
|Stores the sink’s buffer in memory. This is more performant, but less durable. Data will be lost if Vector is restarted forcefully.|
|Applies back pressure when the buffer is full. This prevents data loss, but will cause data to pile up on the edge.|
|Drops new data as it’s received. This data is lost. This should be used when performance is the highest priority.|
The compression strategy used to compress the encoded event data before transmission.
Some cloud storage API clients and browsers will handle decompression transparently, so files may not always appear to be compressed depending how they are accessed.
|Gzip standard DEFLATE compression.|
|Formats as a RFC3339 string|
|Formats as a unix timestamp|
See configuration for more info.
endpointis provided it will override this value since the endpoint includes the region.
Vector checks for AWS credentials in the following order:
credential_processcommand in the AWS config file (usually located at
- The AWS credentials file (usually located at
- The IAM instance profile (only works if running on an EC2 instance with an instance profile/role).
If no credentials are found, Vector’s health check fails and an error is logged. If your AWS credentials expire, Vector will automatically search for up-to-date credentials in the places (and order) described above.
assume_roleoption. This is an optional setting that is helpful for a variety of use cases, such as cross account access.
This component buffers & batches data as shown in the diagram above. You’ll notice that Vector treats these concepts differently, instead of treating them as global concepts, Vector treats them as sink specific concepts. This isolates sinks, ensuring services disruptions are contained and delivery guarantees are honored.
Batches are flushed when 1 of 2 conditions are met:
- The batch age meets or exceeds the configured
- The batch size meets or exceeds the configured
Buffers are controlled via the
If you’d like to exit immediately upon a health check failure, you can pass the
vector --config /etc/vector/vector.toml --require-healthy
partition_key_fieldoption. This option presents an alternate field on your event to use as the partition key value instead. This is useful if you have a field already on your event, and it also pairs nicely with the
remaptransform, which enables you to add partition-related metadata to events.
warning-level log event is logged. The field specified in the
partition_key_fieldoption should thus always contain a value.
Adaptive Requst Concurrency is a feature of Vector that does away with static rate limits and automatically optimizes HTTP concurrency limits based on downstream service responses. The underlying mechanism is a feedback loop inspired by TCP congestion control algorithms. Checkout the announcement blog post,
We highly recommend enabling this feature as it improves performance and reliability of Vector and the systems it communicates with.
To enable, set the
request.concurrency option to
[sinks.my-sink] request.concurrency = "adaptive"
If Adaptive Request Concurrency is not for you, you can manually
set static rate limits with the
[sinks.my-sink] request.rate_limit_duration_secs = 1 request.rate_limit_num = 10 request.concurrency = 10