prometheus sink

Exposes `metric` events to Prometheus metrics service.

The prometheus sink is in beta. Please see the current enhancements and bugs for known issues. We kindly ask that you add any missing issues as it will help shape the roadmap of this component.

The prometheus sink exposes metric events to Prometheus metrics service.

Config File

vector.toml (simple)
vector.toml (advanced)
[sinks.my_sink_id]
type = "prometheus" # must be: "prometheus"
inputs = ["my-source-id"]
address = "0.0.0.0:9598"
namespace = "service"
# For a complete list of options see the "advanced" tab above.

Examples

Histograms
Counters
Gauges

This example demonstrates how Vector's internal histogram metric type is exposed via Prometheus' text-based exposition format.

For example, given the following internal Vector histograms:

[
{
"histogram": {
"name": "response_time_s",
"val": 0.243
}
},
{
"histogram": {
"name": "response_time_s",
"val": 0.546
}
}
]
`

The prometheus sink will expose this data in Prometheus' text-based exposition format:

# HELP response_time_s response_time_s
# TYPE response_time_s histogram
response_time_s_bucket{le="0.005"} 0
response_time_s_bucket{le="0.01"} 1
response_time_s_bucket{le="0.025"} 0
response_time_s_bucket{le="0.05"} 1
response_time_s_bucket{le="0.1"} 0
response_time_s_bucket{le="0.25"} 0
response_time_s_bucket{le="0.5"} 0
response_time_s_bucket{le="1.0"} 0
response_time_s_bucket{le="2.5"} 0
response_time_s_bucket{le="5.0"} 0
response_time_s_bucket{le="10.0"} 0
response_time_s_bucket{le="+Inf"} 0
response_time_s_sum 0.789
response_time_s_count 2

Note, the buckets used are those defined by the buckets option, and the buckets above reflect the default buckets.

It's important that your metric units align with your bucket units. The default buckets are in seconds and any metric values passed should also be in seconds. If your metrics are not in seconds you can override the buckets to reflect your units.

This example demonstrates how Vector's internal counter metric type is exposed via Prometheus' text-based exposition format.

For example, given the following internal Vector gauges:

[
{
"counter": {
"name": "logins",
"val": 1
}
},
{
"counter": {
"name": "logins",
"val": 3
}
}
]
`

The prometheus sink will expose this data in Prometheus' text-based exposition format:

# HELP logins logins
# TYPE logins counter
logins 4

Notice that Vector aggregates the metric and exposes the final value.

This example demonstrates how Vector's internal gauge metric type is exposed via Prometheus' text-based exposition format.

For example, given the following internal Vector gauges:

[
{
"gauge": {
"name": "memory_rss",
"val": 250,
"direction": "plus"
}
},
{
"gauge": {
"name": "memory_rss",
"val": 25
"direction": "minus"
}
}
]
`

The prometheus sink will expose this data in Prometheus' text-based exposition format:

# HELP memory_rss memory_rss
# TYPE memory_rss gauge
memory_rss 225

Notice that Vector aggregates the metric and exposes the final value.

How It Works

Buffers

Due to the nature of Prometheus' pull model design the prometheus sink does not utilize a buffer. You can read more about this in the in-memory aggregation section.

Delivery Guarantee

Due to the nature of this component, it offers a best effort delivery guarantee.

Environment Variables

Environment variables are supported through all of Vector's configuration. Simply add ${MY_ENV_VAR} in your Vector configuration file and the variable will be replaced before being evaluated.

You can learn more in the Environment Variables section.

Exposing & Scraping

The prometheus sink exposes data for scraping. The address option determines the address and port the data is made available on. You'll need to configure your networking so that the configured port is accessible by the downstream service doing the scraping.

High Cardinality Names

High cardinality metric names and labels are discouraged by Prometheus and you should consider alternative strategies to reduce the cardinality as this can provide performance and operation problems. In general, high cardinality analysis should be left logs and storages designed for this use case (not Promtheus).

Histogram Buckets

Choosing the appropriate buckets for Prometheus histgorams is a complicated point of discussion. The Histograms and Summaries Prometheus guide provides a good overview of histograms, buckets, summaries, and how you should think about configuring them. The buckets you choose should align with your known range and distribution of values as well as how you plan to report on them. The aforementioned guide provides examples on how you should align them.

Default buckets

The buckets option defines the global default buckets for histograms:

[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]

These defaults are tailored to broadly measure the response time (in seconds) of a network service. Most likely, however, you will be required to define buckets customized to your use case.

Note: These values are in seconds, therefore, your metric values should also be in seconds. If this is not the case you should adjust your metric or buckets to coincide.

In-Memory Aggregation

Like other Prometheus instances, the prometheus sink aggregates metrics in memory which keeps the memory footprint to a minimum if Prometheus fails to scrape the Vector instance over an extended period of time. The downside is that data will be lost if Vector is restarted. This is by design of Prometheus' pull model approach, but is worth noting if restart Vector frequently.

Metric Definitions

By default, Vector will use the original metric names and labels set upon creation. For most cases this makes Vector low friction, allowing you to define the prometheus sink without tedious metrics definitions. You can see examples of this in the examples section.

Metric Types

As described in the metric data model page, Vector offers a variety of metric types. Their support, as as well as their mappings, are described below:

Vector Metric Type

Prometheus Metric Type

counter

'counter'

gauge

'gauge'

histogram

'histogram'

set

⚠️ not supported

-

'summary'

Sets

Prometheus does not have a set type. Sets are generally specific to Statsd, and if a set is received in the prometheus sink it will be dropped, and a rate limited warning level log will be emitted.

Summaries

Summaries are a Prometheus specific type and Vector does not default to them by default. issue #710 addresses the ability to define metrics, including the ability change their types (such as changing them to summary types).

Troubleshooting

The best place to start with troubleshooting is to check the Vector logs. This is typically located at /var/log/vector.log, then proceed to follow the Troubleshooting Guide.

If the Troubleshooting Guide does not resolve your issue, please:

  1. If encountered a bug, please file a bug report.

  2. If encountered a missing feature, please file a feature request.

  3. If you need help, join our chat/forum community. You can post a question and search previous questions.

OOM Errors

If you experience out of memory (OOM) errors it's likely you're using extremely high cardinality metric names or labels. This is discouraged by Prometheus and you should consider alternative strategies to reduce the cardinality. Such as leveraging logs for high cardinality analysis. Issue #387 discusses the ability to provide safeguards around this. We encourage you to add to that discussion with your use case if you find this to be a problem.

Resources