Docker logs
Collect logs from Docker
status: stable
role: daemon
delivery: best effort
acknowledgements: no
egress: stream
state: stateless
output: log
previously known as: docker
Alias
This component was previously called the docker
source. Make sure to update your
Vector configuration to accommodate the name change:
[sources.my_docker_logs_source]
+type = "docker_logs"
-type = "docker"
Warnings
To avoid collecting logs from itself when deployed as a container,
the Docker source uses current hostname to find out which container
it is inside. If a container’s ID matches the hostname, that container
will be excluded.
If you change container’s hostname, consider manually excluding Vector
container using
exclude_containers
.Configuration
Example configurations
{
"sources": {
"my_source_id": {
"type": "docker_logs",
"docker_host": "http://localhost:2375",
"include_containers": [
"include_"
],
"include_labels": [
"com.example.vendor=Timber Inc."
],
"include_images": [
"httpd"
]
}
}
}
[sources.my_source_id]
type = "docker_logs"
docker_host = "http://localhost:2375"
include_containers = [ "include_" ]
include_labels = [ "com.example.vendor=Timber Inc." ]
include_images = [ "httpd" ]
---
sources:
my_source_id:
type: docker_logs
docker_host: http://localhost:2375
include_containers:
- include_
include_labels:
- com.example.vendor=Timber Inc.
include_images:
- httpd
{
"sources": {
"my_source_id": {
"type": "docker_logs",
"docker_host": "http://localhost:2375",
"tls": null,
"auto_partial_merge": true,
"exclude_containers": [
"exclude_"
],
"include_containers": [
"include_"
],
"include_labels": [
"com.example.vendor=Timber Inc."
],
"include_images": [
"httpd"
],
"retry_backoff_secs": 1,
"multiline": null,
"host_key": "host"
}
}
}
[sources.my_source_id]
type = "docker_logs"
docker_host = "http://localhost:2375"
auto_partial_merge = true
exclude_containers = [ "exclude_" ]
include_containers = [ "include_" ]
include_labels = [ "com.example.vendor=Timber Inc." ]
include_images = [ "httpd" ]
retry_backoff_secs = 1
host_key = "host"
---
sources:
my_source_id:
type: docker_logs
docker_host: http://localhost:2375
tls: null
auto_partial_merge: true
exclude_containers:
- exclude_
include_containers:
- include_
include_labels:
- com.example.vendor=Timber Inc.
include_images:
- httpd
retry_backoff_secs: 1
multiline: null
host_key: host
auto_partial_merge
optional boolSetting this to
false
will disable the automatic merging
of partial events.default:
true
docker_host
common optional string literalThe Docker host to connect to. Use an HTTPS URL to enable TLS encryption.
If absent, Vector will try to use
DOCKER_HOST
environment variable.
If DOCKER_HOST
is also absent, Vector will use default Docker local socket
(/var/run/docker.sock
on Unix platforms, //./pipe/docker_engine
on Windows).exclude_containers
optional [string]A list of container IDs or names to match against for
containers you don’t want to collect logs from. Prefix matches
are supported, so you can supply just the first few characters
of the ID or name of containers you want to exclude. This can be
used in conjunction with
include_containers
.host_key
optional string literalThe key name added to each event representing the current host. This can also be globally set via the
global
host_key
option.default:
host
include_containers
common optional [string]A list of container IDs or names to match against for
containers you want to collect logs from. Prefix matches are
supported, so you can supply just the first few characters of
the ID or name of containers you want to include. This can be
used in conjunction with
exclude_containers
.include_images
common optional [string]A list of image names to match against. If not provided, all
images will be included.
include_labels
common optional [string]A list of container object labels to match against when
filtering running containers. This should follow the
described label’s syntax in docker object labels docs.
multiline
optional objectMultiline parsing configuration. If not specified, multiline parsing is disabled.
multiline.condition_pattern
required string regexCondition regex pattern to look for. Exact behavior is configured via
mode
.multiline.mode
required string literal enumMode of operation, specifies how the
condition_pattern
is interpreted.Enum options
Option | Description |
---|---|
continue_past | All consecutive lines matching this pattern, plus one additional line, are included in the group. This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating that the following line is part of the same message. |
continue_through | All consecutive lines matching this pattern are included in the group. The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern. This is useful in cases such as a Java stack trace, where some indicator in the line (such as leading whitespace) indicates that it is an extension of the preceding line. |
halt_before | All consecutive lines not matching this pattern are included in the group. This is useful where a log line contains a marker indicating that it begins a new message. |
halt_with | All consecutive lines, up to and including the first line matching this pattern, are included in the group. This is useful where a log line ends with a termination marker, such as a semicolon. |
multiline.start_pattern
required string regexStart regex pattern to look for as a beginning of the message.
multiline.timeout_ms
required uintThe maximum time to wait for the continuation. Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.
retry_backoff_secs
optional uintThe amount of time to wait before retrying after an error.
default:
1
(seconds)tls
optional objectTLS options to connect to the Docker daemon. This has no effect unless
docker_host
is an HTTPS URL.
If absent, Vector will try to use environment variable DOCKER_CERT_PATH
and then DOCKER_CONFIG
.
If both environment variables are absent, Vector will try to read certificates in ~/.docker/
.Environment variables
DOCKER_CERT_PATH
common optional string literalPath to look for TLS certificates when tls
configuration is absent.
Vector will use:
$DOCKER_CERT_PATH/ca.pem
: CA certificate.$DOCKER_CERT_PATH/cert.pem
: TLS certificate.$DOCKER_CERT_PATH/key.pem
: TLS key.
DOCKER_CONFIG
common optional string literalPath to look for TLS certificates when both
tls
configuration and DOCKER_CERT_PATH
are absent.DOCKER_HOST
common optional string literalThe Docker host to connect to when
docker_host
configuration is absent.Outputs
<component_id>
Default output stream of the component. Use this component’s ID as an input to downstream transforms and sinks.
Output Data
Logs
Log
A Docker log event
container_created_at
required
timestamp
A UTC timestamp representing when the container was created.
Examples
2020-10-10T17:07:36.452332Z
container_id
required
string
literal
The Docker container ID that the log was collected from.
Examples
9b6247364a03
715ebfcee040
container_name
required
string
literal
The Docker container name that the log was collected from.
Examples
evil_ptolemy
nostalgic_stallman
host
required
string
literal
The local hostname, equivalent to the
gethostname
command.Examples
my-host.local
image
required
string
literal
The image name that the container is based on.
Examples
ubuntu:latest
busybox
timberio/vector:latest-alpine
label
required
object
Each container label is inserted with it’s exact key/value pair.
Examples
{
"mylabel": "myvalue"
}
message
required
string
literal
The raw log message.
Examples
Started GET / for 127.0.0.1 at 2012-03-10 14:28:14 +0100
source_type
required
string
literal
The name of the source type.
Examples
docker
stream
required
string
literal
The standard stream that the log was collected from.
Examples
stdout
stderr
timestamp
required
timestamp
The UTC timestamp extracted from the Docker log event.
Examples
2020-10-10T17:07:36.452332Z
Telemetry
Metrics
linkcommunication_errors_total
counterThe total number of errors stemming from communication with the Docker daemon.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
component_discarded_events_total
counterThe number of events dropped by this component.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
component_errors_total
counterThe total number of errors encountered by this component.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
error_type
The type of the error
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
stage
The stage within the component at which the error occurred.
component_received_bytes_total
counterThe number of raw bytes accepted by this component from source origins.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_received_event_bytes_total
counterThe number of event bytes accepted by this component either from
tagged origins like file and uri, or cumulatively from other origins.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_received_events_total
counterThe number of events accepted by this component either from tagged
origins like file and uri, or cumulatively from other origins.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
component_sent_event_bytes_total
counterThe total number of event bytes emitted by this component.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
output
optional
The specific output of the component.
pid
optional
The process ID of the Vector instance.
component_sent_events_total
counterThe total number of events emitted by this component.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
output
optional
The specific output of the component.
pid
optional
The process ID of the Vector instance.
container_metadata_fetch_errors_total
counterThe total number of errors encountered when fetching container metadata.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
container_processed_events_total
counterThe total number of container events processed.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
containers_unwatched_total
counterThe total number of times Vector stopped watching for container logs.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
containers_watched_total
counterThe total number of times Vector started watching for container logs.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
events_in_total
counterThe number of events accepted by this component either from tagged
origins like file and uri, or cumulatively from other origins.
This metric is deprecated and will be removed in a future version.
Use
component_received_events_total
instead.component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
container_name
optional
The name of the container from which the data originated.
file
optional
The file from which the data originated.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the data originated.
peer_path
optional
The pathname from which the data originated.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the data originated.
uri
optional
The sanitized URI from which the data originated.
events_out_total
counterThe total number of events emitted by this component.
This metric is deprecated and will be removed in a future version.
Use
component_sent_events_total
instead.component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
output
optional
The specific output of the component.
pid
optional
The process ID of the Vector instance.
logging_driver_errors_total
counterThe total number of logging driver errors encountered caused by not using either
the
jsonfile
or journald
driver.component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
processed_bytes_total
counterThe number of bytes processed by the component.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
container_name
optional
The name of the container from which the bytes originate.
file
optional
The file from which the bytes originate.
host
optional
The hostname of the system Vector is running on.
mode
optional
The connection mode used by the component.
peer_addr
optional
The IP from which the bytes originate.
peer_path
optional
The pathname from which the bytes originate.
pid
optional
The process ID of the Vector instance.
pod_name
optional
The name of the pod from which the bytes originate.
uri
optional
The sanitized URI from which the bytes originate.
processed_events_total
counterThe total number of events processed by this component.
This metric is deprecated in place of using
component_received_events_total
and
component_sent_events_total
metrics.component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
source_lag_time_seconds
histogramThe difference between the timestamp recorded in each event and the time when it was ingested, expressed as fractional seconds.
component_id
The Vector component ID.
component_kind
The Vector component kind.
component_name
Deprecated, use
component_id
instead. The value is the same as component_id
.component_type
The Vector component type.
host
optional
The hostname of the system Vector is running on.
pid
optional
The process ID of the Vector instance.
Examples
Dummy Logs
Given this event... ```json
{
"stream": "stdout",
"message": "150.75.72.205 - - [03/Oct/2020:16:11:29 +0000] "HEAD /initiatives HTTP/1.1" 504 117"
}
```
[sources.my_source_id]
type = "docker_logs"
include_images = [ "mingrammer/flog" ]
---
sources:
my_source_id:
type: docker_logs
include_images:
- mingrammer/flog
{
"sources": {
"my_source_id": {
"type": "docker_logs",
"include_images": [
"mingrammer/flog"
]
}
}
}
{
"container_created_at": "2020-10-03T16:11:29.443232Z",
"container_id": "fecc98177eca7fb75a2b2186c418bf9a0cd3a05a1169f2e2293bf8987a9d96ab",
"container_name": "flog",
"host": "my-host.local",
"image": "mingrammer/flog",
"message": "150.75.72.205 - - [03/Oct/2020:16:11:29 +0000] \"HEAD /initiatives HTTP/1.1\" 504 117",
"source_type": "docker",
"stream": "stdout"
}
How it works
Merging Split Messages
Docker, by default, splits log messages that exceed 16kb. This can be a
rather frustrating problem because it produces malformed log messages that are
difficult to work with. Vector’s solves this by default, automatically merging
these messages into a single message. You can turn this off via the
auto_partial_merge
option. Furthermore, you can adjust the marker
that we use to determine if an event is partial via the
partial_event_marker_field
option.