Logstash
Collect logs from a Logstash agent
Configuration
Example configurations
{
"sources": {
"my_source_id": {
"type": "logstash",
"address": "0.0.0.0:9000"
}
}
}
[sources.my_source_id]
type = "logstash"
address = "0.0.0.0:9000"
---
sources:
my_source_id:
type: logstash
address: 0.0.0.0:9000
{
"sources": {
"my_source_id": {
"type": "logstash",
"address": "0.0.0.0:9000",
"receive_buffer_bytes": 65536
}
}
}
[sources.my_source_id]
type = "logstash"
address = "0.0.0.0:9000"
receive_buffer_bytes = 65_536
---
sources:
my_source_id:
type: logstash
address: 0.0.0.0:9000
receive_buffer_bytes: 65536
acknowledgements
optional objectControls how acknowledgements are handled by this source.
This setting is deprecated in favor of enabling acknowledgements
at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
acknowledgements.enabled
optional booladdress
required string literalThe socket address to listen for connections on, or systemd{#N}
to use the Nth socket passed by
systemd socket activation.
If a socket address is used, it must include a port.
connection_limit
optional uintkeepalive
optional objectkeepalive.time_secs
optional uintreceive_buffer_bytes
optional uintThe size of the receive buffer used for each connection.
This generally should not need to be changed.
tls
optional objectsources
, adding metadata from the client certificatetls.alpn_protocols
optional [string]Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.
tls.ca_file
optional string literalAbsolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
tls.client_metadata_key
optional string literaltls.crt_file
optional string literalAbsolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file
must also be set.
tls.enabled
optional boolWhether or not to require TLS for incoming/outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file
for
more information.
tls.key_file
optional string literalAbsolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
tls.key_pass
optional string literalPassphrase used to unlock the encrypted key file.
This has no effect unless key_file
is set.
tls.verify_certificate
optional boolEnables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the validity of certificates.
tls.verify_hostname
optional boolEnables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false
unless you understand the risks of not verifying the remote hostname.
Outputs
<component_id>
Output Data
Logs
Line
hello world
127.0.0.1
logstash
The timestamp field will be set to the first one found of the following:
- The
timestamp
field on the event - The
@timestamp
field on the event if it can be parsed as a timestamp - The current timestamp
The assigned field, timestamp
, could be different depending if you have configured
log_schema.timestamp_key
.
2020-10-10T17:07:36.452332Z
Telemetry
Metrics
linkcomponent_received_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_received_events_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_event_bytes_total
countercomponent_id
instead. The value is the same as component_id
.component_sent_events_total
countercomponent_id
instead. The value is the same as component_id
.connection_errors_total
counterconnection_send_ack_errors_total
counterdecode_errors_total
countercomponent_id
instead. The value is the same as component_id
.events_in_total
countercomponent_received_events_total
instead.component_id
instead. The value is the same as component_id
.events_out_total
countercomponent_sent_events_total
instead.component_id
instead. The value is the same as component_id
.open_connections
gaugeprocessed_bytes_total
countercomponent_id
instead. The value is the same as component_id
.processed_events_total
countercomponent_received_events_total
and
component_sent_events_total
metrics.component_id
instead. The value is the same as component_id
.source_lag_time_seconds
histogramcomponent_id
instead. The value is the same as component_id
.Examples
Logstash message from generator input
Given this event...Logstash input config:
```text
input {
generator {
count => 1
}
}
```
Output if sent to stdout logstash output:
```text
{ "@version" => "1", "@timestamp" => 2021-06-14T20:57:14.230Z, "host" => "c082bb583445", "sequence" => 0, "message" => "Hello world!" }
```
[sources.my_source_id]
type = "logstash"
---
sources:
my_source_id:
type: logstash
{
"sources": {
"my_source_id": {
"type": "logstash"
}
}
}
{
"host": "34.33.222.212",
"line": "2021-06-14T20:57:14.230Z c082bb583445 hello world",
"source_type": "logstash"
}
Message from Elastic Beat Heartbeat agent
Given this event...Heartbeat input config:
```yaml
heartbeat.config.monitors:
path: ${path.config}/monitors.d/*.yml
reload.enabled: false
reload.period: 5s
heartbeat.monitors:
- type: http
schedule: '@every 5s'
urls:
- http://google.com
```
Output if sent to stdout output:
```json
{"@timestamp":"2021-06-14T21:25:37.058Z","@metadata":{"beat":"heartbeat","type":"_doc","version":"7.12.1"},"url":{"full":"http://google.com","scheme":"http","domain":"google.com","port":80},"tcp":{"rtt":{"connect":{"us":18504}}},"event":{"dataset":"uptime"},"ecs":{"version":"1.8.0"},"resolve":{"rtt":{"us":7200},"ip":"172.217.4.174"},"summary":{"up":1,"down":0},"http":{"response":{"mime_type":"text/html; charset=utf-8","headers":{"Content-Length":"219","Date":"Mon, 14 Jun 2021 21:25:37 GMT","Server":"gws","X-Xss-Protection":"0","Location":"http://www.google.com/","Expires":"Wed, 14 Jul 2021 21:25:37 GMT","Content-Type":"text/html; charset=UTF-8","Cache-Control":"public, max-age=2592000","X-Frame-Options":"SAMEORIGIN"},"status_code":301,"body":{"hash":"2178eedd5723a6ac22e94ec59bdcd99229c87f3623753f5e199678242f0e90de","bytes":219}},"rtt":{"response_header":{"us":51481},"validate":{"us":52664},"content":{"us":1182},"total":{"us":71585},"write_request":{"us":134}}},"monitor":{"type":"http","status":"up","duration":{"us":79517},"check_group":"0c8c908a-cd57-11eb-85a4-025000000001","ip":"172.217.4.174","timespan":{"gte":"2021-06-14T21:25:37.137Z","lt":"2021-06-14T21:25:42.137Z"},"id":"auto-http-0X993E1F882355CFD2","name":""},"agent":{"hostname":"docker-desktop","ephemeral_id":"9e15e5bc-86d6-4d47-9067-4262b00c5cce","id":"404c8975-a41b-45bd-8d93-3f6c4449e973","name":"docker-desktop","type":"heartbeat","version":"7.12.1"}}
```
[sources.my_source_id]
type = "logstash"
---
sources:
my_source_id:
type: logstash
{
"sources": {
"my_source_id": {
"type": "logstash"
}
}
}
{
"@metadata": {
"beat": "heartbeat",
"type": "_doc",
"version": "7.12.1"
},
"@timestamp": "2021-06-14T21:25:37.058Z",
"agent": {
"ephemeral_id": "9e15e5bc-86d6-4d47-9067-4262b00c5cce",
"hostname": "docker-desktop",
"id": "404c8975-a41b-45bd-8d93-3f6c4449e973",
"name": "docker-desktop",
"type": "heartbeat",
"version": "7.12.1"
},
"ecs": {
"version": "1.8.0"
},
"event": {
"dataset": "uptime"
},
"host": "34.33.222.212",
"http": {
"response": {
"body": {
"bytes": 219,
"hash": "2178eedd5723a6ac22e94ec59bdcd99229c87f3623753f5e199678242f0e90de"
},
"headers": {
"Cache-Control": "public, max-age=2592000",
"Content-Length": "219",
"Content-Type": "text/html; charset=UTF-8",
"Date": "Mon, 14 Jun 2021 21:25:37 GMT",
"Expires": "Wed, 14 Jul 2021 21:25:37 GMT",
"Location": "http://www.google.com/",
"Server": "gws",
"X-Frame-Options": "SAMEORIGIN",
"X-Xss-Protection": "0"
},
"mime_type": "text/html; charset=utf-8",
"status_code": 301
},
"rtt": {
"content": {
"us": 1182
},
"response_header": {
"us": 51481
},
"total": {
"us": 71585
},
"validate": {
"us": 52664
},
"write_request": {
"us": 134
}
}
},
"monitor": {
"check_group": "0c8c908a-cd57-11eb-85a4-025000000001",
"duration": {
"us": 79517
},
"id": "auto-http-0X993E1F882355CFD2",
"ip": "172.217.4.174",
"name": "",
"status": "up",
"timespan": {
"gte": "2021-06-14T21:25:37.137Z",
"lt": "2021-06-14T21:25:42.137Z"
},
"type": "http"
},
"resolve": {
"ip": "172.217.4.174",
"rtt": {
"us": 7200
}
},
"source_type": "logstash",
"summary": {
"down": 0,
"up": 1
},
"tcp": {
"rtt": {
"connect": {
"us": 18504
}
}
},
"timestamp": "2021-06-14T21:25:37.058Z",
"url": {
"domain": "google.com",
"full": "http://google.com",
"port": 80,
"scheme": "http"
}
}
How it works
Acknowledgement support
Sending data from logstash agents to Vector aggregators
Elastic Beats configuration
To configure one of the Elastic Beats agents to forward to a Vector instance, you can use the following output configuration:
output.logstash:
# update these to point to your vector instance
hosts: ["127.0.0.1:5044"]
Logstash configuration
To configure Logstash to forward to a Vector instance, you can use the following output configuration:
output {
lumberjack {
# update these to point to your vector instance
hosts => ["127.0.0.1"]
port => 5044
ssl_certificate => "/path/to/certificate.crt"
}
}
Note that Logstash requires SSL to be configured.
Transport Layer Security (TLS)
tls.*
options.