Elasticsearch output

Elasticsearch output DEFAULT

The Elasticsearch output sends events directly to Elasticsearch using the Elasticsearch HTTP API.

When sending data to a secured cluster through the output, APM Server can use any of the following authentication methods:

Configuration optionsedit

You can specify the following options in the section of the config file:

The enabled config is a boolean setting to enable or disable the output. If set to , the output is disabled.

The default value is .

The list of Elasticsearch nodes to connect to. The events are distributed to these nodes in round robin order. If one node becomes unreachable, the event is automatically sent to another node. Each Elasticsearch node can be defined as a or . For example: , or . If no port is specified, is used.

When a node is defined as an , the scheme and path are taken from the and config options.

output.elasticsearch: hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] protocol: https path: /elasticsearch

In the previous example, the Elasticsearch nodes are available at and .

The gzip compression level. Setting this value to disables compression. The compression level must be in the range of (best speed) to (best compression).

Increasing the compression level will reduce the network usage but will increase the cpu usage.

The default value is .

Configure escaping of HTML in strings. Set to to enable escaping.

The default value is .

The number of workers per configured host publishing events to Elasticsearch. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).

The default value is .

Instead of using a username and password, you can use API keys to secure communication with Elasticsearch. The value must be the ID of the API key and the API key joined by a colon: .

See Grant access using API keys for more information.

The basic authentication username for connecting to Elasticsearch.

This user needs the privileges required to publish events to Elasticsearch. To create a user like this, see Create a writer user.

The basic authentication password for connecting to Elasticsearch.

Dictionary of HTTP parameters to pass within the url with index operations.

The name of the protocol Elasticsearch is reachable on. The options are: or . The default is . However, if you specify a URL for , the value of is overridden by whatever scheme you specify in the URL.

An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where Elasticsearch listens behind an HTTP reverse proxy that exports the API under a custom prefix.

Custom HTTP headers to add to each request created by the Elasticsearch output. Example:

output.elasticsearch.headers: X-My-Header: Header contents

It is possible to specify multiple header values for the same header name by separating them with a comma.

The URL of the proxy to use when connecting to the Elasticsearch servers. The value may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed. If a value is not specified through the configuration file then proxy environment variables are used. See the Go documentation for more information about the environment variables.

The index name to write events to when you’re using daily indices. The default is (for example, ). If you change this setting, you need to configure the and options (see Elasticsearch index template).

When index lifecycle management (ILM) is enabled, the default is (for example, ). Defining a custom here will disable Customize index lifecycle management.

You can set the index dynamically by using a format string to access any event field. For example, this configuration uses the field, to separate events into different indices:

output.elasticsearch: hosts: ["http://localhost:9200"] index: "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}\"

is a field managed by Beats that is added to every document; It holds the current version of APM Server. We recommend including in the index name to avoid mapping issues when you upgrade APM Server.

To learn how to add custom fields to events, see the option.

See the setting for other ways to set the index dynamically.

An array of index selector rules. Each rule specifies the index to use for events that match the rule. During publishing, APM Server uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the setting is missing or no rule matches, the setting is used.

Rule settings:

The index format string to use. If this string contains field references, such as , the fields must exist, or the rule fails.
A dictionary that takes the value returned by and maps it to a new name.
The default string value to use if does not find a match.
A condition that must succeed in order to execute the current rule.

The following example sets the index based on whether the field contains the specified string:

output.elasticsearch: hosts: ["http://localhost:9200"] indices: - index: "apm-%{[observer.version]}-sourcemap" when.contains: processor.event: "sourcemap" - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}" when.contains: processor.event: "error" - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}" when.contains: processor.event: "transaction" - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}" when.contains: processor.event: "span" - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}" when.contains: processor.event: "metric" - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}" when.contains: processor.event: "onboarding"

refers to APM Server. We recommend including in the name to avoid mapping issues when you upgrade APM Server.

This is the default configuration for APM Server when ILM is disabled, and results in indices named in the following format: For example: .

The following example sets the index by taking the name returned by the format string and mapping it to a new name that’s used for the index:

output.elasticsearch: hosts: ["http://localhost:9200"] indices: - index: "%{[processor.event]}" mappings: sourcemap: "apm-sourcemap" error: "apm-error" transaction: "apm-transaction" span: "apm-span" metric: "apm-metric" onboarding: "apm-onboarding" default: "apm"

This configuration results in indices named , , etc.

The setting simplifies the configuration, but is limited to string values. You cannot specify format strings within the mapping pairs.

A format string value that specifies the ingest node pipeline to write events to.

output.elasticsearch: hosts: ["http://localhost:9200"] pipeline: my_pipeline_id

For more information, see Parse data using ingest node pipelines.

You can set the ingest node pipeline dynamically by using a format string to access any event field. For example, this configuration uses the field, , to set the pipeline for each event:

output.elasticsearch: hosts: ["http://localhost:9200"] pipeline: "%{[processor.event]}_pipeline"

With this configuration, all events with are sent to a pipeline named . Similarly, all events with are sent to a pipeline named .

The default pipeline is . To disable this, or any other pipeline, set .

To learn how to add custom fields to events, see the option.

See the setting for other ways to set the ingest node pipeline dynamically.

An array of pipeline selector rules. Each rule specifies the ingest node pipeline to use for events that match the rule. During publishing, APM Server uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the setting is missing or no rule matches, the setting is used.

Rule settings:

The pipeline format string to use. If this string contains field references, such as , the fields must exist, or the rule fails.
A dictionary that takes the value returned by and maps it to a new name.
The default string value to use if does not find a match.
A condition that must succeed in order to execute the current rule.

The following example sends events to a specific pipeline based on whether the field contains the specified string:

output.elasticsearch: hosts: ["http://localhost:9200"] pipelines: - pipeline: "sourcemap_pipeline" when.contains: processor.event: "sourcemap" - pipeline: "error_pipeline" when.contains: processor.event: "error" - pipeline: "transaction_pipeline" when.contains: processor.event: "transaction" - pipeline: "span_pipeline" when.contains: processor.event: "span" - pipeline: "metric_pipeline" when.contains: processor.event: "metric" - pipeline: "onboarding_pipeline" when.contains: processor.event: "onboarding"

The following example sets the pipeline by taking the name returned by the format string and mapping it to a new name that’s used for the pipeline:

output.elasticsearch: hosts: ["http://localhost:9200"] pipelines: - pipeline: "%{[processor.event]}" mappings: sourcemap: "sourcemap_pipeline" error: "error_pipeline" transaction: "transaction_pipeline" span: "span_pipeline" metric: "metric_pipeline" onboarding: "onboarding_pipeline" default: "apm_pipeline"

With this configuration, all events with are sent to a pipeline named , all events with are sent to a pipeline named , etc.

Defining any pipeline will deactivate the default pipeline.

For more information about ingest node pipelines, see Parse data using ingest node pipelines.

The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped.

Set to a value less than 0 to retry until all events are published.

The default is 3.

The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50.

Events can be collected into batches. APM Server will split batches larger than into multiple batches.

Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.

Setting to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch.

The number of seconds to wait before trying to reconnect to Elasticsearch after a network error. After waiting seconds, APM Server tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to . After a successful connection, the backoff timer is reset. The default is .

The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error. The default is .

The http request timeout in seconds for the Elasticsearch request. The default is 90.

Configuration options for Kerberos authentication.

See Kerberos for more information.

Sours: https://www.elastic.co/guide/en/apm/server/current/elasticsearch-output.html

The Elasticsearch output sends events directly to Elasticsearch using the Elasticsearch HTTP API.

When sending data to a secured cluster through the output, Filebeat can use any of the following authentication methods:

Configuration optionsedit

You can specify the following options in the section of the config file:

The enabled config is a boolean setting to enable or disable the output. If set to , the output is disabled.

The default value is .

The list of Elasticsearch nodes to connect to. The events are distributed to these nodes in round robin order. If one node becomes unreachable, the event is automatically sent to another node. Each Elasticsearch node can be defined as a or . For example: , or . If no port is specified, is used.

When a node is defined as an , the scheme and path are taken from the and config options.

output.elasticsearch: hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] protocol: https path: /elasticsearch

In the previous example, the Elasticsearch nodes are available at and .

The gzip compression level. Setting this value to disables compression. The compression level must be in the range of (best speed) to (best compression).

Increasing the compression level will reduce the network usage but will increase the cpu usage.

The default value is .

Configure escaping of HTML in strings. Set to to enable escaping.

The default value is .

The number of workers per configured host publishing events to Elasticsearch. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).

The default value is .

Instead of using a username and password, you can use API keys to secure communication with Elasticsearch. The value must be the ID of the API key and the API key joined by a colon: .

See Grant access using API keys for more information.

The basic authentication username for connecting to Elasticsearch.

This user needs the privileges required to publish events to Elasticsearch. To create a user like this, see Create a publishing user.

The basic authentication password for connecting to Elasticsearch.

Dictionary of HTTP parameters to pass within the url with index operations.

The name of the protocol Elasticsearch is reachable on. The options are: or . The default is . However, if you specify a URL for , the value of is overridden by whatever scheme you specify in the URL.

An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where Elasticsearch listens behind an HTTP reverse proxy that exports the API under a custom prefix.

Custom HTTP headers to add to each request created by the Elasticsearch output. Example:

output.elasticsearch.headers: X-My-Header: Header contents

It is possible to specify multiple header values for the same header name by separating them with a comma.

If set to all proxy settings, including and variables are ignored.

The URL of the proxy to use when connecting to the Elasticsearch servers. The value may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed. If a value is not specified through the configuration file then proxy environment variables are used. See the Go documentation for more information about the environment variables.

Additional headers to send to proxies during CONNECT requests.

The index name to write events to when you’re using daily indices. The default is , for example, . If you change this setting, you also need to configure the and options (see Elasticsearch index template).

If you are using the pre-built Kibana dashboards, you also need to set the option (see Kibana dashboards).

When index lifecycle management (ILM) is enabled, the default is , for example, . Custom settings are ignored when ILM is enabled. If you’re sending events to a cluster that supports index lifecycle management, see Index lifecycle management (ILM) to learn how to change the index name.

You can set the index dynamically by using a format string to access any event field. For example, this configuration uses a custom field, , to set the index:

output.elasticsearch: hosts: ["http://localhost:9200"] index: "%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}"

We recommend including in the name to avoid mapping issues when you upgrade.

With this configuration, all events with are sent to an index named , and all events with are sent to an index named .

To learn how to add custom fields to events, see the option.

See the setting for other ways to set the index dynamically.

An array of index selector rules. Each rule specifies the index to use for events that match the rule. During publishing, Filebeat uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the setting is missing or no rule matches, the setting is used.

Similar to , defining custom will disable Index lifecycle management (ILM).

Rule settings:

The index format string to use. If this string contains field references, such as , the fields must exist, or the rule fails.
A dictionary that takes the value returned by and maps it to a new name.
The default string value to use if does not find a match.
A condition that must succeed in order to execute the current rule. All the conditions supported by processors are also supported here.

The following example sets the index based on whether the field contains the specified string:

output.elasticsearch: hosts: ["http://localhost:9200"] indices: - index: "warning-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: message: "WARN" - index: "error-%{[agent.version]}-%{+yyyy.MM.dd}" when.contains: message: "ERR"

This configuration results in indices named and (plus the default index if no matches are found).

The following example sets the index by taking the name returned by the format string and mapping it to a new name that’s used for the index:

output.elasticsearch: hosts: ["http://localhost:9200"] indices: - index: "%{[fields.log_type]}" mappings: critical: "sev1" normal: "sev2" default: "sev3"

This configuration results in indices named , , and .

The setting simplifies the configuration, but is limited to string values. You cannot specify format strings within the mapping pairs.

A format string value that specifies the ingest pipeline to write events to.

output.elasticsearch: hosts: ["http://localhost:9200"] pipeline: my_pipeline_id

For more information, see Parse data using an ingest pipeline.

You can set the ingest pipeline dynamically by using a format string to access any event field. For example, this configuration uses a custom field, , to set the pipeline for each event:

output.elasticsearch: hosts: ["http://localhost:9200"] pipeline: "%{[fields.log_type]}_pipeline"

With this configuration, all events with are sent to a pipeline named , and all events with are sent to a pipeline named .

To learn how to add custom fields to events, see the option.

See the setting for other ways to set the ingest pipeline dynamically.

An array of pipeline selector rules. Each rule specifies the ingest pipeline to use for events that match the rule. During publishing, Filebeat uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the setting is missing or no rule matches, the setting is used.

Rule settings:

The pipeline format string to use. If this string contains field references, such as , the fields must exist, or the rule fails.
A dictionary that takes the value returned by and maps it to a new name.
The default string value to use if does not find a match.
A condition that must succeed in order to execute the current rule. All the conditions supported by processors are also supported here.

The following example sends events to a specific pipeline based on whether the field contains the specified string:

output.elasticsearch: hosts: ["http://localhost:9200"] pipelines: - pipeline: "warning_pipeline" when.contains: message: "WARN" - pipeline: "error_pipeline" when.contains: message: "ERR"

The following example sets the pipeline by taking the name returned by the format string and mapping it to a new name that’s used for the pipeline:

output.elasticsearch: hosts: ["http://localhost:9200"] pipelines: - pipeline: "%{[fields.log_type]}" mappings: critical: "sev1_pipeline" normal: "sev2_pipeline" default: "sev3_pipeline"

With this configuration, all events with are sent to , all events with are sent to a , and all other events are sent to .

For more information about ingest pipelines, see Parse data using an ingest pipeline.

Filebeat ignores the setting and retries indefinitely.

The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50.

Events can be collected into batches. Filebeat will split batches larger than into multiple batches.

Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.

Setting to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch.

The number of seconds to wait before trying to reconnect to Elasticsearch after a network error. After waiting seconds, Filebeat tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to . After a successful connection, the backoff timer is reset. The default is .

The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error. The default is .

The http request timeout in seconds for the Elasticsearch request. The default is 90.

Configuration options for Kerberos authentication.

See Kerberos for more information.

Specifies the behavior when the elasticsearch cluster explicitly rejects documents, for example on mapping conflicts.

The default behaviour, when an event is explicitly rejected by elasticsearch it is dropped.

output.elasticsearch: hosts: ["http://localhost:9200"] non_indexable_policy.drop: ~

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

On an explicit rejection, this policy will retry the event in the next batch. However, the target index will change to index specified. In addition, the structure of the event will be change to the following fields:

message
Contains the escaped json of the original event.
error.type
Contains the status code
error.message
Contains status returned by elasticsearch, describing the reason
The index to send rejected events to.
output.elasticsearch: hosts: ["http://localhost:9200"] non_indexable_policy.dead_letter_index: index: "my-dead-letter-index"
Sours: https://www.elastic.co/guide/en/beats/filebeat/master/elasticsearch-output.html
  1. Wv aaa schools
  2. Wilshire 5000
  3. Obd2 health meter
  4. Wolf drawing anime

When you specify Elasticsearch for the output, Filebeat sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API.

To enable SSL, just add to all URLs defined under hosts.

If the Elasticsearch nodes are defined by , then add to the yaml file.

If you are indexing large amounts of time-series data, you might also want to configure Filebeat to use index lifecycle management. For more information about configuring and using index lifecycle management with Filebeat, see Set up index lifecycle management.

Configuration optionsedit

You can specify the following options in the section of the config file:

The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled.

The default value is true.

The list of Elasticsearch nodes to connect to. The events are distributed to these nodes in round robin order. If one node becomes unreachable, the event is automatically sent to another node. Each Elasticsearch node can be defined as a or . For example: , or . If no port is specified, is used.

When a node is defined as an , the scheme and path are taken from the and config options.

output.elasticsearch: hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] protocol: https path: /elasticsearch

In the previous example, the Elasticsearch nodes are available at and .

The gzip compression level. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression).

Increasing the compression level will reduce the network usage but will increase the cpu usage.

The default value is 0.

Configure escaping of HTML in strings. Set to to disable escaping.

The default value is .

The number of workers per configured host publishing events to Elasticsearch. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).

The default value is 1.

The basic authentication username for connecting to Elasticsearch.

The basic authentication password for connecting to Elasticsearch.

Dictionary of HTTP parameters to pass within the url with index operations.

The name of the protocol Elasticsearch is reachable on. The options are: or . The default is . However, if you specify a URL for , the value of is overridden by whatever scheme you specify in the URL.

An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where Elasticsearch listens behind an HTTP reverse proxy that exports the API under a custom prefix.

Custom HTTP headers to add to each request created by the Elasticsearch output. Example:

output.elasticsearch.headers: X-My-Header: Header contents

It is generally possible to specify multiple header values for the same header name by separating them with a comma.

The URL of the proxy to use when connecting to the Elasticsearch servers. The value may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed. If a value is not specified through the configuration file then proxy environment variables are used. See the Go documentation for more information about the environment variables.

The index name to write events to. The default is (for example, ). If you change this setting, you also need to configure the and options (see Load the Elasticsearch index template). If you are using the pre-built Kibana dashboards, you also need to set the option (see Load the Kibana dashboards).

You can set the index dynamically by using a format string to access any event field. For example, this configuration uses a custom field, , to set the index:

output.elasticsearch: hosts: ["http://localhost:9200"] index: "%{[fields.log_type]}-%{[beat.version]}-%{+yyyy.MM.dd}"

We recommend including in the name to avoid mapping issues when you upgrade.

With this configuration, all events with are sent to an index named , and all events with are sent to an index named .

To learn how to add custom fields to events, see the option.

See the setting for other ways to set the index dynamically.

An array of index selector rules. Each rule specifies the index to use for events that match the rule. During publishing, Filebeat uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the setting is missing or no rule matches, the setting is used.

Rule settings:

The index format string to use. If this string contains field references, such as , the fields must exist, or the rule fails.
A dictionary that takes the value returned by and maps it to a new name.
The default string value to use if does not find a match.
A condition that must succeed in order to execute the current rule. All the conditions supported by processors are also supported here.

The following example sets the index based on whether the field contains the specified string:

output.elasticsearch: hosts: ["http://localhost:9200"] indices: - index: "warning-%{[beat.version]}-%{+yyyy.MM.dd}" when.contains: message: "WARN" - index: "error-%{[beat.version]}-%{+yyyy.MM.dd}" when.contains: message: "ERR"

This configuration results in indices named and (plus the default index if no matches are found).

The following example sets the index by taking the name returned by the format string and mapping it to a new name that’s used for the index:

output.elasticsearch: hosts: ["http://localhost:9200"] indices: - index: "%{[fields.log_type]}" mappings: critical: "sev1" normal: "sev2" default: "sev3"

This configuration results in indices named , , and .

The setting simplifies the configuration, but is limited to string values. You cannot specify format strings within the mapping pairs.

A format string value that specifies the ingest node pipeline to write events to.

output.elasticsearch: hosts: ["http://localhost:9200"] pipeline: my_pipeline_id

For more information, see Parse data by using ingest node.

You can set the ingest node pipeline dynamically by using a format string to access any event field. For example, this configuration uses a custom field, , to set the pipeline for each event:

output.elasticsearch: hosts: ["http://localhost:9200"] pipeline: "%{[fields.log_type]}_pipeline"

With this configuration, all events with are sent to a pipeline named , and all events with are sent to a pipeline named .

To learn how to add custom fields to events, see the option.

See the setting for other ways to set the ingest node pipeline dynamically.

An array of pipeline selector rules. Each rule specifies the ingest node pipeline to use for events that match the rule. During publishing, Filebeat uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the setting is missing or no rule matches, the setting is used.

Rule settings:

The pipeline format string to use. If this string contains field references, such as , the fields must exist, or the rule fails.
A dictionary that takes the value returned by and maps it to a new name.
The default string value to use if does not find a match.
A condition that must succeed in order to execute the current rule. All the conditions supported by processors are also supported here.

The following example sends events to a specific pipeline based on whether the field contains the specified string:

output.elasticsearch: hosts: ["http://localhost:9200"] pipelines: - pipeline: "warning_pipeline" when.contains: message: "WARN" - pipeline: "error_pipeline" when.contains: message: "ERR"

The following example sets the pipeline by taking the name returned by the format string and mapping it to a new name that’s used for the pipeline:

output.elasticsearch: hosts: ["http://localhost:9200"] pipelines: - pipeline: "%{[fields.log_type]}" mappings: critical: "sev1_pipeline" normal: "sev2_pipeline" default: "sev3_pipeline"

With this configuration, all events with are sent to , all events with are sent to a , and all other events are sent to .

For more information about ingest node pipelines, see Parse data by using ingest node.

Filebeat ignores the setting and retries indefinitely.

The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50.

Events can be collected into batches. Filebeat will split batches larger than into multiple batches.

Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.

Setting to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch.

The number of seconds to wait before trying to reconnect to Elasticsearch after a network error. After waiting seconds, Filebeat tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to . After a successful connection, the backoff timer is reset. The default is 1s.

The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error. The default is 60s.

The http request timeout in seconds for the Elasticsearch request. The default is 90.

Configuration options for SSL parameters like the certificate authority to use for HTTPS-based connections. If the section is missing, the host CAs are used for HTTPS connections to Elasticsearch.

See Specify SSL settings for more information.

Sours: https://www.elastic.co/guide/en/beats/filebeat/6.8/elasticsearch-output.html
Elasticdump : Export and migrate data from Elasticsearch

Configure the Elasticsearch outputedit

The Elasticsearch output sends events directly to Elasticsearch by using the Elasticsearch HTTP API.

Compatibility: This output works with all compatible versions of Elasticsearch. See the Elastic Support Matrix.

This example configures an Elasticsearch output called in the file:

outputs: default: type: elasticsearch hosts: [127.0.0.1:9200] username: elastic password: changeme

Elasticsearch output configuration settingsedit

The output type supports the following settings, grouped by category. Many of these settings have sensible defaults that allow you to run Elastic Agent with minimal configuration.

Commonly used settingsedit

Setting Description

(boolean) Enables or disables the output. If set to , the output is disabled.

Default:

(list) The list of Elasticsearch nodes to connect to. The events are distributed to these nodes in round robin order. If one node becomes unreachable, the event is automatically sent to another node. Each Elasticsearch node can be defined as a or . For example: , or . If no port is specified, is used.

When a node is defined as an , the scheme and path are taken from the and settings.

outputs: default: type: elasticsearch hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] protocol: https path: /elasticsearch

In this example, the Elasticsearch nodes are available at and .

(string) The name of the protocol Elasticsearch is reachable on. The options are: or . The default is . However, if you specify a URL for , the value of is overridden by whatever scheme you specify in the URL.

(boolean) If set to , all proxy settings, including and variables, are ignored.

Default:

(string) Additional headers to send to proxies during CONNECT requests.

(string) The URL of the proxy to use when connecting to the Elasticsearch servers. The value may be either a complete URL or a , in which case the scheme is assumed. If a value is not specified through the configuration file then proxy environment variables are used. See the Go documentation for more information about the environment variables.

Authentication settingsedit

Settings for authenticating with Elasticsearch.

When sending data to a secured cluster through the output, Elastic Agent can use any of the following authentication methods:

Basic authentication credentialsedit

outputs: default: type: elasticsearch hosts: ["https://myEShost:9200"] username: "your-username" password: "your-password"
Setting Description

(string) The basic authentication password for connecting to Elasticsearch.

(string) The basic authentication username for connecting to Elasticsearch.

This user needs the privileges required to publish events to Elasticsearch.

Token-based (API key) authenticationedit

outputs: default: type: elasticsearch hosts: ["https://myEShost:9200"] api_key: "KnR6yE41RrSowb0kQ0HWoA"
Setting Description

(string) Instead of using a username and password, you can use API keys to secure communication with Elasticsearch. The value must be the ID of the API key and the API key joined by a colon: .

Public Key Infrastructure (PKI) certificatesedit

outputs: default: type: elasticsearch hosts: ["https://myEShost:9200"] ssl.certificate: "/etc/pki/client/cert.pem" ssl.key: "/etc/pki/client/cert.key"

There are a number of SSL configuration settings available depending on whether you are configuring the client, server, or both. See the following tables for available settings:

Table 1. Common configuration options

Setting Description

(string) This configures a certificate pin that you can use to ensure that a specific certificate is part of the verified chain.

The pin is a base64 encoded string of the SHA-256 of the certificate.

This check is not a replacement for the normal SSL validation, but it adds additional validation. If this setting is used with set to , the check will always fail because it will not receive any verified chains.

(list) The list of cipher suites to use. The first entry has the highest priority. If this option is omitted, the Go crypto library’s default suites are used (recommended). Note that TLS 1.3 cipher suites are not individually configurable in Go, so they are not included in this list.

The following cipher suites are available:

  • ECDHE-ECDSA-AES-128-CBC-SHA
  • ECDHE-ECDSA-AES-128-CBC-SHA256: TLS 1.2 only. Disabled by default.
  • ECDHE-ECDSA-AES-128-GCM-SHA256: TLS 1.2 only.
  • ECDHE-ECDSA-AES-256-CBC-SHA
  • ECDHE-ECDSA-AES-256-GCM-SHA384: TLS 1.2 only.
  • ECDHE-ECDSA-CHACHA20-POLY1305: TLS 1.2 only.
  • ECDHE-ECDSA-RC4-128-SHA: Disabled by default. RC4 not recommended.
  • ECDHE-RSA-3DES-CBC3-SHA
  • ECDHE-RSA-AES-128-CBC-SHA
  • ECDHE-RSA-AES-128-CBC-SHA256: TLS 1.2 only. Disabled by default.
  • ECDHE-RSA-AES-128-GCM-SHA256: TLS 1.2 only.
  • ECDHE-RSA-AES-256-CBC-SHA
  • ECDHE-RSA-AES-256-GCM-SHA384: TLS 1.2 only.
  • ECDHE-RSA-CHACHA20-POLY1205: TLS 1.2 only.
  • ECDHE-RSA-RC4-128-SHA: Disabled by default. RC4 not recommended.
  • RSA-3DES-CBC3-SHA
  • RSA-AES-128-CBC-SHA
  • RSA-AES-128-CBC-SHA256: TLS 1.2 only. Disabled by default.
  • RSA-AES-128-GCM-SHA256: TLS 1.2 only.
  • RSA-AES-256-CBC-SHA
  • RSA-AES-256-GCM-SHA384: TLS 1.2 only.
  • RSA-RC4-128-SHA: Disabled by default. RC4 not recommended.

Here is a list of acronyms used in defining the cipher suites:

  • 3DES: Cipher suites using triple DES
  • AES-128/256: Cipher suites using AES with 128/256-bit keys.
  • CBC: Cipher using Cipher Block Chaining as block cipher mode.
  • ECDHE: Cipher suites using Elliptic Curve Diffie-Hellman (DH) ephemeral key exchange.
  • ECDSA: Cipher suites using Elliptic Curve Digital Signature Algorithm for authentication.
  • GCM: Galois/Counter mode is used for symmetric key cryptography.
  • RC4: Cipher suites using RC4.
  • RSA: Cipher suites using RSA.
  • SHA, SHA256, SHA384: Cipher suites using SHA-1, SHA-256 or SHA-384.

(list) The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange).

The following elliptic curve types are available:

(boolean) Enables or disables the SSL configuration.

Default:

SSL settings are disabled if either is set to or the section is missing.

(list) List of allowed SSL/TLS versions. If the SSL/TLS server supports none of the specified versions, the connection will be dropped during or after the handshake. The list of allowed protocol versions include: , for TLS version 1.0, , , , and .

Default:

Table 2. Client configuration options

Setting Description

(string) The path to the certificate for SSL client authentication. This setting is only required if is specified. If is not specified, client authentication is not available, and the connection might fail if the server requests client authentication. If the SSL server does not require client authentication, the certificate will be loaded, but not requested or used by the server.

Example:

ssl.certificate: "/path/to/cert.pem"

When this setting is configured, the setting is also required.

Specify a path, or embed a certificate directly in the configuration:

ssl.certificate: | -----BEGIN CERTIFICATE----- CERTIFICATE CONTENT APPEARS HERE -----END CERTIFICATE-----

(list) The list of root certificates for verifications (required). If is empty or not set, the system keystore is used. If is self-signed, the host system needs to trust that CA cert as well.

Example:

ssl.certificate_authorities: ["/path/to/root/ca.pem"]

Specify a list of files that Elastic Agent will read, or embed a certificate directly in the configuration:

ssl.certificate_authorities: - | -----BEGIN CERTIFICATE----- CERTIFICATE CONTENT APPEARS HERE -----END CERTIFICATE-----

(string) The client certificate key used for client authentication. Only required if is configured.

Example:

ssl.key: "/path/to/cert.key"

Specify a path, or embed the private key directly in the configuration:

ssl.key: | -----BEGIN PRIVATE KEY----- KEY CONTENT APPEARS HERE -----END PRIVATE KEY-----

(string) The passphrase used to decrypt an encrypted key stored in the configured file.

(string) Controls the verification of server certificates. Valid values are:

Verifies that the provided certificate is signed by a trusted authority (CA) and also verifies that the server’s hostname (or IP address) matches the names identified within the certificate.
Verifies that the provided certificate is signed by a trusted authority (CA) and also verifies that the server’s hostname (or IP address) matches the names identified within the certificate. If the Subject Alternative Name is empty, it returns an error.
Verifies that the provided certificate is signed by a trusted authority (CA), but does not perform any hostname verification.
Performs no verification of the server’s certificate. This mode disables many of the security benefits of SSL/TLS and should only be used after cautious consideration. It is primarily intended as a temporary diagnostic mechanism when attempting to resolve TLS errors; its use in production environments is strongly discouraged.

Default:

Table 3. Server configuration options

Setting Description

(string) The path to the certificate for SSL server authentication. If the certificate is not specified, startup will fail.

Example:

ssl.certificate: "/path/to/server/cert.pem"

When this setting is configured, the setting is also required.

Specify a path, or embed a certificate directly in the configuration:

ssl.certificate: | -----BEGIN CERTIFICATE----- CERTIFICATE CONTENT APPEARS HERE -----END CERTIFICATE-----

(list) The list of root certificates for client verifications is only required if is configured. If is empty or not set, and is configured, the system keystore is used. If is self-signed, the host system needs to trust that CA cert too.

Example:

ssl.certificate_authorities: ["/path/to/root/ca.pem"]

Specify a list of files that Elastic Agent will read, or embed a certificate directly in the configuration:

ssl.certificate_authorities: - | -----BEGIN CERTIFICATE----- CERTIFICATE CONTENT APPEARS HERE -----END CERTIFICATE-----

(string) Configures client authentication. The valid options are:

Disables client authentication.
When a client certificate is supplied, the server will verify it.
Requires clients to provide a valid certificate.

Default: (if is set); otherwise,

(string) The server certificate key used for authentication (required).

Example:

ssl.key: "/path/to/server/cert.key"

Specify a path, or embed the private key directly in the configuration:

ssl.key: | -----BEGIN PRIVATE KEY----- KEY CONTENT APPEARS HERE -----END PRIVATE KEY-----

(string) The passphrase used to decrypt an encrypted key stored in the configured file.

(string) Configures the type of TLS renegotiation to support. The valid options are:

Disables renegotiation.
Allows a remote server to request renegotiation once per connection.
Allows a remote server to request renegotiation repeatedly.

Default:

(string) Controls the verification of client certificates. Valid values are:

Verifies that the provided certificate is signed by a trusted authority (CA) and also verifies that the server’s hostname (or IP address) matches the names identified within the certificate.
Verifies that the provided certificate is signed by a trusted authority (CA) and also verifies that the server’s hostname (or IP address) matches the names identified within the certificate. If the Subject Alternative Name is empty, it returns an error.
Verifies that the provided certificate is signed by a trusted authority (CA), but does not perform any hostname verification.
Performs no verification of the server’s certificate. This mode disables many of the security benefits of SSL/TLS and should only be used after cautious consideration. It is primarily intended as a temporary diagnostic mechanism when attempting to resolve TLS errors; its use in production environments is strongly discouraged.

Default:

The following encryption types are supported:

  • aes128-cts-hmac-sha1-96
  • aes128-cts-hmac-sha256-128
  • aes256-cts-hmac-sha1-96
  • aes256-cts-hmac-sha384-192
  • des3-cbc-sha1-kd
  • rc4-hmac

Example output config with Kerberos password-based authentication:

outputs: default: type: elasticsearch hosts: ["http://my-elasticsearch.elastic.co:9200"] kerberos.auth_type: password kerberos.username: "elastic" kerberos.password: "changeme" kerberos.config_path: "/etc/krb5.conf" kerberos.realm: "ELASTIC.CO"

The service principal name for the Elasticsearch instance is constructed from these options. Based on this configuration, the name would be:

Setting Description

(string) The type of authentication to use with Kerberos KDC:

When specified, also set and .
When specified, also set and . The keytab must contain the keys of the selected principal, or authentication fails.

Default:

(string) Path to the . Elastic Agent uses this setting to find the Kerberos KDC to retrieve a ticket.

(boolean) Enables or disables the Kerberos configuration.

Kerberos settings are disabled if either is set to or the section is missing.

(boolean) If , enables Kerberos FAST authentication. This may conflict with some Active Directory installations.

Default:

(string) If is , provide the path to the keytab of the selected principal.

(string) If is , provide a password for the selected principal.

(string) Name of the realm where the output resides.

(string) Name of the principal used to connect to the output.

Data parsing, filtering, and manipulation settingsedit

Settings used to parse, filter, and transform data.

Setting Description

(boolean) Configures escaping of HTML in strings. Set to to enable escaping.

Default:

(string) A format string value that specifies the ingest pipeline to write events to.

outputs: default: type: elasticsearchoutput.elasticsearch: hosts: ["http://localhost:9200"] pipeline: my_pipeline_id

You can set the ingest pipeline dynamically by using a format string to access any event field. For example, this configuration uses a custom field, , to set the pipeline for each event:

outputs: default: type: elasticsearch hosts: ["http://localhost:9200"] pipeline: "%{[fields.log_type]}_pipeline"

With this configuration, all events with are sent to a pipeline named , and all events with are sent to a pipeline named .

To learn how to add custom fields to events, see the option.

See the setting for other ways to set the ingest pipeline dynamically.

An array of pipeline selector rules. Each rule specifies the ingest pipeline to use for events that match the rule. During publishing, Elastic Agent uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the setting is missing or no rule matches, the setting is used.

Rule settings:

The pipeline format string to use. If this string contains field references, such as , the fields must exist, or the rule fails.
A dictionary that takes the value returned by and maps it to a new name.
The default string value to use if does not find a match.
A condition that must succeed in order to execute the current rule.

All the conditions supported by processors are also supported here.

The following example sends events to a specific pipeline based on whether the field contains the specified string:

outputs: default: type: elasticsearch hosts: ["http://localhost:9200"] pipelines: - pipeline: "warning_pipeline" when.contains: message: "WARN" - pipeline: "error_pipeline" when.contains: message: "ERR"

The following example sets the pipeline by taking the name returned by the format string and mapping it to a new name that’s used for the pipeline:

outputs: default: type: elasticsearch hosts: ["http://localhost:9200"] pipelines: - pipeline: "%{[fields.log_type]}" mappings: critical: "sev1_pipeline" normal: "sev2_pipeline" default: "sev3_pipeline"

With this configuration, all events with are sent to , all events with are sent to a , and all other events are sent to .

Settings that modify the HTTP requests sent to Elasticsearch.

Setting Description

Custom HTTP headers to add to each request created by the Elasticsearch output.

Example:

outputs: default: type: elasticsearch headers: X-My-Header: Header contents

Specify multiple header values for the same header name by separating them with a comma.

Dictionary of HTTP parameters to pass within the url with index operations.

(string) An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where Elasticsearch listens behind an HTTP reverse proxy that exports the API under a custom prefix.

Performance tuning settingsedit

Settings that may affect performance.

Setting Description

(string) The number of seconds to wait before trying to reconnect to Elasticsearch after a network error. After waiting seconds, Elastic Agent tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to . After a successful connection, the backoff timer is reset.

Default:

(string) The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error.

Default:

(int) The maximum number of events to bulk in a single Elasticsearch bulk API index request.

Events can be collected into batches. Elastic Agent will split batches larger than into multiple batches.

Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.

Setting to values less than or equal to 0 turns off the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch.

Default:

(int) The gzip compression level. Set this value to to disable compression. The compression level must be in the range of (best speed) to (best compression).

Increasing the compression level reduces network usage but increases CPU usage.

Default:

(int) The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped.

Set to a value less than 0 to retry until all events are published.

Default:

(string) The HTTP request timeout in seconds for the Elasticsearch request.

Default:

(int) The number of workers per configured host publishing events to {output-type}. This is best used with load balancing mode enabled. Example: If you have two hosts and three workers, in total six workers are started (three for each host).

Default:

Sours: https://www.elastic.co/guide/en/fleet/master/elasticsearch-output.html

Output elasticsearch

Elasticsearch output pluginedit

  • Plugin version: v11.0.5
  • Released on: 2021-08-30
  • Changelog

For other versions, see the Versioned plugin docs.

For questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.

Elasticsearch provides near real-time search and analytics for all types of data. The Elasticsearch output plugin can store both time series datasets (such as logs, events, and metrics) and non-time series data in Elasticsearch.

You can learn more about Elasticsearch on the website landing page or in the Elasticsearch documentation.

Compatibility Note

When connected to Elasticsearch 7.x, modern versions of this plugin don’t use the document-type when inserting documents, unless the user explicitly sets .

If you are using an earlier version of Logstash and wish to connect to Elasticsearch 7.x, first upgrade Logstash to version 6.8 to ensure it picks up changes to the Elasticsearch index template.

If you are using a custom , ensure your template uses the document-type before connecting to Elasticsearch 7.x.

Hosted Elasticsearch Service on Elastic Cloudedit

You can run Elasticsearch on your own hardware or use our hosted Elasticsearch Service that is available on AWS, GCP, and Azure. Try the Elasticsearch Service for free.

Compatibility with the Elastic Common Schema (ECS)edit

This plugin will persist events to Elasticsearch in the shape produced by your pipeline, and cannot be used to re-shape the event structure into a shape that complies with ECS. To produce events that fully comply with ECS, you will need to populate ECS-defined fields throughout your pipeline definition.

However, the Elasticsearch Index Templates it manages can be configured to be ECS-compatible by setting . By having an ECS-compatible template in place, we can ensure that Elasticsearch is prepared to create and index fields in a way that is compatible with ECS, and will correctly reject events with fields that conflict and cannot be coerced.

The Elasticsearch output plugin can store both time series datasets (such as logs, events, and metrics) and non-time series data in Elasticsearch.

The data stream options are recommended for indexing time series datasets (such as logs, metrics, and events) into Elasticsearch:

Data stream configuration examplesedit

Example: Basic default configuration

output { elasticsearch { hosts => "hostname" data_stream => "true" } }

This example shows the minimal settings for processing data streams. Events with fields are routed to the appropriate data streams. If the fields are missing, routing defaults to .

Example: Customize data stream name

output { elasticsearch { hosts => "hostname" data_stream => "true" data_stream_type => "metrics" data_stream_dataset => "foo" data_stream_namespace => "bar" } }

Writing to different indices: best practicesedit

You cannot use dynamic variable substitution when is and when using .

If you’re sending events to the same Elasticsearch cluster, but you’re targeting different indices you can:

  • use different Elasticsearch outputs, each one with a different value for the parameter
  • use one Elasticsearch output and use the dynamic variable substitution for the parameter

Each Elasticsearch output is a new client connected to the cluster:

  • it has to initialize the client and connect to Elasticsearch (restart time is longer if you have more clients)
  • it has an associated connection pool

In order to minimize the number of open connections to Elasticsearch, maximize the bulk size and reduce the number of "small" bulk requests (which could easily fill up the queue), it is usually more efficient to have a single Elasticsearch output.

Example:

output { elasticsearch { index => "%{[some_field][sub_field]}-%{+YYYY.MM.dd}" } }

What to do in case there is no field in the event containing the destination index prefix?

You can use the filter and conditionals to add a field to set the destination index for each event. The fields will not be sent to Elasticsearch.

Example:

filter { if [log_type] in [ "test", "staging" ] { mutate { add_field => { "[@metadata][target_index]" => "test-%{+YYYY.MM}" } } } else if [log_type] == "production" { mutate { add_field => { "[@metadata][target_index]" => "prod-%{+YYYY.MM.dd}" } } } else { mutate { add_field => { "[@metadata][target_index]" => "unknown-%{+YYYY}" } } } } output { elasticsearch { index => "%{[@metadata][target_index]}" } }

The retry policy has changed significantly in the 8.1.1 release. This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience either partial or total failures. The bulk API sends batches of requests to an HTTP endpoint. Error codes for the HTTP request are handled differently than error codes for individual documents.

HTTP requests to the bulk API are expected to return a 200 response code. All other response codes are retried indefinitely.

The following document errors are handled as follows:

  • 400 and 404 errors are sent to the dead letter queue (DLQ), if enabled. If a DLQ is not enabled, a log message will be emitted, and the event will be dropped. See DLQ Policy for more info.
  • 409 errors (conflict) are logged as a warning and dropped.

Note that 409 exceptions are no longer retried. Please set a higher value if you experience 409 exceptions. It is more performant for Elasticsearch to retry these exceptions than this plugin.

Mapping (404) errors from Elasticsearch can lead to data loss. Unfortunately mapping errors cannot be handled without human intervention and without looking at the field that caused the mapping mismatch. If the DLQ is enabled, the original events causing the mapping errors are stored in a file that can be processed at a later time. Often times, the offending field can be removed and re-indexed to Elasticsearch. If the DLQ is not enabled, and a mapping error happens, the problem is logged as a warning, and the event is dropped. See Dead letter queues (DLQ) for more information about processing events in the DLQ.

Index Lifecycle Managementedit

The Index Lifecycle Management feature requires plugin version or higher.

This feature requires an Elasticsearch instance of 6.6.0 or higher with at least a Basic license

Logstash can use Index Lifecycle Management to automate the management of indices over time.

The use of Index Lifecycle Management is controlled by the setting. By default, this setting detects whether the Elasticsearch instance supports ILM, and uses it if it is available. can also be set to or to override the automatic detection, or disable ILM.

This will overwrite the index settings and adjust the Logstash template to write the necessary settings for the template to support index lifecycle management, including the index policy and rollover alias to be used.

Logstash will create a rollover alias for the indices to be written to, including a pattern for how the actual indices will be named, and unless an ILM policy that already exists has been specified, a default policy will also be created. The default policy is configured to rollover an index when it reaches either 50 gigabytes in size, or is 30 days old, whichever happens first.

The default rollover alias is called , with a default pattern for the rollover index of , which will name indices on the date that the index is rolled over, followed by an incrementing number. Note that the pattern must end with a dash and a number that will be incremented.

See the Rollover API documentation for more details on naming.

The rollover alias, ilm pattern and policy can be modified.

See config below for an example:

output { elasticsearch { ilm_rollover_alias => "custom" ilm_pattern => "000001" ilm_policy => "custom_policy" } }

Custom ILM policies must already exist on the Elasticsearch cluster before they can be used.

If the rollover alias or pattern is modified, the index template will need to be overwritten as the settings and are automatically written to the template

If the index property is supplied in the output definition, it will be overwritten by the rollover alias.

This plugin attempts to send batches of events to the Elasticsearch Bulk API as a single request. However, if a batch exceeds 20MB we break it up into multiple bulk requests. If a single document exceeds 20MB it is sent as a single request.

This plugin uses the JVM to lookup DNS entries and is subject to the value of networkaddress.cache.ttl, a global setting for the JVM.

As an example, to set your DNS TTL to 1 second you would set the environment variable to .

Keep in mind that a connection with keepalive enabled will not reevaluate its DNS value while the keepalive is in effect.

This plugin supports request compression, and handles compressed responses from Elasticsearch.

To enable request compression, use the setting on this plugin.

Authentication to a secure Elasticsearch cluster is possible using one of the /, or options.

Authorization to a secure Elasticsearch cluster requires permission at index level and permissions at cluster level. The permission at cluster level is necessary to perform periodic connectivity checks.

Elasticsearch Output Configuration Optionsedit

This plugin supports the following configuration options plus the Common Options described later.

SettingInput typeRequired

string

No

password

No

string

No

a valid filesystem path

No

password

No

string

No

hash

No

string, one of

No

boolean

No

string

No

string

No

boolean

No

string

No

boolean

No

string

No

string

No

string

No

array

No

string

No

uri

No

boolean

No

string, one of

No

string

No

string

No

string

No

string

No

a valid filesystem path

No

password

No

boolean

No

hash

No

string

No

password

No

string

No

string

No

number

No

number

No

uri

No

number

No

number

No

number

No

number

No

string

No

string

No

string

No

string, one of

No

string

No

boolean

No

boolean

No

number

No

string

No

boolean

No

boolean

No

a valid filesystem path

No

string

No

boolean

No

number

No

a valid filesystem path

No

password

No

string

No

string

No

number

No

string

No

string, one of

No

Also see Common Options for a list of options supported by all output plugins.

 

  • Value type is string
  • Default value is for data streams, and for non-time series data.

The Elasticsearch action to perform. Valid actions are:

  • : indexes a document (an event from Logstash).
  • : deletes a document by id (An id is required for this action)
  • : indexes a document, fails if a document by that id already exists in the index.
  • : updates a document by id. Update has a special case where you can upsert — update a document if not already present. See the option. NOTE: This does not work and is not supported in Elasticsearch 1.x. Please upgrade to ES 2.x or greater to use this feature with Logstash!
  • A sprintf style string to change the action based on the content of the event. The value would use the foo field for the action

For more details on actions, check out the Elasticsearch bulk API documentation.

  • Value type is password
  • There is no default value for this setting.

Authenticate using Elasticsearch API key. Note that this option also requires enabling the option.

Format is where and are as returned by the Elasticsearch Create API key API.

  • Value type is string
  • There is no default value for this setting.

HTTP Path to perform the _bulk requests to this defaults to a concatenation of the path parameter and "_bulk"

  • Value type is path
  • There is no default value for this setting.

The .cer or .pem file to validate the server’s certificate.

  • Value type is password
  • There is no default value for this setting.

Cloud authentication string ("<username>:<password>" format) is an alternative for the / pair.

For more details, check out the Logstash-to-Cloud documentation.

  • Value type is string
  • There is no default value for this setting.

Cloud ID, from the Elastic Cloud web console. If set should not be used.

For more details, check out the Logstash-to-Cloud documentation.

  • Value can be any of: , and
  • Default is in Logstash 7.x and starting in Logstash 8.0.

Defines whether data will be indexed into an Elasticsearch data stream. The other settings will be used only if this setting is enabled.

Logstash handles the output as a data stream when the supplied configuration is compatible with data streams and this value is set to .

edit

  • Value type is boolean
  • Default value is .

Automatically routes events by deriving the data stream name using specific event fields with the format.

If enabled, the event fields will take precedence over the , , and settings, but will fall back to them if any of the fields are missing from the event.

  • Value type is string
  • Default value is .

The data stream dataset used to construct the data stream at index time.

edit

  • Value type is string
  • Default value is .

The data stream namespace used to construct the data stream at index time.

edit

  • Value type is boolean
  • Default value is

Automatically adds and syncs the event fields if they are missing from the event. This ensures that fields match the name of the data stream that is receiving events.

If existing event fields do not match the data stream name and is disabled, the event fields will be overwritten with a warning.

  • Value type is string
  • Default value is .

The data stream type used to construct the data stream at index time. Currently, only , and are supported.

  • Value type is boolean
  • Default value is

Enable for update mode. Create a new document with source if doesn’t exist in Elasticsearch.

  • Value type is string
  • There is no default value for this setting.

The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.

  • Value type is string
  • There is no default value for this setting.
  • This option is deprecated

This value is ignored and has no effect for Elasticsearch clusters .

This sets the document type to write events to. Generally you should try to write only similar events to the same type. String expansion works here. If you don’t set a value for this option:

  • for elasticsearch clusters 8.x: no value will be used;
  • for elasticsearch clusters 7.x: the value of _doc will be used;
  • for elasticsearch clusters 6.x: the value of doc will be used;
  • for elasticsearch clusters 5.x and below: the event’s type field will be used, if the field is not present the value of doc will be used.
  • Value type is string
  • Supported values are:

    • : does not provide ECS-compatible templates
    • : provides defaults that are compatible with v1 of the Elastic Common Schema
  • Default value depends on which version of Logstash is running:

    • When Logstash provides a setting, its value is used as the default
    • Otherwise, the default value is .

Controls this plugin’s compatibility with the Elastic Common Schema (ECS), including the installation of ECS-compatible index templates. The value of this setting affects the default values of:

edit

  • Value type is array
  • Default value is

Set the Elasticsearch errors in the whitelist that you don’t want to log. A useful example is when you want to skip all 409 errors which are .

  • Value type is hash
  • There is no default value for this setting.

Pass a set of key value pairs as the headers sent in each request to an elasticsearch node. The headers will be used for any kind of request (_bulk request, template installation, health checks and sniffing). These custom headers will be overidden by settings like .

  • Value type is string
  • There is no default value for this setting.

HTTP Path where a HEAD request is sent when a backend is marked down the request is sent in the background to see if it has come back again before it is once again eligible to service requests. If you have custom firewall rules you may need to change this

  • Value type is uri
  • Default value is

Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the parameter. Remember the protocol uses the http address (eg. 9200, not 9300).

Examples:

`"127.0.0.1"` `["127.0.0.1:9200","127.0.0.2:9200"]` `["http://127.0.0.1"]` `["https://127.0.0.1:9200"]` `["https://127.0.0.1:9200/mypath"]` (If using a proxy on a subpath)

Exclude dedicated master nodes from the list to prevent Logstash from sending bulk requests to the master nodes. This parameter should reference only data or client nodes in Elasticsearch.

Any special characters present in the URLs here MUST be URL escaped! This means should be put in as for instance.

  • Value type is boolean
  • Default value is

Enable gzip compression on requests.

This setting allows you to reduce this plugin’s outbound network traffic by compressing each bulk request to Elasticsearch.

This output plugin reads compressed responses from Elasticsearch regardless of the value of this setting.

  • Value can be any of: , ,
  • Default value is

The default setting of will automatically enable Index Lifecycle Management, if the Elasticsearch cluster is running Elasticsearch version or higher with the ILM feature enabled, and disable it otherwise.

Setting this flag to will disable the Index Lifecycle Management feature, even if the Elasticsearch cluster supports ILM. Setting this flag to will enable Index Lifecycle Management feature, if the Elasticsearch cluster supports it. This is required to enable Index Lifecycle Management on a version of Elasticsearch earlier than version .

This feature requires a Basic License or above to be installed on an Elasticsearch cluster version 6.6.0 or later.

  • Value type is string
  • Default value is

Pattern used for generating indices managed by Index Lifecycle Management. The value specified in the pattern will be appended to the write alias, and incremented automatically when a new index is created by ILM.

Date Math can be used when specifying an ilm pattern, see Rollover API docs for details.

Updating the pattern will require the index template to be rewritten.

The pattern must finish with a dash and a number that will be automatically incremented when indices rollover.

  • Value type is string
  • Default value is

Modify this setting to use a custom Index Lifecycle Management policy, rather than the default. If this value is not set, the default policy will be automatically installed into Elasticsearch

If this setting is specified, the policy must already exist in Elasticsearch cluster.

  • Value type is string
  • Default value depends on whether is enabled:

    • ECS Compatibility disabled:
    • ECS Compatibility enabled:

The rollover alias is the alias where indices managed using Index Lifecycle Management will be written to.

If both and are specified, takes precedence.

Updating the rollover alias will require the index template to be rewritten.

does NOT support dynamic variable substitution as does.

  • Value type is string
  • Default value depends on whether is enabled:

    • ECS Compatibility disabled:
    • ECS Compatibility enabled:

The index to write events to. This can be dynamic using the syntax. The default value will partition your indices by day so you can more easily delete old data or only search specific date ranges. Indexes may not contain uppercase characters. For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}. Logstash uses Joda formats for the index pattern from event timestamp.

  • Value type is path
  • There is no default value for this setting.

The keystore used to present a certificate to the server. It can be either .jks or .p12

  • Value type is password
  • There is no default value for this setting.

Set the keystore password

  • Value type is boolean
  • Default value is

From Logstash 1.3 onwards, a template is applied to Elasticsearch during Logstash’s startup if one with the name does not already exist. By default, the contents of this template is the default template for which always matches indices based on the pattern . Should you require support for other index names, or would like to change the mappings in the template in general, a custom template can be specified by setting to the path of a template file.

Setting to false disables this feature. If you require more control over template creation, (e.g. creating indices dynamically based on field names) you should set to false and use the REST API to apply your templates manually.

  • Value type is hash
  • There is no default value for this setting.

Pass a set of key value pairs as the URL query string. This query string is added to every host listed in the hosts configuration. If the hosts list contains urls that already have query strings, the one specified here will be appended.

  • Value type is string
  • Default value is

For child documents, ID of the associated parent. This can be dynamic using the syntax.

  • Value type is password
  • There is no default value for this setting.

Password to authenticate to a secure Elasticsearch cluster

  • Value type is string
  • There is no default value for this setting.

HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remaps the root path for the Elasticsearch HTTP API lives. Note that if you use paths as components of URLs in the hosts field you may not also set this field. That will raise an error at startup

  • Value type is string
  • Default value is

Set which ingest pipeline you wish to execute for an event. You can also use event dependent configuration here like . The pipeline parameter won’t be set if the value resolves to empty string ("").

  • Value type is number
  • Default value is

While the output tries to reuse connections efficiently we have a maximum. This sets the maximum number of open connections the output will create. Setting this too low may mean frequently closing / opening connections which is bad.

  • Value type is number
  • Default value is

While the output tries to reuse connections efficiently we have a maximum per endpoint. This sets the maximum number of open connections per endpoint the output will create. Setting this too low may mean frequently closing / opening connections which is bad.

  • Value type is uri
  • There is no default value for this setting.

Set the address of a forward HTTP proxy. This setting accepts only URI arguments to prevent leaking credentials. An empty string is treated as if proxy was not set. This is useful when using environment variables e.g. .

  • Value type is number
  • Default value is

How frequently, in seconds, to wait between resurrection attempts. Resurrection is the process by which backend endpoints marked down are checked to see if they have come back to life

edit

  • Value type is number
  • Default value is

Set initial interval in seconds between bulk retries. Doubled on each retry up to

  • Value type is number
  • Default value is

Set max interval in seconds between bulk retries.

  • Value type is number
  • Default value is

The number of times Elasticsearch should internally retry an update/upserted document.

  • Value type is string
  • There is no default value for this setting.

A routing override to be applied to all processed events. This can be dynamic using the syntax.

  • Value type is string
  • Default value is

Set script name for scripted update mode

Example:

output { elasticsearch { script => "ctx._source.message = params.event.get('message')" } }
  • Value type is string
  • Default value is

Set the language of the used script. When using indexed (stored) scripts on Elasticsearch 6.0 and higher, you must set this parameter to (empty string).

  • Value can be any of: , ,
  • Default value is

Define the type of script referenced by "script" variable inline : "script" contains inline script indexed : "script" contains the name of script directly indexed in elasticsearch file : "script" contains the name of script stored in elasticsearch’s config directory

  • Value type is string
  • Default value is

Set variable name passed to script (scripted update)

  • Value type is boolean
  • Default value is

if enabled, script is in charge of creating non-existent document (scripted update)

  • Value type is boolean
  • Default value is

This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list. For Elasticsearch 5.x and 6.x any nodes with (on by default) will be added to the hosts list, excluding master-only nodes.

  • Value type is number
  • Default value is

How long to wait, in seconds, between sniffing attempts

  • Value type is string
  • There is no default value for this setting.

HTTP Path to be used for the sniffing requests the default value is computed by concatenating the path value and "_nodes/http" if sniffing_path is set it will be used as an absolute path do not use full URL here, only paths, e.g. "/sniff/_nodes/http"

  • Value type is boolean
  • There is no default value for this setting.

Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme is specified in the URLs listed in hosts. If no explicit protocol is specified plain HTTP will be used. If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in hosts

  • Value type is path
  • There is no default value for this setting.

You can set the path to your own template here, if you so desire. If not set, the included template will be used.

  • Value type is string
  • Default value depends on whether is enabled:

    • ECS Compatibility disabled:
    • ECS Compatibility enabled:

This configuration option defines how the template is named inside Elasticsearch. Note that if you have used the template management features and subsequently change this, you will need to prune the old template manually, e.g.

where is whatever the former setting was.

  • Value type is boolean
  • Default value is

The template_overwrite option will always overwrite the indicated template in Elasticsearch with either the one indicated by template or the included one. This option is set to false by default. If you always want to stay up to date with the template provided by Logstash, this option could be very useful to you. Likewise, if you have your own template file managed by puppet, for example, and you wanted to be able to update it regularly, this option could help there as well.

Please note that if you are using your own customized version of the Logstash template (logstash), setting this to true will make Logstash to overwrite the "logstash" template (i.e. removing all customized settings)

  • Value type is number
  • Default value is

Set the timeout, in seconds, for network operations and requests sent Elasticsearch. If a timeout occurs, the request will be retried.

  • Value type is path
  • There is no default value for this setting.

The truststore to validate the server’s certificate. It can be either .jks or .p12. Use either or .

  • Value type is password
  • There is no default value for this setting.

Set the truststore password

  • Value type is string
  • Default value is

Set upsert content for update mode. Create a new document with this parameter as json string if doesn’t exists

  • Value type is string
  • There is no default value for this setting.

Username to authenticate to a secure Elasticsearch cluster

edit

  • Value type is number
  • Default value is

How long to wait before checking for a stale connection to determine if a keepalive request is needed. Consider setting this value lower than the default, possibly to 0, if you get connection errors regularly.

This client is based on Apache Commons. Here’s how the Apache Commons documentation describes this option: "Defines period of inactivity in milliseconds after which persistent connections must be re-validated prior to being leased to the consumer. Non-positive value passed to this method disables connection validation. This check helps detect connections that have become stale (half-closed) while kept inactive in the pool."

  • Value type is string
  • There is no default value for this setting.

The version to use for indexing. Use sprintf syntax like to use a field value here. See the versioning support blog for more information.

  • Value can be any of: , , , ,
  • There is no default value for this setting.

The version_type to use for indexing. See the versioning support blog and Version types in the Elasticsearch documentation.

The following configuration options are supported by all output plugins:

  • Value type is boolean
  • Default value is

Disable or enable metric logging for this specific plugin instance. By default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

  • Value type is string
  • There is no default value for this setting.

Add a unique to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type. For example, if you have 2 elasticsearch outputs. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.

output { elasticsearch { id => "my_plugin_id" } }

Variable substitution in the field only supports environment variables and does not support the use of values from the secret store.

Sours: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
Elasticsearch basic concepts - Introduction to elasticsearch - elasticsearch architecture in depth

Elasticsearch output pluginedit

  • Plugin version: v9.4.0
  • Released on: 2019-02-06
  • Changelog

For other versions, see the Versioned plugin docs.

For plugins not bundled by default, it is easy to install by running . See Working with plugins for more details.

For questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.

Compatibility Note

Starting with Elasticsearch 5.3, there’s an HTTP setting called . If this option is set to , and you are using Logstash 2.4 through 5.2, you need to update the Elasticsearch output plugin to version 6.2.5 or higher.

If you plan to use the Kibana web interface, use the Elasticsearch output plugin to get your log data into Elasticsearch.

This output only speaks the HTTP protocol. HTTP is the preferred protocol for interacting with Elasticsearch as of Logstash 2.0. We strongly encourage the use of HTTP over the node protocol for a number of reasons. HTTP is only marginally slower, yet far easier to administer and work with. When using the HTTP protocol one may upgrade Elasticsearch versions without having to upgrade Logstash in lock-step.

You can learn more about Elasticsearch at https://www.elastic.co/products/elasticsearch

Template management for Elasticsearch 5.xedit

Index template for this version (Logstash 5.0) has been changed to reflect Elasticsearch’s mapping changes in version 5.0. Most importantly, the subfield for string multi-fields has changed from to to match ES default behavior.

Users installing ES 5.x and LS 5.x

This change will not affect you and you will continue to use the ES defaults.

Users upgrading from LS 2.x to LS 5.x with ES 5.x

LS will not force upgrade the template, if template already exists. This means you will still use for sub-fields coming from 2.x. If you choose to use the new template, you will have to reindex your data after the new template is installed.

The retry policy has changed significantly in the 8.1.1 release. This plugin uses the Elasticsearch bulk API to optimize its imports into Elasticsearch. These requests may experience either partial or total failures. The bulk API sends batches of requests to an HTTP endpoint. Error codes for the HTTP request are handled differently than error codes for individual documents.

HTTP requests to the bulk API are expected to return a 200 response code. All other response codes are retried indefinitely.

The following document errors are handled as follows:

  • 400 and 404 errors are sent to the dead letter queue (DLQ), if enabled. If a DLQ is not enabled, a log message will be emitted, and the event will be dropped. See DLQ Policy for more info.
  • 409 errors (conflict) are logged as a warning and dropped.

Note that 409 exceptions are no longer retried. Please set a higher value if you experience 409 exceptions. It is more performant for Elasticsearch to retry these exceptions than this plugin.

Mapping (404) errors from Elasticsearch can lead to data loss. Unfortunately mapping errors cannot be handled without human intervention and without looking at the field that caused the mapping mismatch. If the DLQ is enabled, the original events causing the mapping errors are stored in a file that can be processed at a later time. Often times, the offending field can be removed and re-indexed to Elasticsearch. If the DLQ is not enabled, and a mapping error happens, the problem is logged as a warning, and the event is dropped. See Dead Letter Queues for more information about processing events in the DLQ.

Index Lifecycle Managementedit

The Index Lifecycle Management feature requires plugin version or higher.

This feature requires an Elasticsearch instance of 6.6.0 or higher with at least a Basic license

Logstash can use Index Lifecycle Management to automate the management of indices over time.

The use of Index Lifecycle Management is controlled by the setting. The flag can be set to , (default) or .

The use of Index Lifecycle Management is controlled by the setting. The flag can be set to , (default) or .

Setting to will automatically detect whether the Elasticsearch instance is version and above and has Index Lifecycle Management enabled, and will enable its usage if so.

Setting to will enable the Index Lifecycle Management feature if it is available on the Elasticsearch cluster - the plugin will not start if is set to and the Elasticsearch cluster does not support ILM.

Enabling ILM support will overwrite the index settings and adjust the Logstash template to write the necessary settings for the template to support index lifecycle management, including the index policy and rollover alias to be used.

Logstash will create a rollover alias for the indices to be written to, including a pattern for how the actual indices will be named, and unless an ILM policy that already exists has been specified, a default policy will also be created. The default policy is configured to rollover an index when it reaches either 50 gigabytes in size, or is 30 days old, whichever happens first.

The default rollover alias is called , with a default pattern for the rollover index of , which will name indices on the date that the index is rolled over, followed by an incrementing number. Note that the pattern must end with a dash and a number that will be incremented.

See the Rollover API documentation for more details on naming.

The rollover alias, ilm pattern and policy can be modified.

See config below for an example:

output { elasticsearch { ilm_enabled => true ilm_rollover_alias => "custom" ilm_pattern => "000001" ilm_policy => "custom_policy" } }

Custom ILM policies must already exist on the Elasticsearch cluster before they can be used.

If the rollover alias or pattern is modified, the index template will need to be overwritten as the settings and are automatically written to the template

If the index property is supplied in the output definition, it will be overwritten by the rollover alias.

This plugin attempts to send batches of events as a single request. However, if a request exceeds 20MB we will break it up into multiple batch requests. If a single document exceeds 20MB it will be sent as a single request.

This plugin uses the JVM to lookup DNS entries and is subject to the value of networkaddress.cache.ttl, a global setting for the JVM.

As an example, to set your DNS TTL to 1 second you would set the environment variable to .

Keep in mind that a connection with keepalive enabled will not reevaluate its DNS value while the keepalive is in effect.

This plugin supports request and response compression. Response compression is enabled by default and for Elasticsearch versions 5.0 and later, the user doesn’t have to set any configs in Elasticsearch for it to send back compressed response. For versions before 5.0, must be set to in Elasticsearch to take advantage of response compression when using this plugin

For requests compression, regardless of the Elasticsearch version, users have to enable setting in their Logstash config file.

Elasticsearch Output Configuration Optionsedit

This plugin supports the following configuration options plus the Common Options described later.

SettingInput typeRequired

string

No

string

No

a valid filesystem path

No

hash

No

boolean

No

string

No

string

No

array

No

string

No

uri

No

boolean

No

string, one of

No

string

No

string

No

string

No

string

No

a valid filesystem path

No

password

No

boolean

No

hash

No

string

No

password

No

string

No

string

No

number

No

number

No

uri

No

number

No

number

No

number

No

number

No

string

No

string

No

string

No

string, one of

No

string

No

boolean

No

boolean

No

number

No

string

No

boolean

No

boolean

No

a valid filesystem path

No

string

No

boolean

No

number

No

a valid filesystem path

No

password

No

string

No

string

No

number

No

string

No

string, one of

No

Also see Common Options for a list of options supported by all output plugins.

 

  • Value type is string
  • Default value is

Protocol agnostic (i.e. non-http, non-java specific) configs go here Protocol agnostic methods The Elasticsearch action to perform. Valid actions are:

  • index: indexes a document (an event from Logstash).
  • delete: deletes a document by id (An id is required for this action)
  • create: indexes a document, fails if a document by that id already exists in the index.
  • update: updates a document by id. Update has a special case where you can upsert — update a document if not already present. See the option. NOTE: This does not work and is not supported in Elasticsearch 1.x. Please upgrade to ES 2.x or greater to use this feature with Logstash!
  • A sprintf style string to change the action based on the content of the event. The value would use the foo field for the action

For more details on actions, check out the Elasticsearch bulk API documentation

  • Value type is string
  • There is no default value for this setting.

HTTP Path to perform the _bulk requests to this defaults to a concatenation of the path parameter and "_bulk"

  • Value type is path
  • There is no default value for this setting.

The .cer or .pem file to validate the server’s certificate

  • Value type is boolean
  • Default value is

Enable for update mode. Create a new document with source if doesn’t exist in Elasticsearch

  • Value type is string
  • There is no default value for this setting.

The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID.

  • Value type is string
  • There is no default value for this setting.
  • This option is deprecated

Note: This option is deprecated due to the removal of types in Elasticsearch 6.0. It will be removed in the next major version of Logstash. This sets the document type to write events to. Generally you should try to write only similar events to the same type. String expansion works here. If you don’t set a value for this option:

  • for elasticsearch clusters 7.x and above: the value of _doc will be used;
  • for elasticsearch clusters 6.x: the value of doc will be used;
  • for elasticsearch clusters 5.x and below: the event’s type field will be used, if the field is not present the value of doc will be used.

edit

  • Value type is array
  • Default value is

Set the Elasticsearch errors in the whitelist that you don’t want to log. A useful example is when you want to skip all 409 errors which are .

  • Value type is hash
  • There is no default value for this setting.

Pass a set of key value pairs as the headers sent in each request to an elasticsearch node. The headers will be used for any kind of request (_bulk request, template installation, health checks and sniffing). These custom headers will be overidden by settings like .

  • Value type is string
  • There is no default value for this setting.

HTTP Path where a HEAD request is sent when a backend is marked down the request is sent in the background to see if it has come back again before it is once again eligible to service requests. If you have custom firewall rules you may need to change this

  • Value type is uri
  • Default value is

Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the parameter. Remember the protocol uses the http address (eg. 9200, not 9300). (If using a proxy on a subpath) It is important to exclude dedicated master nodes from the list to prevent LS from sending bulk requests to the master nodes. So this parameter should only reference either data or client nodes in Elasticsearch.

Any special characters present in the URLs here MUST be URL escaped! This means should be put in as for instance.

  • Value type is boolean
  • Default value is

Enable gzip compression on requests. Note that response compression is on by default for Elasticsearch v5.0 and beyond

  • Value can be any of: , ,
  • Default value is

The default setting of will disable the Index Lifecycle Management feature.

Setting this flag to will enable Index Lifecycle Management feature, if the Elasticsearch cluster supports it. This is required to enable Index Lifecycle Management on a version of Elasticsearch earlier than version .

Setting this flag to will automatically enable the Index Lifecycle Management feature, if the Elasticsearch cluster is running Elasticsearch version or higher with the ILM feature enabled, and disable it otherwise.

This feature requires a Basic License or above to be installed on an Elasticsearch cluster version 6.6.0 or later

  • Value type is string
  • Default value is

Pattern used for generating indices managed by Index Lifecycle Management. The value specified in the pattern will be appended to the write alias, and incremented automatically when a new index is created by ILM.

Date Math can be used when specifying an ilm pattern, see Rollover API docs for details

Updating the pattern will require the index template to be rewritten

The pattern must finish with a dash and a number that will be automatically incremented when indices rollover.

  • Value type is string
  • Default value is

Modify this setting to use a custom Index Lifecycle Management policy, rather than the default. If this value is not set, the default policy will be automatically installed into Elasticsearch

If this setting is specified, the policy must already exist in Elasticsearch cluster.

  • Value type is string
  • Default value is

The rollover alias is the alias where indices managed using Index Lifecycle Management will be written to.

If both and are specified, takes precedence.

Updating the rollover alias will require the index template to be rewritten

  • Value type is string
  • Default value is

The index to write events to. This can be dynamic using the syntax. The default value will partition your indices by day so you can more easily delete old data or only search specific date ranges. Indexes may not contain uppercase characters. For weekly indexes ISO 8601 format is recommended, eg. logstash-%{+xxxx.ww}. LS uses Joda to format the index pattern from event timestamp. Joda formats are defined here.

  • Value type is path
  • There is no default value for this setting.

The keystore used to present a certificate to the server. It can be either .jks or .p12

  • Value type is password
  • There is no default value for this setting.

Set the keystore password

  • Value type is boolean
  • Default value is

From Logstash 1.3 onwards, a template is applied to Elasticsearch during Logstash’s startup if one with the name does not already exist. By default, the contents of this template is the default template for which always matches indices based on the pattern . Should you require support for other index names, or would like to change the mappings in the template in general, a custom template can be specified by setting to the path of a template file.

Setting to false disables this feature. If you require more control over template creation, (e.g. creating indices dynamically based on field names) you should set to false and use the REST API to apply your templates manually.

  • Value type is hash
  • There is no default value for this setting.

Pass a set of key value pairs as the URL query string. This query string is added to every host listed in the hosts configuration. If the hosts list contains urls that already have query strings, the one specified here will be appended.

  • Value type is string
  • Default value is

For child documents, ID of the associated parent. This can be dynamic using the syntax.

  • Value type is password
  • There is no default value for this setting.

Password to authenticate to a secure Elasticsearch cluster

  • Value type is string
  • There is no default value for this setting.

HTTP Path at which the Elasticsearch server lives. Use this if you must run Elasticsearch behind a proxy that remaps the root path for the Elasticsearch HTTP API lives. Note that if you use paths as components of URLs in the hosts field you may not also set this field. That will raise an error at startup

  • Value type is string
  • Default value is

Set which ingest pipeline you wish to execute for an event. You can also use event dependent configuration here like

  • Value type is number
  • Default value is

While the output tries to reuse connections efficiently we have a maximum. This sets the maximum number of open connections the output will create. Setting this too low may mean frequently closing / opening connections which is bad.

  • Value type is number
  • Default value is

While the output tries to reuse connections efficiently we have a maximum per endpoint. This sets the maximum number of open connections per endpoint the output will create. Setting this too low may mean frequently closing / opening connections which is bad.

  • Value type is uri
  • There is no default value for this setting.

Set the address of a forward HTTP proxy. This used to accept hashes as arguments but now only accepts arguments of the URI type to prevent leaking credentials.

  • Value type is number
  • Default value is

How frequently, in seconds, to wait between resurrection attempts. Resurrection is the process by which backend endpoints marked down are checked to see if they have come back to life

edit

  • Value type is number
  • Default value is

Set initial interval in seconds between bulk retries. Doubled on each retry up to

  • Value type is number
  • Default value is

Set max interval in seconds between bulk retries.

  • Value type is number
  • Default value is

The number of times Elasticsearch should internally retry an update/upserted document See the partial updates for more info

  • Value type is string
  • There is no default value for this setting.

A routing override to be applied to all processed events. This can be dynamic using the syntax.

  • Value type is string
  • Default value is

Set script name for scripted update mode

Example:

output { elasticsearch { script => "ctx._source.message = params.event.get('message')" } }
  • Value type is string
  • Default value is

Set the language of the used script. If not set, this defaults to painless in ES 5.0. When using indexed (stored) scripts on Elasticsearch 6 and higher, you must set this parameter to (empty string).

  • Value can be any of: , ,
  • Default value is

Define the type of script referenced by "script" variable inline : "script" contains inline script indexed : "script" contains the name of script directly indexed in elasticsearch file : "script" contains the name of script stored in elasticsearch’s config directory

  • Value type is string
  • Default value is

Set variable name passed to script (scripted update)

  • Value type is boolean
  • Default value is

if enabled, script is in charge of creating non-existent document (scripted update)

  • Value type is boolean
  • Default value is

This setting asks Elasticsearch for the list of all cluster nodes and adds them to the hosts list. For Elasticsearch 1.x and 2.x any nodes with (on by default) will be added to the hosts list, including master-only nodes! For Elasticsearch 5.x and 6.x any nodes with (on by default) will be added to the hosts list, excluding master-only nodes.

  • Value type is number
  • Default value is

How long to wait, in seconds, between sniffing attempts

  • Value type is string
  • There is no default value for this setting.

HTTP Path to be used for the sniffing requests the default value is computed by concatenating the path value and "_nodes/http" if sniffing_path is set it will be used as an absolute path do not use full URL here, only paths, e.g. "/sniff/_nodes/http"

  • Value type is boolean
  • There is no default value for this setting.

Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this unspecified will use whatever scheme is specified in the URLs listed in hosts. If no explicit protocol is specified plain HTTP will be used. If SSL is explicitly disabled here the plugin will refuse to start if an HTTPS URL is given in hosts

  • Value type is path
  • There is no default value for this setting.

You can set the path to your own template here, if you so desire. If not set, the included template will be used.

  • Value type is string
  • Default value is

This configuration option defines how the template is named inside Elasticsearch. Note that if you have used the template management features and subsequently change this, you will need to prune the old template manually, e.g.

where is whatever the former setting was.

  • Value type is boolean
  • Default value is

The template_overwrite option will always overwrite the indicated template in Elasticsearch with either the one indicated by template or the included one. This option is set to false by default. If you always want to stay up to date with the template provided by Logstash, this option could be very useful to you. Likewise, if you have your own template file managed by puppet, for example, and you wanted to be able to update it regularly, this option could help there as well.

Please note that if you are using your own customized version of the Logstash template (logstash), setting this to true will make Logstash to overwrite the "logstash" template (i.e. removing all customized settings)

  • Value type is number
  • Default value is

Set the timeout, in seconds, for network operations and requests sent Elasticsearch. If a timeout occurs, the request will be retried.

  • Value type is path
  • There is no default value for this setting.

The truststore to validate the server’s certificate. It can be either .jks or .p12. Use either or .

  • Value type is password
  • There is no default value for this setting.

Set the truststore password

  • Value type is string
  • Default value is

Set upsert content for update mode. Create a new document with this parameter as json string if doesn’t exists

  • Value type is string
  • There is no default value for this setting.

Username to authenticate to a secure Elasticsearch cluster

edit

  • Value type is number
  • Default value is

How long to wait before checking if the connection is stale before executing a request on a connection using keepalive. You may want to set this lower, if you get connection errors regularly Quoting the Apache commons docs (this client is based Apache Commmons): Defines period of inactivity in milliseconds after which persistent connections must be re-validated prior to being leased to the consumer. Non-positive value passed to this method disables connection validation. This check helps detect connections that have become stale (half-closed) while kept inactive in the pool. See these docs for more info

The following configuration options are supported by all output plugins:

  • Value type is boolean
  • Default value is

Disable or enable metric logging for this specific plugin instance. By default we record all the metrics we can, but you can disable metrics collection for a specific plugin.

  • Value type is string
  • There is no default value for this setting.

Add a unique to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type. For example, if you have 2 elasticsearch outputs. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.

output { elasticsearch { id => "my_plugin_id" } }
Sours: https://www.elastic.co/guide/en/logstash/6.8/plugins-outputs-elasticsearch.html

Now discussing:

When you specify Elasticsearch for the output, Journalbeat sends the transactions directly to Elasticsearch by using the Elasticsearch HTTP API.

To enable SSL, just add to all URLs defined under hosts.

If the Elasticsearch nodes are defined by , then add to the yaml file.

If you are indexing large amounts of time-series data, you might also want to configure Journalbeat to use index lifecycle management. For more information about configuring and using index lifecycle management with Journalbeat, see Set up index lifecycle management.

Configuration optionsedit

You can specify the following options in the section of the config file:

The enabled config is a boolean setting to enable or disable the output. If set to false, the output is disabled.

The default value is true.

The list of Elasticsearch nodes to connect to. The events are distributed to these nodes in round robin order. If one node becomes unreachable, the event is automatically sent to another node. Each Elasticsearch node can be defined as a or . For example: , or . If no port is specified, is used.

When a node is defined as an , the scheme and path are taken from the and config options.

output.elasticsearch: hosts: ["10.45.3.2:9220", "10.45.3.1:9230"] protocol: https path: /elasticsearch

In the previous example, the Elasticsearch nodes are available at and .

The gzip compression level. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression).

Increasing the compression level will reduce the network usage but will increase the cpu usage.

The default value is 0.

Configure escaping of HTML in strings. Set to to disable escaping.

The default value is .

The number of workers per configured host publishing events to Elasticsearch. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).

The default value is 1.

The basic authentication username for connecting to Elasticsearch.

The basic authentication password for connecting to Elasticsearch.

Dictionary of HTTP parameters to pass within the url with index operations.

The name of the protocol Elasticsearch is reachable on. The options are: or . The default is . However, if you specify a URL for , the value of is overridden by whatever scheme you specify in the URL.

An HTTP path prefix that is prepended to the HTTP API calls. This is useful for the cases where Elasticsearch listens behind an HTTP reverse proxy that exports the API under a custom prefix.

Custom HTTP headers to add to each request created by the Elasticsearch output. Example:

output.elasticsearch.headers: X-My-Header: Header contents

It is generally possible to specify multiple header values for the same header name by separating them with a comma.

The URL of the proxy to use when connecting to the Elasticsearch servers. The value may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed. If a value is not specified through the configuration file then proxy environment variables are used. See the Go documentation for more information about the environment variables.

The index name to write events to. The default is (for example, ). If you change this setting, you also need to configure the and options (see Load the Elasticsearch index template). If you are using the pre-built Kibana dashboards, you also need to set the option (see Load the Kibana dashboards).

You can set the index dynamically by using a format string to access any event field. For example, this configuration uses a custom field, , to set the index:

output.elasticsearch: hosts: ["http://localhost:9200"] index: "%{[fields.log_type]}-%{[beat.version]}-%{+yyyy.MM.dd}"

We recommend including in the name to avoid mapping issues when you upgrade.

With this configuration, all events with are sent to an index named , and all events with are sent to an index named .

To learn how to add custom fields to events, see the option.

See the setting for other ways to set the index dynamically.

An array of index selector rules. Each rule specifies the index to use for events that match the rule. During publishing, Journalbeat uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the setting is missing or no rule matches, the setting is used.

Rule settings:

The index format string to use. If this string contains field references, such as , the fields must exist, or the rule fails.
A dictionary that takes the value returned by and maps it to a new name.
The default string value to use if does not find a match.
A condition that must succeed in order to execute the current rule. All the conditions supported by processors are also supported here.

The following example sets the index based on whether the field contains the specified string:

output.elasticsearch: hosts: ["http://localhost:9200"] indices: - index: "warning-%{[beat.version]}-%{+yyyy.MM.dd}" when.contains: message: "WARN" - index: "error-%{[beat.version]}-%{+yyyy.MM.dd}" when.contains: message: "ERR"

This configuration results in indices named and (plus the default index if no matches are found).

The following example sets the index by taking the name returned by the format string and mapping it to a new name that’s used for the index:

output.elasticsearch: hosts: ["http://localhost:9200"] indices: - index: "%{[fields.log_type]}" mappings: critical: "sev1" normal: "sev2" default: "sev3"

This configuration results in indices named , , and .

The setting simplifies the configuration, but is limited to string values. You cannot specify format strings within the mapping pairs.

A format string value that specifies the ingest node pipeline to write events to.

output.elasticsearch: hosts: ["http://localhost:9200"] pipeline: my_pipeline_id

For more information, see Parse data by using ingest node.

You can set the ingest node pipeline dynamically by using a format string to access any event field. For example, this configuration uses a custom field, , to set the pipeline for each event:

output.elasticsearch: hosts: ["http://localhost:9200"] pipeline: "%{[fields.log_type]}_pipeline"

With this configuration, all events with are sent to a pipeline named , and all events with are sent to a pipeline named .

To learn how to add custom fields to events, see the option.

See the setting for other ways to set the ingest node pipeline dynamically.

An array of pipeline selector rules. Each rule specifies the ingest node pipeline to use for events that match the rule. During publishing, Journalbeat uses the first matching rule in the array. Rules can contain conditionals, format string-based fields, and name mappings. If the setting is missing or no rule matches, the setting is used.

Rule settings:

The pipeline format string to use. If this string contains field references, such as , the fields must exist, or the rule fails.
A dictionary that takes the value returned by and maps it to a new name.
The default string value to use if does not find a match.
A condition that must succeed in order to execute the current rule. All the conditions supported by processors are also supported here.

The following example sends events to a specific pipeline based on whether the field contains the specified string:

output.elasticsearch: hosts: ["http://localhost:9200"] pipelines: - pipeline: "warning_pipeline" when.contains: message: "WARN" - pipeline: "error_pipeline" when.contains: message: "ERR"

The following example sets the pipeline by taking the name returned by the format string and mapping it to a new name that’s used for the pipeline:

output.elasticsearch: hosts: ["http://localhost:9200"] pipelines: - pipeline: "%{[fields.log_type]}" mappings: critical: "sev1_pipeline" normal: "sev2_pipeline" default: "sev3_pipeline"

With this configuration, all events with are sent to , all events with are sent to a , and all other events are sent to .

For more information about ingest node pipelines, see Parse data by using ingest node.

The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped.

Set to a value less than 0 to retry until all events are published.

The default is 3.

The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 50.

Events can be collected into batches. Journalbeat will split batches larger than into multiple batches.

Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.

Setting to values less than or equal to 0 disables the splitting of batches. When splitting is disabled, the queue decides on the number of events to be contained in a batch.

The number of seconds to wait before trying to reconnect to Elasticsearch after a network error. After waiting seconds, Journalbeat tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to . After a successful connection, the backoff timer is reset. The default is 1s.

The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error. The default is 60s.

The http request timeout in seconds for the Elasticsearch request. The default is 90.

Configuration options for SSL parameters like the certificate authority to use for HTTPS-based connections. If the section is missing, the host CAs are used for HTTPS connections to Elasticsearch.

See Specify SSL settings for more information.

Sours: https://www.elastic.co/guide/en/beats/journalbeat/6.8/elasticsearch-output.html


615 616 617 618 619