Why Redis or RabbitMQ For Logstash Persistent Queue is not needed?
Why Redis or RabbitMQ For Logstash Persistent Queue is not needed?
Redis or RabbitMQ was used for persistent queue for Logstash for logstash version 6 and below.
Persistent Queue was introduced as a beta feature in logstash 5.1, adding disk-based resiliency to Logstash pipelines. In Logstash 5.4 it was officially promoted to General Availability as a production-ready alternative to the in-memory queue.
Instead of deploying and managing a message broker, such as Redis, RabbitMQ, or Apache Kafka, to facilitate a buffered publish-subscriber model, we can enable persistent queues to buffer events on disk and remove the message broker.
Absorbs bursts of events without needing an external buffering mechanism like Redis or Apache Kafka.
By using internal persistent queue in logstash, we do not need to install different software ans this less the task of managing another different application.
References
https://www.elastic.co/blog/logstash-persistent-queue
https://www.elastic.co/guide/en/logstash/current/persistent-queues.html
Data Resiliency
As data flows through the event processing pipeline, Logstash may encounter situations that prevent it from delivering events to the configured output. For example, the data might contain unexpected data types, or Logstash might terminate abnormally.
To guard against data loss and ensure that events flow through the pipeline without interruption, Logstash provides the following data resiliency features.
Persistent Queues protect against data loss by storing events in an internal queue on disk.
Dead Letter Queues provide on-disk storage for events that Logstash is unable to process. You can easily reprocess events in the dead letter queue by using the dead_letter_queue input plugin.
These resiliency features are disabled by default. To turn on these features, you must explicitly enable them in the Logstash
Persistent Queues
By default, Logstash uses in-memory bounded queues between pipeline stages (inputs → pipeline workers) to buffer events. The size of these in-memory queues is fixed and not configurable. If Logstash experiences a temporary machine failure, the contents of the in-memory queue will be lost. Temporary machine failures are scenarios where Logstash or its host machine are terminated abnormally but are capable of being restarted.
In order to protect against data loss during abnormal termination, Logstash has a persistent queue feature which will store the message queue on disk. Persistent queues provide durability of data within Logstash.
Persistent queues are also useful for Logstash deployments that need large buffers. Instead of deploying and managing a message broker, such as Redis, RabbitMQ, or Apache Kafka, to facilitate a buffered publish-subscriber model, you can enable persistent queues to buffer events on disk and remove the message broker.
In summary, the benefits of enabling persistent queues are as follows:
Absorbs bursts of events without needing an external buffering mechanism like Redis or Apache Kafka.
Provides an at-least-once delivery guarantee against message loss during a normal shutdown as well as when Logstash is terminated abnormally.
If Logstash is restarted while events are in-flight, Logstash will attempt to deliver messages stored in the persistent queue until delivery succeeds at least once.
Redis or RabbitMQ was used for persistent queue for Logstash for logstash version 6 and below.
Persistent Queue was introduced as a beta feature in logstash 5.1, adding disk-based resiliency to Logstash pipelines. In Logstash 5.4 it was officially promoted to General Availability as a production-ready alternative to the in-memory queue.
Instead of deploying and managing a message broker, such as Redis, RabbitMQ, or Apache Kafka, to facilitate a buffered publish-subscriber model, we can enable persistent queues to buffer events on disk and remove the message broker.
Absorbs bursts of events without needing an external buffering mechanism like Redis or Apache Kafka.
By using internal persistent queue in logstash, we do not need to install different software ans this less the task of managing another different application.
References
https://www.elastic.co/blog/logstash-persistent-queue
https://www.elastic.co/guide/en/logstash/current/persistent-queues.html
Data Resiliency
As data flows through the event processing pipeline, Logstash may encounter situations that prevent it from delivering events to the configured output. For example, the data might contain unexpected data types, or Logstash might terminate abnormally.
To guard against data loss and ensure that events flow through the pipeline without interruption, Logstash provides the following data resiliency features.
Persistent Queues protect against data loss by storing events in an internal queue on disk.
Dead Letter Queues provide on-disk storage for events that Logstash is unable to process. You can easily reprocess events in the dead letter queue by using the dead_letter_queue input plugin.
These resiliency features are disabled by default. To turn on these features, you must explicitly enable them in the Logstash
Persistent Queues
By default, Logstash uses in-memory bounded queues between pipeline stages (inputs → pipeline workers) to buffer events. The size of these in-memory queues is fixed and not configurable. If Logstash experiences a temporary machine failure, the contents of the in-memory queue will be lost. Temporary machine failures are scenarios where Logstash or its host machine are terminated abnormally but are capable of being restarted.
In order to protect against data loss during abnormal termination, Logstash has a persistent queue feature which will store the message queue on disk. Persistent queues provide durability of data within Logstash.
Persistent queues are also useful for Logstash deployments that need large buffers. Instead of deploying and managing a message broker, such as Redis, RabbitMQ, or Apache Kafka, to facilitate a buffered publish-subscriber model, you can enable persistent queues to buffer events on disk and remove the message broker.
In summary, the benefits of enabling persistent queues are as follows:
Absorbs bursts of events without needing an external buffering mechanism like Redis or Apache Kafka.
Provides an at-least-once delivery guarantee against message loss during a normal shutdown as well as when Logstash is terminated abnormally.
If Logstash is restarted while events are in-flight, Logstash will attempt to deliver messages stored in the persistent queue until delivery succeeds at least once.