Over the past several months, I observed a recurring issue affecting a number of Windows-based Logstash data collectors. While the collectors would initially perform as expected, ingestion throughput would gradually decline over time, eventually reaching a point where log processing rates were significantly below normal operating levels.
A simple restart of the Logstash service would immediately restore ingestion performance, suggesting that the underlying issue was accumulating during runtime rather than being related to external dependencies or data sources.
Initial Investigation
My first assumption was that the issue was related to memory management within the Java Virtual Machine (JVM) running Logstash.
To test this theory, I increased the heap allocation available to the affected collectors. Unfortunately, additional memory had little to no impact on the rate of degradation.
The next step was to upgrade Logstash to a newer release in case the issue had already been addressed by the Elastic development team. While upgrading is generally considered good practice, the behaviour persisted after the upgrade.
To gain further visibility, I enabled JVM Garbage Collection (GC) logging and submitted the logs to a GC analysis platform. The results indicated healthy memory management with no obvious signs of excessive garbage collection, memory pressure, or heap exhaustion.
Despite these findings, ingestion performance continued to decline over time.
Operational Considerations
The most straightforward solution would have been to schedule a periodic service restart using Windows Task Scheduler or another external orchestration mechanism.
However, there were several reasons why I wanted to avoid this approach:
- I did not have full administrative control of the underlying servers.
- Introducing additional scheduled tasks would create an external dependency.
- I preferred a self-contained solution managed entirely within Logstash itself.
- Reducing operational complexity would make future maintenance easier.
With those requirements in mind, I began looking for a method to periodically recycle Logstash without relying on external tooling.
The Solution
The solution was to leverage Logstash’s built-in configuration reload functionality.
By enabling automatic configuration reloads, Logstash can periodically monitor its configuration files and restart its pipelines whenever a change is detected.
The following settings were added to the logstash.yml file:
# Periodically check if the configuration has changed and reload the pipeline
config.reload.automatic: true
# How often to check if the pipeline configuration has changed
config.reload.interval: 86000s
In this example, Logstash checks for configuration changes approximately once per day.
Triggering the Reload
Enabling automatic reloads alone is not sufficient; a configuration change must actually occur for Logstash to reload its pipelines.
To achieve this, I created a lightweight exec input configuration that periodically updates a small configuration file within the Logstash configuration directory.
The process is straightforward:
- An
execinput runs on a defined schedule. - The command writes a simple text string to a file such as
test.conf. - The file modification timestamp changes.
- Logstash detects the configuration update.
- Logstash automatically reloads its pipelines.
Because I organise my Logstash configuration into separate module files, introducing a small “heartbeat” configuration file was simple and had no impact on production pipelines.
For testing purposes, I initially configured the update interval to occur every hour:
3600 seconds
In production, the interval can be aligned with the configured reload interval, for example:
86400 seconds (24 hours)
This effectively provides a controlled, periodic pipeline recycle without requiring service restarts, scheduled tasks, or third-party automation.
Results
Since implementing this approach, ingestion rates have remained consistent and I have not observed the gradual throughput degradation that previously affected these collectors.
While the underlying cause of the degradation remains unclear, the automatic pipeline reload mechanism has proven to be an effective and low-maintenance workaround.
Most importantly, the solution remains entirely self-contained within Logstash, avoiding the need for external scheduling mechanisms or additional operational dependencies.
For environments where periodic service restarts are undesirable or difficult to implement, leveraging Logstash’s native configuration reload functionality may provide a simple and elegant alternative.
About the Author
