Building a Scalable Data Pipeline with Zookeeper, Kafka, and the ELK Stack
Setting Up a Zookeeper Ensemble
A Zookeeper cluster maintains coordination metadata for distributed systems. For production, deploy an odd number of nodes (minimum three) to achieve quorum.
Configuration Essentials
On each node, prepare the configuration file conf/zoo.cfg:
tickTime=2000
dataDir=/var/lib/zookeeper
dataLogDir=/var/log/zookeeper
c ...
Posted on Mon, 22 Jun 2026 17:18:48 +0000 by bios
Building a Scalable Log Processing Pipeline with Filebeat, Kafka, Logstash, and Elasticsearch
Distributed log processing systems are essential for modern application monitoring and analysis. A common approach involves using Filebeat for log collection, Kafka as a message buffer, Logstash for transformation, Elasticsearch for storage, and Kibana for visualization. Grafana can also integrate with Elasticsearch for real-time monitoring das ...
Posted on Fri, 15 May 2026 04:29:59 +0000 by wgh
Practical Logstash Usage Patterns for Timezone Handling, Log Parsing, and Multi-index Routing in ELK
Aligning Timestamps Across Time Zones
When shipping data from Logstash to Elasticsearch, the @timestamp field often reflects UTC time, causing mismatch with local time zones. A permanent fix involves adjusting the timestamp in the filter stage.
ruby {
code => "evt.set('local_ts', evt.get('@timestamp').time.localtime + 28800)"
}
r ...
Posted on Wed, 13 May 2026 04:44:46 +0000 by Delcypher