Overview of the Elastic Stack

Effective observability relies on a cohesive data pipeline. The Elastic ecosystem comprises several components working together. ElasticSearch serves as the distributed search and analytics engine capable of storing and indexing vast amounts of data. Kibana provides a browser-based interface for visualizing stored metrics and logs.

Logstash acts as the central processing unit within this stack, handling data ingestion, transformation, and forwarding. It supports numerous protocols to capture logs from diverse sources. Additionally, lightweight agents like Filebeat are often deployed on source hosts to ship logs directly to Logstash, optimizing resource usage compared to heavier installation methods.

Architecture Components

The Logstash workflow follows a standardized pipeline model consisting of three distinct stages:

1. Inputs: These plugins determine how data enters the system. Common sources include local file monitoring, network streams (TCP/UDP), or web services (HTTP). Inputs define the source path and polling behavior.
2. Filters: Before reaching storage, raw events undergo processing here. Filters can parse complex log lines into structured fields using pattern matching (e.g., Grok), enrich data with geo-IP information, or strip sensitive details.
3. Outputs: This stage defines where processed data resides. The most common destination is an ElasticSearch cluster, but outputs also support files, message queues, or cloud storage systems.

Installation Prerequisites

Since Logstash operates on the Java Virtual Machine (JVM), a compatible JDK installation is mandatory.

Ensure no conflicting OpenJDK versions remain in the system environment before proceeding with the specific installation package.

Deployment Steps

Setting up the Environment

First, download the distribution tarball from a official mirror or repository.

tar -xvf logstash-7.9.0-linux-x86_64.tar.gz

Extract the archive and move it to the desired directory structure.

mv logstash-7.9.0 /opt/services/
cd /opt/services/logstash-7.9.0

Configuring the Pipeline

Pipeline definitions reside in .conf files located typicallly under /etc/logstash/conf.d/. A basic configuration must explicitly declare the input, filter, and output sections.

Input Configuration

To monitor specific application logs, use the file input plugin. Define the target path using glob patterns if multiple files are involved.

input {
  file {
    # Path to the target log directory
    path => "/var/log/myapp/access*.log"
    
    # Start reading from the end of the file by default
    start_position => "beginning"
    
    # Optional: Track file state to avoid reprocessing on restart
    sincedb_path => "/dev/null"
  }
}

Note: Setting sincedb_path to /dev/null ensures every run reads from the beginning, useful for testing pipelines repeatedly.

Filter Configuration

Parsing unstructured text into usable fields is critical. The Grok filter translates raw strings into named fields based on predefined patterns.

Sample Apache NCSA Log Entry:

192.168.1.5 - john [10/Oct/2023:13:55:36 +0000] "GET /index.html HTTP/1.1" 200 2326

Configuration Example:

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  
  mutate {
    # Rename fields for clarity in downstream dashboards
    rename => { "sourcehost" => "server_node" }
  }
}

If no parsing is required, the filter block may be omitted entirely.

Output Configuration

Directing data to ElasticSearch requires the node address and an index naming convention. Dynamic index names based on timestamps help organize data daily.

output {
  elasticsearch {
    hosts => ["http://es-node-01.local:9200"]
    index => "app-logs-%{+YYYY.MM.dd}"
  }

  # Debugging output to console
  stdout {
    codec => rubydebug
  }
}

Execution and Verification

Once the configuration file (e.g., pipeline.conf) is ready, launch Logstash pointing to that file.

./bin/logstash -f config/pipeline.conf

For production environments, background execution prevents session termination issues.

nohup ./bin/logstash -f config/pipeline.conf --config.reload.automatic > /var/log/logstash/startup.log 2>&1 &

Using --config.reload.automatic enables hot-reloading; changes to the .conf file apply without restarting the process.

Verify that data flows correctly by checking the Elasticsearch index created by the configuration:

curl http://es-node-01.local:9200/_cat/indices?v

Visualization in Kibana

After confirming data ingestion via shell commands, configure the visualization layer.

1. Access Kibana UI and navigate to Stack Management > Index Patterns.
2. Create a new pattern matching the dynamic index name app-logs-*.
3. Select Discover to query the newly indexed documents and verify field visibility.
4. Use Dashboard to build visual representations such as line charts showing error rates over time.