Log management platforms have become essential for modern operations. They enable real-time, centraliezd log viewing, helping teams extract value from log data, improve operational efficiency, and enhance service management.

Architecture Overview

The solution consists of three main components:

• Filebeat: A lightweight log shipper that collects data from various sources
• Easysearch: A distributed search engine responsible for data ingestion, analysis, and storage
• Console: A web-based visualization tool for querying and reportign

This guide walks through building a unified log management platform by collecting system logs with Filebeat, storing them in Easysearch, and visualizing the data through Console.

Implementation Steps

1. Prerequisites

Deploy Easysearch

Edit the easysearch.yml configuration file and enable API compatibility mode:

elasticsearch.api_compatibility: true

Deploy Console

Set up the Console instance according to you're network requirements.

2. Install and Configure Filebeat

Download Filebeat version 7.10.2 from the official releases page.

Modify the filebeat.yml configuration:

setup.template.name: "filebeat"
setup.template.pattern: "system-log*"
setup.template.fields: "${path.config}/fields.yml"

output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: "https"
  ssl.verification_mode: none
  username: "admin"
  password: "admin_password_here"
  index: "system-log"

3. Enable System Module and Load Pipelines

./filebeat modules enable system
./filebeat setup --pipelines --modules system

4. Create Index Template and Initial Index

Apply ZSTD compression with source reuse to optimize disk usage:

PUT _template/system_log
{
  "order": 100,
  "index_patterns": [
    "system_log*"
  ],
  "settings": {
    "index": {
      "format": "7",
      "lifecycle": {
        "name": "ilm_.infini_metrics-30days-retention",
        "rollover_alias": "system_log"
      },
      "codec": "ZSTD",
      "source_reuse": true,
      "number_of_shards": "1",
      "translog": {
        "durability": "async"
      }
    }
  },
  "mappings": {
    "dynamic_templates": [
      {
        "strings": {
          "mapping": {
            "ignore_above": 256,
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ]
  }
}

PUT system-log-00001
{
  "aliases": {
    "system-log": {
      "is_write_index": true
    }
  }
}

5. Start Filebeat

nohup ./filebeat -c filebeat.yml 2>&1 > /dev/null &

6. Query and Explore Logs

Open Console and navigate to the search interface to view and query collected logs.

7. Build Dashboards

Create custom dashboards in Console to visualize log patterns, filter by severity levels, and generate reports for operational analysis.