Ensure Docker and Docker Compose are installed on your system before proceeding.

Docker Compose Configuration

Below is the docker-compose.yaml file for the EFK stack:

version: '3.3'

services:

  elasticsearch:
    image: "docker.elastic.co/elasticsearch/elasticsearch:7.17.5"
    container_name: es-primary
    restart: always
    environment:
    - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    - "discovery.type=single-node"
    - "cluster.name=log-cluster"
    - "node.name=es-node"
    ulimits:
      memlock:
        soft: -1
        hard: -1 
    networks:
      log-network:
        ipv4_address: 172.28.110.10
        aliases:
        - es-primary
        - log-es
    ports:
    - "9200:9200"
    - "9300:9300"
    volumes:
    - /opt/efk/config/:/usr/share/elasticsearch/config
    - /opt/efk/es/data/:/usr/share/elasticsearch/data
  
  kibana:
    image: "docker.elastic.co/kibana/kibana:7.17.5"
    restart: always
    environment:
      ELASTICSEARCH_URL: http://172.28.110.10:9200
      ELASTICSEARCH_HOSTS: '["http://172.28.110.10:9200"]'
      I18N_LOCALE: en-US
    networks:
      log-network:
        ipv4_address: 172.28.110.20
        aliases:
          - kibana-ui
          - kb
    ports:
    - "5601:5601"
    depends_on:
    - "elasticsearch"
  
  filebeat:
    image: "docker.elastic.co/beats/filebeat:7.17.5"
    restart: always
    networks:
      log-network:
        ipv4_address: 172.28.110.30
        aliases:
          - filebeat-collector
          - fb
    user: root
    command: ["--strict.perms=false"]
    volumes:
    - /opt/efk/filebeat.yaml:/usr/share/filebeat/filebeat.yml
    - /var/lib/docker:/var/lib/docker:ro
    - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
    - "elasticsearch"
    - "kibana"
  
  es-head:
    image: "mobz/elasticsearch-head:5"
    container_name: es-head-ui
    restart: always
    networks:
      log-network:
        ipv4_address: 172.28.110.50
    ports:
    - "9100:9100"
    depends_on:
    - "elasticsearch"

networks:
  log-network:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.110.0/24

Execute docker-compose up -d to start the services.

Check container status with docker ps:

CONTAINER ID   IMAGE                                                  COMMAND                  CREATED          STATUS                          PORTS                                                                                  NAMES
a3b7c5d2e1f0   mobz/elasticsearch-head:5                              "grunt serve"            7 minutes ago    Up 7 minutes                    0.0.0.0:9100->9100/tcp, :::9100->9100/tcp                                              es-head-ui
b4c8d6e3f1a2   docker.elastic.co/beats/filebeat:7.17.5                "/usr/bin/tini -- /u…"   21 minutes ago   Up 21 minutes                                                                                                          log-filebeat_1
c5d9e0f1a2b3   docker.elastic.co/kibana/kibana:7.17.5                 "/bin/tini -- /usr/l…"   21 minutes ago   Up 21 minutes                   0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                              log-kibana_1
d6e0f1a2b3c4   docker.elastic.co/elasticsearch/elasticsearch:7.17.5   "/bin/tini -- /usr/l…"   21 minutes ago   Up 20 minutes                   0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9300->9300/tcp   es-primary

Elasticsearch Fundamentals

Elasticsearch serves as the database for log storage. In Elasticsearch, an Index is comparable to a database in relational systems. A Document represents a data unit in Elasticsearch, similar to a record or message.

ES-head is a plugin that connects to Elasticsearch and provides a visual interface for managing and querying data.

Elasticsearch Terminology

Document

A Document is the smallest unit of data stored in Elasticsearch (similar to a row in a database table). Each document has a unique ID that can be specified manually or auto-generated by Elasticsearch.

Index

An Index is a collection of Documents (similar to a table in a database).

Field

In Elasticsearch, a Document is a JSON object composed of multiple Fields, each with a specific data type.

Relationship Summary

An Index contains multiple Documents, each Document is a JSON object, and each JSON object consists of multiple Fields.

Operations and API Usage

Elasticsearch operations differ from traditional databases as they use RESTful API principles over HTTP. Resources are specified via URIs, and operations are defined using HTTP methods like GET, POST, PUT, and DELETE.

Accessing Elasticsearch

Two primary methods for accessing Elasticsearch:

• cURL commands for direct access
• Kibana for web-based interface

Kibana enables:

• Analysis and visualization of data
• Search and discovery of hidden insights
• Creation of charts, dashboards, and visualizations
• Adding search functionality to applications
• Log analysis and security vulnerability detection
• Management of Elastic Stack components

Managing Indices in Kibana

Creating an Index

# Create an index
PUT /logs_index

Viewing Indices

# List all indices
GET _cat/indices

# View specific index
GET system-logs-2024.07.01

# Create an index with response
PUT /app_logs
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "app_logs"
}

# View index details
GET /app_logs
{
  "app_logs": {
    "aliases": {},
    "mappings": {},
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "number_of_shards": "1",
        "provided_name": "app_logs",
        "creation_date": "1720143452731",
        "number_of_replicas": "1",
        "uuid": "FRaCW3lgSny9e11v8WSUmw",
        "version": {
          "created": "7170599"
        }
      }
    }
  }
}

Deleting an Index

# Delete an index
DELETE /app_logs
{
  "acknowledged": true
}

Managing Documents in Kibana

# Create an index
PUT /app_logs_01

# Create a document with specific ID
POST /app_logs_01/_doc/1
{
  "user_name": "john_doe",
  "user_age": 28,
  "salary": 85000
}

# Create a document with auto-generated ID
POST /app_logs_01/_doc
{
  "user_name": "jane_smith",
  "user_age": 32,
  "salary": 92000
}

Querying Documents

# Retrieve a specific document by ID
GET /app_logs_01/_doc/1
{
  "_index": "app_logs_01",
  "_type": "_doc",
  "_id": "1",
  "_version": 1,
  "_seq_no": 1,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "user_name": "john_doe",
    "user_age": 28,
    "salary": 85000
  }
}

# Search all documents in an index
GET /app_logs_01/_search

Filebeat Configuration

Filebeat can be installed using various methods including binary packages, RPM, or tar archives. The following example uses a tar package:

# Download and extract Filebeat
tar -zxvf filebeat-8.11.1-linux-x86_64.tar.gz
mv filebeat-8.11.1-linux-x86_64 ./filebeat
cd ./filebeat

# Configure filebeat.yml
vi filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/messages
  encoding: utf-8
  fields:
    log_type: "system"
  fields_under_root: true

output.elasticsearch:
  hosts: ["172.28.110.10:9200"]
  indices:
    - index: "system-logs-%{+yyyy.MM.dd}"
      when.contains:
        log_type: "system"

Start Filebeat with:

# Foreground execution
./filebeat -e -c filebeat.yml

# Background execution
nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 & disown

Filebeat Tips

• If you delete an index in Kibana and want to re-upload logs from Filebeat, delete the registry file at filebeat/data/registry/filebeat/log.. Do not delete meta. as this will cause errors on restart.
• When configuring multiple Filebeat clients, use consistent index names to consolidate data in a single index, or use different index names with Kibana index patterns for better organization.

Troubleshooting Common Issues

Issue 1: Configuration Volume Mounting

If Elasticsearch fails to start after adding a configuration volume mount:

1. Start the container without mounting the config directory
2. Copy the config directory from the container to the host:

# Create and start a temporary container
docker run -tid --name es-temp docker.elastic.co/elasticsearch/elasticsearch:7.17.5

# Copy configuration to host
docker cp es-temp:/usr/share/elasticsearch/config /opt/efk/

# Stop and remove the temporary container
docker stop es-temp
docker rm es-temp

Now update your docker-compose.yaml to include the volume mount and restart the service.

Issue 2: Elasticsearch-head Connection

If Elasticsearch-head starts successfully but cannot connect to Elasticsearch:

1. Edit the elasticsearch.yml file:

# /opt/efk/config/elasticsearch.yml
cluster.name: "log-cluster"
network.host: 0.0.0.0
http.cors.enabled: true   
http.cors.allow-origin: "*"

1. Restart the Elasticsearch container:

docker restart es-primary

The connection should now work successfully.

Kibana Usage Tips

1. Access Stack Management through the menu to view all indices. Elasticsearch indices function like database table names.
2. In Filebeat configuration, define index names to organize logs:

output.elasticsearch:
  hosts: ["172.28.110.10:9200"]
  indices:
    - index: "app-logs-%{+yyyy.MM.dd}"
      when.contains:
        log_type: "application"

Using consistent index names across multiple clients allows data consolidation in a single index.

For different index names, create Kibana index patterns to group similar indices together for easier searching.

In Kibana's Discover interface:

• Click Discover → Add Filter → Select index pattern (e.g., app-logs*)
• Use search functionality (e.g., message: "ERROR") to filter logs
• Click the + button on the right to select and display specific fields