Docker Installation of Elasticsearch

For most production environments, Elasticsearch is pre-installed on servers. If you need to set it up yourself, Docker provides the easiest method.

Pull the Docker image:

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.5.2

Create and start the container:

docker run -d --name elasticsearch-node -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.5.2

The port mappings -p 9200:9200 -p 9300:9300 bind container ports to host ports, allowing external access.

Modify the configuration:

Access the container shel and navigate to the config directory:

docker exec -it container_id /bin/sh
cd config
vi elasticsearch.yml

elasticsearch.yml configuration:

cluster.name: "my-cluster"
network.host: 0.0.0.0
transport.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"

This configuration permits external access from any IP and enables CORS headers required by Kibana and Elasticsearch Head plugins.

After restarting the container, verify the installation by accessing http://your_ip:9200 in your browser.

Enable password authentication:

Add the following to elasticsearch.yml:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

Restart the container and execute the password setup:

docker restart container_id
docker exec -it container_id /bin/bash
./bin/elasticsearch-setup-passwords auto

The auto flag generates random passwords automatically.

Installing Logstash

Download Logstash from the official repository and extract it to your preferred location:

tar -zxvf logstash-7.0.0.tar.gz

Navigate to the config directory and create a custom pipeline configuration:

cd logstash-7.0.0/config
cp logstash-sample.conf my-pipeline.conf

Basic pipeline configuration:

input {
  file {
    path => "/usr/local/logstash/data/*.log"
    start_position => "beginning"
  }
}
output {
  elasticsearch {
    hosts => ["http://your_es_ip:9200"]
  }
}

This configuration collects all .log files from the specified directory and forwards them to Elasticsearch.

Production pipeline configuration:

input {
  file {
    path => "/var/opt/application/logs/service/*.log"
    type => "application"
    exclude => "*.gz"
    start_position => "beginning"
  }
  file {
    path => "/var/opt/application/logs/audit/audit.log"
    codec => multiline {
      pattern => "^\[%{WORD}"
      negate => true
      what => "previous"
    }
    type => "audit"
    start_position => "beginning"
  }
}
filter {
  if [type] == "application" {
    multiline {
      pattern => "^\[%{WORD}-%{WORD}:%{IP}:%{NUMBER}\]"
      negate => true
      what => "previous"
    }
    grok {
      match => {
        "message" => "\[(?<app_service>[a-z]+-[a-z]+):%{IP:client_ip}:%{NUMBER:port}]\s+(?<timestamp>\d*[./-]\d*[./-]\d* \d*:\d*:\d*[.,][0-9]+)\s+%{WORD:loglevel}\s+%{NUMBER:process_id}\s+(?<payload>[\w\W]*)"
      }
    }
  }
  if [type] == "audit" {
    grok {
      match => {
        "message" => "[a-z]+:\=+(?<action>[A-Za-z\u4e00-\u9fa5]+)[\=\s\n]+[\u4e00-\u9fa5]+:(?<component>[a-z\-]+)[\s\n\u4e00-\u9fa5]+:(?<datetime>[a-zA-Z\d\-\.:]+)(?<content>[\w\W]*)"
      }
    }
  }
  mutate {
    remove_field => ["message", "@version", "host"]
  }
}
output {
  if [type] == "application" {
    elasticsearch {
      hosts => ["http://your_es_ip:9200"]
      index => "app-logs-%{app_service}"
    }
  } else if [type] == "audit" {
    elasticsearch {
      hosts => ["http://your_es_ip:9200"]
      index => "audit-logs-%{component}"
    }
  }
  stdout { codec => rubydebug }
}

Adjust the path parameters to match your actual log file locations and update the Elasticsearch host address.

Install the multiline filter plugin:

cd logstash-7.0.0/bin
./logstash-plugin install logstash-filter-multiline

Start Logstash:

cd logstash-7.0.0/bin
nohup ./logstash -f ../config/my-pipeline.conf &

Verify the setup by creating a test .log file in your configured input directory. If the data appears in Elasticsearch, your pipeline is functioning correctly.

multiline codec parameters:

codec => multiline {
  charset => "UTF-8"
  max_bytes => "10MB"
  max_lines => 500
  pattern => "your_regex_pattern"
  negate => true
  what => "previous"
}

Installing Kibana for Log Visualization

Pull the Kibana image, ensuring version compatibility with Elasticsearch:

docker pull kibana:7.5.0

Create a local configuration file:

mkdir -p /etc/kibana/config
vim /etc/kibana/config/kibana.yml

kibana.yml configuration:

server.name: "kibana-dashboard"
server.host: "0"
elasticsearch.hosts: ["http://your_es_ip:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "your_password"
xpack.monitoring.ui.container.elasticsearch.enabled: true

Launch the container:

docker run --name kibana-dashboard \
  -v /etc/kibana/config:/usr/share/kibana/config \
  -p 8089:5601 \
  -d kibana:7.5.0

Access Kibana at http://your_ip:8089 and authenticate with your Elasticsearch credentials.

Configure index patterns:

Navigate to Management → Index Patterns → Create index pattern.

Enter app-logs-* or log-* depending on your Logstash index naming convention, then select the timestamp field for time-based filtering.

After creating the pattern, go to Discover to explore and search through your collected logs.