Why Centralized Log Management Matters

Modern host-based applications generate vast volumes of heterogeneous log data—spanning application logs, system events, access records, and error traces. Without a unified collection strategy, these logs remain siloed across filesystems, making root-cause analysis slow, correlation impossible, and compliance auditing fragile. A robust logging pipeline must therefore unify ingestion, enrich with context, enable precise querying, support real-time alerting, and ensure long-term retention—all while preserving performance and scalability. ### Deployment Workflow

1. Deploy the DataKit Agent

• Navigate to the observability platform’s console → Integrations → DataKit. - Copy the platform-provided installation command (e.g., curl -sL https://static.datakit.dev/install.sh | sh) and execute it on the target host. - After successful deployment, verify host registration under Infrastructure → Hosts. #### 2. Configure Single-Source Log Ingestion

• Go to /usr/local/datakit/conf.d/log/, duplicate logging.conf.sample, and rename it to logging.conf. - Edit the file to specify log paths and assign a unique source identifier (e.g., "app-auth-service"). Example configuration: ``` [[inputs.logging]] logfiles = ["/var/log/auth-service/*.log"] source = "auth-service" service = "auth" pipeline = "auth_pipeline.p"

- Restart DataKit to apply changes: ```
sudo datakit service -R

Note: DataKit only ingests new log lines appended after startup—equivalent to tail -f. Historical entries are not backfilled. #### 3. Scale to Multi-Source Log Collection

For multiple services or rotating log sets, create additional config files (e.g., payment-service.conf, gateway-access.conf) in the same directory. Each file defines its own logfiles, source, service, and optional pipeline. No naming convention is enforced—only .conf extension matters. #### Glob Pattern Support

DataKit supports POSIX-compliant glob syntax for flexible file matching: | Pattern | Meaning | Example Match | Example Non-Match | |---|---|---|---| | * | Zero or more characters | access.log.2024-03-* → access.log.2024-03-28 | error.log | | ? | Exactly one character | app?.log → app1.log, appX.log | app12.log | | [a-z] | One lowercase letter | log-[0-9].log → log-5.log | log-A.log | | ** | Recursive directory traversal | /var/log/**/app-*.log | N/A |

Querying Logs Efficiently

Full-Text Search

The log explorer supports: - Literal terms: timeout - Phrases (exact match): "connection refused" - Wildcards: err?r matches error, errxr - Boolean logic: status:500 AND (method:POST OR method:PUT)#### Structured JSON Field Search

When log messages contain valid JSON, use dot-notation field queries: - @level:ERROR - @request.headers.user-agent:"curl" - @response.body.code:401### Log Parsing with Pipelines

Custom parsing transforms raw log lines into structured fields usable for filtering, aggregation, and visualization. Below is an example pipeline script for Java application logs: ```

Input line:

2024-03-29 11:54:54,535 WARN http-nio-8081-exec-1 (org.jasig.cas.web.view.CasReloadableMessageBundle:76) - The code [screen.logout.security] cannot be found...

Pipeline (auth_pipeline.p):

grok(_, "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{NOTSPACE:thread} \(%{NOTSPACE:class}:%{NUMBER:line}\) - %{GREEDYDATA:message}") default_time(timestamp, "Asia/Shanghai") set_tag("env", "prod") set_tag("region", "cn-east-1")

This yields structured attributes: `level`, `thread`, `class`, `line`, `message`, `env`, `region`, enabling powerful slicing/dicing. ### Visualizing Insights

#### Dashboard Creation

From the **Scenarios** → **Dashboard** menu: - Select chart type (bar, pie, time-series). - Choose log data source. - Apply filters (e.g., `level == "WARN"`). - Group by field (e.g., `class` or `service`). - Save as reusable template. #### Cross-Dimensional Correlation

- **Click-through linking:** Embed hyperlinks in charts that navigate directly to filtered log views (e.g., clicking a spike in error count opens logs matching `level:ERROR` in that time window). - **Built-in view binding:** Attach metric dashboards (CPU, memory), trace heatmaps, or infrastructure topology maps directly to log detail panels—enabling side-by-side contextual analysis. ### Proactive Alerting

Create log-based monitors under **Monitoring** → **New Monitor** → **Log Detector**: - Define detection logic: e.g., `count(level == "FATAL") > 5 in last 5m` - Configure notification channels (email, Slack, webhook) - Set severity, auto-resolution rules, and suppression windows Multiple detector types are supported: threshold, anomaly (statistical deviation), pattern (regex), and rate-of-change. ### Long-Term Retention & Export

Log forwarding enables archival to durable storage: - Internal object store (default, encrypted, 180-day minimum TTL) - External endpoints: AWS S3, Alibaba Cloud OSS, Huawei Cloud OBS, Kafka topics To configure: - Go to **Logs** → **Data Forwarding** → **New Rule** - Specify filter expression (e.g., `service == "payment" AND level IN ["ERROR","FATAL"]`) - Choose destination and retention policy Backed-up logs remain queryable via time-range selection down to hourly granularity. ### Operational Readiness

By integrating ingestion, parsing, search, visualization, alerting, and archiving into a single workflow, teams eliminate manual log hunting, accelerate incident resolution, strengthen audit trails, and unlock behavioral analytics across distributed host environments.