Problem Statement

A common data architecture involves creating a new Elasticsearch index for each day's data, often with a date-based suffix. While this simplifies data lifecycle management, it can lead to performance issues when applications execute queries that span multiple days, causing significant load on the cluster. To mitigate this, we can enforce a policy that restricts queries to a single daily index. Requests for broader date ranges should be denied and require a formal approval process.

Solution: Path-Based Query Filtering

The INFINI Gateway can act as a proxy and enforcer for this policy. By configuring a request_path_filter within the gateway's flow, we can inspect the request path and allow or deny it based on a defined pattern. This filter will ensure that the request targets only one index name that conforms to a specific date format.

Configuration

The following configuration snippet demonstrates how to implement this restriction. It is placed within the gateway's flow definition, typically in the default_flow. The regular expression is designed to match a single index name (alphanumeric characters) followed by a date in the format YYYY-MM-DD, YYYY.MM.DD, or YYYYMMDD.

filters:
  - request_path_filter:
      message: "This request is not permitted. Single-index queries only are allowed. For multi-day queries, please submit a request."
      must:
        suffix:
          - _search
        regex:
          - \/[a-zA-Z0-9]+[-.]?\d{4}[-.]?\d{1,2}[-.]?\d{1,2}\/

Example Implementation

1. Creating Test Indices

First, let's establish a set of test indices with different valid date formats to validate our filter.

POST logs-2023-11-15/_doc
{
  "message": "This is a test log entry."
}

POST metrics.2023.11.15/_doc
{
  "value": 42
}

POST events20231115/_doc
{
  "event": "system startup"
}

2. Testing the Filter

We can now test the configuration using curl commands. Queries that target a single, correctly formatted index should succeed, while those targeting multiple indices or using wildcards should fail.

Expected to Succeed

# Single index with hyphenated date
curl http://localhost:8000/logs-2023-11-15/_search?pretty

# Single index with dot-separated date
curl http://localhost:8000/metrics.2023.11.15/_search?pretty

# Single index with concatenated date
curl http://localhost:8000/events20231115/_search?pretty

Expected to Fail

# Multiple indices specified
curl http://localhost:8000/logs-2023-11-15,logs-2023-11-16/_search?pretty

# Wildcard query
curl http://localhost:8000/logs-*/_search?pretty

# Root-level wildcard query
curl http://localhost:8000/*/_search?pretty

Handling Legitimate Multi-Day Queries

For scenarios where a legitimate business need requries querying data across multiple days, a exception process can be implemented using Elasticsearch aliases. An alias can be created to point to the specific indices needed for a particular task. It's crucial that the alias name itself also adheres to the naming convention to be accessible through the gateway.

Creating a Alias for a Date Range

The following command creates an alias named november-2023-logs that points to the three test indices created earlier.

POST /_aliases
{
  "actions": [
    {
      "add": {
        "indices": ["logs-2023-11-15", "metrics.2023.11.15", "events20231115"],
        "alias": "november-2023-logs"
      }
    }
  ]
}

After the business query is complete, the alias can be removed to clean up.

POST /_aliases
{
  "actions": [
    {
      "remove": {
        "indices": ["logs-2023-11-15", "metrics.2023.11.15", "events20231115"],
        "alias": "november-2023-logs"
      }
    }
  ]
}

By tightly controlling the creation and removal of these aliases, we can enforce our data access policy while still allowing for necessary exceptions.