Ensuring Data Privacy in Search Environments

Organizations frequently store sensitive information within their search clusters, such as personal identifiers, financial data, or confidential business details. Permitting unrestricted access to this data, even for troubleshooting or internal analysis, poses significant compliance and security risks. Addressing this challenge requires robust mechanisms to obscure sensitive fields when accessed by unauthorized users, while still allowing operational activities.

Easysearch provides built-in field masking capabilities, enabling administrators to define granular access controls that automatically transform sensitive data into a masked representation based on user roles. This article explores how to configure these protections using both hashing and regular expression-based masking techniques.

Setting Up Sample Sensitive Records

To demonstrate field masking, let's index a few documents into an example index, confidential_records. For this illustration, we will consider the fields sensitive_details and origin_country as containing private information that should not be exposed directly to all users.

POST confidential_records/_doc/1
{
  "record_id": "ABC-001",
  "project_name": "Internal Project Alpha",
  "sensitive_details": "This is highly confidential information.",
  "timestamp": "2023-10-26T10:00:00Z"
}

POST confidential_records/_doc/2
{
  "record_id": "XYZ-002",
  "project_name": "Customer Data Beta",
  "sensitive_details": "Customer's private contact details.",
  "origin_country": "Confidential Country X",
  "timestamp": "2023-10-26T10:05:00Z"
}

Hash-Based Field Masking Configuration

Easysearch's security module allows for one-way hashing of field content. By default, it employs the BLAKE2b algorithm, though other JVM-supported hash algorithms like MD5 or SHA-256 can be specified. This method is ideal when the original data needs to be irreversibly obscured, but a consistent, hashed representation is acceptable (e.g., for analytics that don't require the original value).

Creating a Role and User for Hash Masking

We'll create a security role, hashed_data_viewer, granting read access to the confidential_records index. For this role, the sensitive_details and origin_country fields will be automatically masked using the default BLAKE2b hash algorithm when accessed by users assigned this role.

PUT _security/role/hashed_data_viewer
{
  "cluster": [],
  "indices": [
    {
      "names": ["confidential_records"],
      "field_mask": ["sensitive_details", "origin_country"],
      "privileges": ["read"]
    }
  ]
}

PUT _security/user/restricted_hash_user
{
  "password": "strong_password",
  "roles": ["hashed_data_viewer"]
}

Verifying Hash Masking

When restricted_hash_user queries the confidential_records index, the content of the specified sensitive fields will appear as a hash value. For insatnce, a search query would return results where sensitive_details and origin_country are replaced by a string of characters representing their BLAKE2b hash.

A user with full privileges, however, would still see the original, unmasked content. This ensures that only authorized personnel can access the raw data.

Regular Expression-Based Field Masking

For scenarios where a partial view of the sensitive data is permissible, or where specific patterns need to be replaced, regular expression masking offers greater flexibility. This method allows you to define one or more regex patterns to match and replace parts of a field's content.

Creating a Role and User for Regex Masking

Let's define a new role, regex_data_viewer. This role will also have read access to confidential_records. However, the masking for sensitive_details and origin_country will use a regular expression to show only the first character, replacing the rest with [REDACTED].

PUT _security/role/regex_data_viewer
{
  "cluster": [],
  "indices": [
    {
      "names": ["confidential_records"],
      "field_mask": [
        "sensitive_details::/(.)(.*)/::$1[REDACTED]",
        "origin_country::/(.)(.*)/::$1[REDACTED]"
      ],
      "privileges": ["read"]
    }
  ]
}

PUT _security/user/restricted_regex_user
{
  "password": "another_strong_password",
  "roles": ["regex_data_viewer"]
}

Verifying Regex Masking

When restricted_regex_user performs a search, the sensitive_details and origin_country fields will display they first character followed by [REDACTED]. For example, "This is highly confidential information." would appear as "T[REDACTED]", and "Confidential Country X" would become "C[REDACTED]".

Testing Masking with User Impersonation

In a production environment, directly logging in as a restricted user to verify masking might be impractical due to credential management. Easysearch's "run-as" feature allows an administrative user to execute requests as if they were another user, facilitating testing and auditing of access controls without sharing sensitive credentials. An administrator could, for example, use the X-Security-Run-As header (or similar mechanism depending on the client) to test the masked views.

Integration with External Traffic Proxies

When using an external proxy or gateway, such as Infini Gateway, for traffic analysis or routing, it's crucial to understand how data masking interacts with these components. Because Easysearch performs the field masking internally before sending results, any proxy positioned in front of the Easysearch cluster will only receive and log the already masked data. This ensures that even network traffic inspection tools do not inadvertently expose sensitive information, maintaining the integrity of the data protection strategy.

For example, querying via a gateway (assuming it's configured on port 8000 and proxies to Easysearch) as the restricted_regex_user would show masked data in the proxy's logs:

curl -ku restricted_regex_user:another_strong_password http://localhost:8000/confidential_records/_search?pretty

The logs captured by the gateway for this request would contain the masked results, confirming that the data remains protected throughout the data flow.