Streamlining HTTP API Testing Assertions

When testing HTTP APIs with Loadgen, configuring assertions for response validation can become complex. While Loadgen provides comprehensive condition testing capabilities, writing detailed assertions in YAML format often results in verbose and difficult-to-reead configurations.

Consider this sample YAML configuration:

variables:
  - name: document_id
    type: sequential
runner:
  validate_errors: true
  validate_invalid_responses: true
requests:
  - request:
      method: PUT
      url: $[[env.API_SERVER]]/create_document_$[[document_id]]
    assert:
      equals:
        _ctx.response.body_json.success: true
    register:
      - collection_name: _ctx.response.body_json.collection
  - request:
      method: POST
      url: $[[env.API_SERVER]]/$[[collection_name]]/document
      body: '{"content": "test data"}'
    assert:
      equals:
        _ctx.response.body_json.result: created

As validation requirements grow more complex, the assertion section can become extensive and hard to maintain. For example, validating a comprehensive response structure:

{
  "processing_time": 17,
  "timeout_occurred": false,
  "results": {
    "total_count": {
      "value": 100,
      "comparison": "equal"
    },
    "maximum_score": 1.0,
    "matches": [...]
  },
  "statistics": {
    "average_value": {
      "value": 51.0
    },
    "count_value": {
      "value": 50
    },
    "maximum_value": {
      "value": 100
    },
    "minimum_value": {
      "value": 2
    },
    "sum_value": {
      "value": 2550
    }
  }
}

The corresponding YAML assertions become lengthy:

assert:
  and:
    - range:
        _ctx.response.body_json.processing_time:
          less_than: 50
    - equals:
        _ctx.response.status: 200
        _ctx.response.body_json.timeout_occurred: false
        _ctx.response.body_json.results.total_count.value: 100
        _ctx.response.body_json.maximum_score: 1.0
        _ctx.response.body_json.statistics.average_value.value: 51
        _ctx.response.body_json.statistics.count_value.value: 50
        _ctx.response.body_json.statistics.maximum_value.value: 100
        _ctx.response.body_json.statistics.minimum_value.value: 2
        _ctx.response.body_json.statistics.sum_value.value: 2550
    - regexp:
        _ctx.response.body_json.results.total_count.comparison: equal|greater|greater_equal

Entroducing Loadgen DSL for Cleaner Assertions

Loadgen DSL provides a more intuitive syntax for assertions that closely mirrors the response structure:

{
  processing_time: <50,
  timeout_occurred: false,
  results: {
    total_count: {
      value: 100,
      comparison: /equal|greater|greater_equal/,
    },
  },
  maximum_score: 1.0,
  statistics: {
    average_value.value: 51,
    count_value.value: 50,
    maximum_value.value: 100,
    minimum_value.value: 2,
    sum_value.value: 2550,
  },
}

The DSL is fully JSON-compatible, allowing direct use of response bodies as templates:

{
  "processing_time": <50,
  "timeout_occurred": false,
  "results": {
    "total_count": {
      "value": 100,
      "comparison": /equal|greater|greater_equal/
    },
    "maximum_score": 1.0
  },
  "statistics": {
    "average_value": {
      "value": 51.0
    },
    "count_value": {
      "value": 50
    },
    "maximum_value": {
      "value": 100
    },
    "minimum_value": {
      "value": 2
    },
    "sum_value": {
      "value": 2550
    }
  }
}

The DSL supports logical operators and functon calls for complex conditions:

{
  processing_time: >0 and <50,
  comparison: "equal" or "greater" or "greater_equal",
  results.total_count.value: not 0,
  article.title: starts_with("TECH"),
  article.tags: contains("Testing"),
  article.status: in(["published", "archived"]),
}

Complete Request Configuration with DSL

The DSL can also simplify entire request configurations:

PUT $[[env.API_SERVER]]/create_document_$[[document_id]]
# register: [{
#   collection_name: "_ctx.response.body_json.collection",
# }],
# assert: {
#   _ctx.response.body_json: {success: true},
# },

POST $[[env.API_SERVER]]/$[[collection_name]]/document
{"content": "test data"}
# {result: "created"}

Short form syntax provides concise validation:

POST $[[env.API_SERVER]]/$[[collection_name]]/document
{"content": "test data"}
# 200
# {result: "created"}

Global configuration options can be specified at the beginning:

# variables: [
#   {name: "document_id", type: "sequential"},
# ],
# runner: {
#   validate_errors: true,
#   validate_invalid_responses: true,
# },

PUT $[[env.API_SERVER]]/create_document_$[[document_id]]
# register: [{
#   collection_name: "_ctx.response.body_json.collection",
# }],
# assert: {
#   _ctx.response.body_json: {success: true},
# },

POST $[[env.API_SERVER]]/$[[collection_name]]/document
{"content": "test data"}
# {result: "created"}