Conformance tests

This directory holds data only. There is no runner here: each engine writes a small adapter that loads these files and asserts the expectations with its own test framework. The schemas in schema/ are the contract.

Tiers

Unit cases exercise one operator or transformation in isolation. ret is 1 when an operator matches, or when a transformation changed its input. input/output are JSON strings, so \u0000 and other escapes are legal. They are byte strings: every code point MUST be at most U+00FF and denotes exactly one byte, so adapters encode them with Latin-1 (never UTF-8) before handing them to the engine. The importer derives them from the SecRules Test Set the way libmodsecurity’s own runner does (test/unit/unit_test.cc json2bin): a literal \xHH or \uHHHH sequence in the corpus becomes one byte (the low 8 bits), every other character becomes the bytes of its UTF-8 encoding, and no other backslash sequence is decoded. Adapters MUST NOT unescape anything after JSON parsing. Cases imported from the corpus are converted on import.

Engine profiles load rules, then run each stage’s input as a transaction. Their data fields are ordinary UTF-8 text, unlike unit-tier strings (below). Header values in headers are field values as a server delivers them to the engine: without the optional whitespace around the value (RFC 9110), which adapters pass on verbatim. When a stage has a response, the adapter must feed it as the backend response so phases 3–5 run. output asserts on rule IDs and interruption only; log wording is never asserted beyond log_contains/no_log_contains substrings.

Interruptions

interruption is the engine’s disruptive result for the transaction: deny, drop or redirect with the rule id (0 when a body limit rather than a rule caused it) and status. allow is not an interruption in any engine; profiles observe it through triggered_rules/non_triggered_rules and assert no_interruption: true.

Log assertions

log_contains and no_log_contains are substring checks against the engine’s error log for the transaction (or its matched-rule messages, which is what the log is built from). Profiles use them only with a rule’s msg text (spec/10-logging.md#error-log, ADR-0021); never with the engine prefix, file, line or any other field.

What “triggered” means

A rule is triggered when its operator matched, or for a chain when every member matched, regardless of log, nolog, auditlog or noauditlog. Adapters MUST derive triggered_rules from the engine’s matched-rule list, not from log output, because several profiles assert on nolog rules.

Unit tier provenance

The files under unit/operators and unit/transformations are generated by tools/import_sts.py from the SecRules Test Set (the secrules-language-tests repository, vendored by libmodsecurity v3 under test/test-cases/). The converter replaces the corpus’s double escaping with plain JSON escapes, attaches spec anchors, and drops cases that depend on an external resource file. Do not edit generated files by hand: change the converter, or add cases in a *-extra.json file next to them. A case may carry re_groups, the full match followed by the capture groups an engine stores in TX:0..9 when capture is present.

Fields that link tests to the spec

• spec: NN-file.md#anchor points at the heading in spec/ the case verifies. The validator fails if the anchor does not exist or has no **Status:** line.
• requires: [NN-file.md#anchor, ...] (engine tier) lists the Extended features the profile needs, by spec anchor. An engine that does not implement one of them must skip the file, not fail it. The validator rejects an entry that is not an Extended feature. Unit cases have no requires; an engine skips a unit case whose spec anchor is an Extended feature it does not declare.
• files: {relative/path: content} (engine tier) ships auxiliary files. The adapter writes each one relative to a temporary directory, then loads rules with that directory as the configuration directory, so Include relative/path and @pmFromFile relative/path resolve.

Adding a case

1. Pick the tier and copy an existing file.
2. Set spec to the heading you are testing. If the heading does not exist yet, add it to spec/ with a status line first.
3. Run uv run python tools/validate.py from the repo root.