Writing suites & checks

Suite structure

{
  "name": "My suite",
  "cases": [
    { "name": "case-id", "prompt": "the prompt", "checks": [ /* one or more checks */ ] }
  ]
}

Each case runs against every model; each check contributes one pass/fail to that model's score.

Check types

Type	Fields	Passes when
contains	value, ignoreCase?	output includes the substring
notContains	value, ignoreCase?	output does not include it
regex	value, flags?	the pattern matches the output
equals	value, trim?	output equals the value (optionally trimmed)
jsonValid	—	output parses as JSON
minLength	value	output length ≥ value
maxLatencyMs	value	generation took ≤ value ms

Tips

• Keep prompts deterministic and constrained ("answer with one word", "only JSON") so checks are meaningful.
• Combine a correctness check with maxLatencyMs to balance quality and speed.
• Use notContains for refusal/format guards (e.g. no "I cannot").
• Group related cases into one suite per capability (reasoning, JSON, extraction…).

Programmatic checks

For anything beyond the built-ins, run the suite and inspect report.results[].output yourself — every raw output is included so you can add custom scoring.