Recipes

A JSON-extraction suite

{
  "name": "Extraction",
  "cases": [
    {
      "name": "invoice-json",
      "prompt": "Extract {\"total\": number} from: Total due: 42.50 EUR. Only JSON.",
      "checks": [{ "type": "jsonValid" }, { "type": "regex", "value": "\"total\"\\s*:\\s*42\\.5" }]
    }
  ]
}

Compare models and save a report

ollama pull llama3.2 && ollama pull qwen2.5
npx ollabench extraction.suite.json --models llama3.2,qwen2.5 --out report.md

Fail CI if quality drops

import { loadSuite, runSuite, OllamaModelClient } from 'ollabench';

const report = await runSuite(loadSuite('smoke.suite.json'), ['llama3.2'], new OllamaModelClient());
const model = report.models[0];
if (model.passRate < 0.9) {
  console.error(`Pass rate ${(model.passRate * 100).toFixed(0)}% below threshold`);
  process.exit(1);
}

Latency-focused suite (for a Pi)

{
  "name": "Latency",
  "cases": [
    { "name": "short-answer", "prompt": "Capital of France? One word.",
      "checks": [{ "type": "contains", "value": "Paris" }, { "type": "maxLatencyMs", "value": 4000 }] }
  ]
}

Inspect raw outputs for custom scoring

const report = await runSuite(suite, ['llama3.2'], new OllamaModelClient());
for (const r of report.results) myScorer(r.output); // r.output is the raw text