Getting started

Requirements

• Node.js 20 or newer
• Ollama running, with the models you want to test pulled:

  ollama pull llama3.2
  ollama pull qwen2.5
  

Run a benchmark

npx ollabench examples/basics.suite.json --models llama3.2,qwen2.5
npx ollabench my.suite.json --models llama3.2,qwen2.5 --out report.md
npx ollabench my.suite.json --models llama3.2 --host http://localhost:11434

The report ranks models by pass rate (then latency) and includes a per-case matrix so you can see exactly where a model falls short.

Interpreting results

• Pass rate — fraction of all checks (across all cases) that passed.
• Avg latency — mean generation time per case; useful for picking a model for a Pi or a latency-sensitive feature.
• Per-case — passed/total per model, or ⚠️ if that model errored on the case.

Library usage

import { loadSuite, runSuite, renderReport, OllamaModelClient } from 'ollabench';

const report = await runSuite(loadSuite('my.suite.json'), ['llama3.2'], new OllamaModelClient());
console.log(renderReport(report));      // markdown
console.log(report.models[0].passRate); // programmatic access