Architecture

ollabench runs every case against every model, scores the outputs with declarative checks, and summarizes into a ranked report.

flowchart LR
  SU["suite.json<br/>(cases + checks)"] --> RUN["runSuite"]
  M["--models a,b"] --> RUN
  RUN -->|per model × case| CLI["OllamaModelClient.generate"]
  CLI --> OUT["output + latency"]
  OUT --> SC["scoreCheck<br/>(contains · regex · jsonValid · maxLatencyMs · …)"]
  SC --> REP["renderReport<br/>(ranked table + per-case matrix)"]

Module map

Module	Responsibility
suite.ts	Load/validate a suite JSON
scorers.ts	Check types and scoreCheck (pure)
model.ts	ModelClient interface + OllamaModelClient
run.ts	Run cases × models, aggregate per-model summaries
report.ts	Render the markdown comparison

Design principles

• Deterministic scoring — checks are pure functions, so results are reproducible and testable.
• Errors don't abort — a model failing a case is captured, not thrown; the run completes.
• Offline — talks only to your local Ollama server; no keys.
• Extensible — new check types and model backends slot in behind the same interfaces.