FAQ & troubleshooting

Do I need internet?

No. ollabench talks to your local Ollama server; pull the models first (ollama pull llama3.2) and everything runs offline.

A model shows ⚠️ / 0 passed

That model errored on the case — usually it isn't pulled, or the Ollama host is wrong. Check ollama list and the --host value. Errors are captured per case, not thrown, so the rest of the run still completes.

Isn't checking contains/regex too simplistic for evals?

For deterministic tasks (facts, JSON, formats) it's exactly right and fast. For open-ended quality, use the checks as a filter and inspect report.results[].output for your own scoring, or add a model-graded check.

How do I make results comparable?

Constrain the prompts ("answer with one word", "only JSON") so checks are meaningful, and keep the same suite across models. Combine a correctness check with maxLatencyMs.

Can I run each case multiple times?

Not built in yet — a single pass per model. Repeat-runs and variance reporting are on the roadmap; PRs welcome.

What does "pass rate" count?

The fraction of all checks across all cases that passed for that model, not just whole cases — so partial credit shows up.