PERFORMANCE Signal 486
Homebench – Benchmark local LLMs for speed, memory, and quality
homebench is a single-command TUI that benchmarks locally installed LLMs across tokens-per-second, time-to-first-token, resident memory, and a 31-task quality suite, auto-detecting Ollama, LM Studio, llama.cpp, vLLM, or any OpenAI-compatible server.
Engineers running models on their own hardware currently stitch together llama-bench for speed and lm-evaluation-harness for quality, with no shared view of how those numbers trade off on the same machine. homebench removes that glue by caching results, diffing successive runs, and exporting Markdown or JSON reports, so a hardware swap, quantization change, or model upgrade produces a comparable record. The optional LLM-as-judge and a throughput sweep at concurrencies of 1, 2, 4, and 8 make it usable for both laptop sizing and small-server capacity planning.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
It auto-discovers models from the major local runners and grades them on speed, memory, and a 31-task deterministic quality suite, with an optional LLM-as-judge for open-ended tasks.
Built-in caching plus `homebench diff` and run-labeling turn benchmarks from a one-off exercise into a tracked history you can compare after tuning, swapping a quant, or changing runners.
The `throughput` and `fit` subcommands extend it past single-prompt latency into concurrency scaling and hardware-fit recommendations, which is the missing piece in most local LLM tooling.
THE CLUSTER