AI Signal 231
smevals - a small eval suite for evaluating models, prompts, and harnesses
↗
For engineers comparing model capabilities or testing prompt and harness variations, smevals provides a structured, file-based workflow that decouples running evaluations from grading them. Its YAML-based eval definitions and support for custom grading scripts—including model-assisted checks—make it portable and adaptable to different assessment strategies.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Eval suites are defined as directories of YAML files containing tasks, and can be run against multiple model configurations (including different prompts, parameters, and agent harnesses) in a single command.
The tool separates the execution phase (runs) from the evaluation phase (grading), where graders apply a sequence of checks ranging from simple string matching to custom scripts that can invoke other models.
Results can be explored via a localhost web server or exported as static HTML reports, making it straightforward to share and compare evaluation outcomes across configurations.
THE CLUSTER