Open-source tool checks for regressions in Claude coding-agent configurations across releases and edits

The tool addresses a specific pain point in AI-assisted development: configuration drift in coding agents. When teams rely on Claude.md files, custom skills, or hooks, changes from model updates, teammate edits, or new releases can silently alter behavior. This creates a testing gap where failures are only discovered when developers notice unexpected outcomes, often weeks later. The config-drift-checker automates this detection by running test cases against the actual agent setup in isolated environments.

Implementation focuses on practical integration with existing workflows. The tool generates starter test cases from current configurations, then runs them on every relevant trigger, new Claude Code versions, pull requests modifying setup files, or manual runs. Results include not just pass/fail status but detailed explanations from graders, tool call logs, and full responses. This granularity helps engineers quickly identify whether a failure stems from model changes, configuration errors, or specific skill implementations.

The ablation feature provides quantitative insight into configuration value. By running identical test cases with and without specific skills or hooks, teams can measure their actual impact. The demo shows how this revealed one guard hook as critical for preventing destructive commands while another conventions skill added no measurable value. This data helps teams prioritize maintenance efforts and avoid configuration bloat that increases failure surface area without clear benefits.

Technical implementation emphasizes security and self-hosting. The tool processes API keys locally and stores results in the user's repository, avoiding data exposure to third-party services. GitHub Actions integration provides familiar CI/CD patterns with Slack alerts and HTML reports for visibility. The FSL-1.1-Apache-2.0 license allows free use and modification while preventing commercial service competition, aligning with common open-source patterns for developer tools.

Current limitations center on scope and maintenance. The tool tests agent behavior rather than general model quality, meaning it won't catch broader performance changes unless they affect configured tasks. Generated test cases depend on the setup skill's ability to accurately model real-world scenarios. As Claude Code evolves, the tool itself may require updates to maintain compatibility with new features or evaluation formats, creating a potential maintenance burden for adopters.