27B-parameter AI agent reportedly outperforms Claude Opus 4.8 and GPT-5.5 in research replication tasks

Faraday introduces a 27B-parameter AI agent trained via long-horizon reinforcement learning to replicate research figures from papers. The agent operates without access to the original plots, requiring it to infer methods and results from text alone. This approach mimics the manual replication process used by scientists like Michael Faraday, who historically validated findings through hands-on experimentation. The system’s performance is benchmarked against Claude Opus 4.8 and GPT-5.5, with Faraday reportedly achieving higher fidelity in replications across 310 tasks spanning machine learning and AI-for-science domains.

The agent’s training relies on Replica, a task suite designed to simulate the underspecified nature of real-world research. Tasks are constrained by time and compute budgets, forcing Faraday to optimize for efficiency. While it excels in domains like structural biology and materials science, its advantage is less pronounced in others, indicating uneven generalization. The use of an LLM-based judge to evaluate replication quality introduces noise, which the team mitigates with per-task rubrics and multi-sample aggregation. However, the stochasticity of the underlying model remains a limitation, particularly for long-horizon tasks.

Faraday’s ability to direct larger models like GPT-5.5 Codex as tools suggests a scalable approach to scientific oversight. The agent adapts to more capable models at test-time, implying potential for future integration with advancing coding agents. Unlike prior AI scientist agents, Faraday does not rely on hand-coded evolutionary harnesses or test-time rewards, instead learning to value discoveries intrinsically. This could reduce the need for manual intervention in research automation, though the lack of explicit reward signals may complicate debugging and error analysis.

The system’s reliance on large-scale models and reinforcement learning introduces practical challenges. Training instability and noisy reward signals could hinder adoption in environments with limited compute resources. Additionally, Faraday’s performance in replicating recent research, where the base model lacks pre-training exposure, highlights its ability to generalize but also underscores the risk of overfitting to familiar domains. The absence of a clear failure mode for novel or highly ambiguous tasks may limit its utility in exploratory research.

Faraday’s development marks a step toward AI-driven scientific innovation, but its current focus on replication rather than discovery leaves open questions about its broader applicability. The team’s emphasis on “research taste” as a metric for success suggests a shift toward qualitative evaluation, though this remains difficult to quantify. For engineers, the agent’s potential to automate validation tasks could reduce manual labor, but its dependency on proprietary models and unclear scalability may restrict its use to well-funded labs.