Dependence-aware aggregation improves LLM judge panel accuracy by modeling correlated outputs

LLM judge panels often produce inflated confidence when multiple judges agree, because agreement can stem from shared biases rather than independent evidence. Judges that share a prompt template, training lineage, or model family tend to make the same errors, making simple vote counts misleading. This limitation undermines the reliability of LLM-as-a-judge systems used to evaluate retrieval-augmented generation or other AI outputs.

The proposed dependence-aware aggregation treats the judge panel as a network, using an Ising model to capture both individual judge reliability and pairwise dependencies between judges. By learning these relationships from judge outputs alone, the method distinguishes correlated mistakes from genuine consensus, adjusting the aggregate score accordingly. This approach works in an unsupervised setting, requiring no human reference labels for training.

Experiments on three tasks, relevance classification, toxicity detection, and summarization assessment, show that the method improves accuracy by 9-14% over the best weighted-majority-vote baseline when using panels of ten judges. The gains demonstrate that accounting for judge correlation yields more trustworthy evaluations than relying solely on vote counts or historical accuracy weighting.

While the method offers clearer confidence estimates, it requires sufficient panel size and variability to learn dependency parameters reliably; highly homogeneous panels may provide limited information for the Ising model. Additionally, the computational overhead of learning pairwise dependencies grows with the number of judges, which could be a concern for very large panels. Practitioners should therefore assess panel diversity and consider the trade-off between modeling complexity and expected accuracy improvements.