Two MMLU Scores Reportedly Show Inconsistencies in Benchmark Comparability

The reported MMLU scores of 0.781 and 0.79 for two builds of the same model family indicate a small numerical difference. However, the underlying conditions under which these scores were obtained were not identical, which raises questions about the validity of directly comparing these scores.

The claim that both scores are associated with the same benchmark name does not account for the variances in implementation details, such as dataset splits, prompt formats, and grading methodologies. These factors can significantly affect the accuracy reported, making it crucial for engineers to understand the context of each score.

Engineers should be cautious when interpreting benchmark results. The discrepancies in the two MMLU scores suggest that simply labeling a benchmark does not provide assurance of performance comparability, and deeper analysis into each model's testing conditions is necessary.

Furthermore, the use of terminology like 'MMLU' as a benchmark label might create a false sense of security about the reliability of the reported scores. As the details reveal, the models' evaluation records are valid yet show different runner and grader setups, emphasizing the need for transparency in benchmarking practices.

Ultimately, this situation highlights the importance of scrutinizing not only the numbers but also the methodologies behind them when evaluating model performance. Engineers must ensure that they assess the full context of benchmark claims to make informed decisions regarding model adoption and usage.