EUREKA: Evaluating and Understanding Large Foundation Models

Eureka is a reusable and open evaluation framework for standardizing evaluations of large foundation models beyond single-score reporting and rankings. Eureka-Bench is an extensible collection of benchmarks testing capabilities that (i) are still challenging for state-of-the-art foundation models and (ii) represent fundamental but overlooked capabilities for completing tasks in both language and vision modalities. Using the framework and Eureka-Bench, an analysis of 12 state-of-the-art models is conducted, providing in-depth insights for failure understanding and model comparison by disaggregating the measurements across important subcategories of data.

Because these benchmarks are not saturated, they enable the discovery of meaningful differences between models at the capability level. These insights help uncover granular weaknesses for specific capabilities and can inform more targeted improvements in future models. Eureka is released as open-source to foster transparent and reproducible evaluation practices for large foundation models.