These tools and metrics are designed to help AI actors develop and use trustworthy AI systems and applications that respect human rights and are fair, transparent, explainable, robust, secure and safe.
MLE-bench
MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, OpenAI curated 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments.
OpenAI established human baselines for each competition using Kaggle's publicly available leaderboards. Using open-source agent scaffolds to evaluate several frontier language models on OpenAI's benchmark, finding that the best-performing setup - OpenAI's o1-preview with AIDE scaffolding - achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. Various forms of resource scaling for AI agents and the impact of contamination from pre-training were investigated. The benchmark code is open-sourced to facilitate future research in understanding the ML engineering capabilities of AI agents.
About the tool
You can click on the links to see the associated tools
Developing organisation(s):
Objective(s):
Purpose(s):
Country of origin:
Type of approach:
Usage rights:
Target groups:
Target users:
Tags:
- evaluation
- machine learning testing
- ai agent
- red-teaming
- benchmarking
Use Cases
Would you like to submit a use case for this tool?
If you have used this tool, we would love to know more about your experience.
Add use case