MLAgentBench · official benchmark · 13 tasks

What does each agent start with?

A field guide to every starter model, empty scaffold, dummy prediction, and deliberately slow implementation in the official benchmark.

Starter atlas

Thirteen starting points

Showing 13 of 13

01 Implemented

Canonical

CIFAR-10

A compact LeNet-style convolutional network.

Two convolution and max-pooling blocks feed three fully connected layers. It trains for five epochs with SGD, momentum, and cross-entropy.

Image classificationPyTorch

Open starter code

02 Skeleton

Canonical

IMDb

A blank fine-tuning scaffold for DistilBERT.

The data loader and submission shape are present, but preprocessing, model definition, and training are left to the agent. The reference score is random 50% accuracy.

Text classificationHugging Face

Open starter code

03 Implemented

Canonical

ogbn-arxiv

A two-layer node-feature MLP that ignores graph edges.

The 16-unit hidden layer uses batch normalization and no dropout. GraphSAGE is imported and shown as a commented alternative, but it is not the starter model.

Graph classificationPyTorch Geometric

Open starter code

04 Skeleton

Classic Kaggle

House Prices

Prepared numeric features with the regressor left blank.

The starter selects 25 numeric housing features and creates a train-validation split. RandomForestRegressor is imported but never instantiated.

Tabular regressionscikit-learn

Open starter code

05 Skeleton

Classic Kaggle

Spaceship Titanic

One-hot features with an empty classifier section.

The cabin field is split and six categorical variables are encoded. LogisticRegression is imported as a likely starting point, but no model is trained.

Tabular classificationscikit-learn

Open starter code

06 Implemented

Kaggle Challenge

Parkinson’s Disease

Four random forests driven only by visit month.

One multi-output regressor is trained per UPDRS target to forecast measurements at 0, 6, 12, and 24 months, without protein or peptide features.

Time-series regressionscikit-learn

Open starter code

07 Skeleton

Kaggle Challenge

FathomNet

A nominal four-block CNN treated as non-runnable.

The file sketches four convolutional blocks and a 290-class sigmoid head with binary cross-entropy. The paper nevertheless classifies it among starters that do not run by themselves.

Multilabel imagesKeras

Open starter code

08 Dummy

Kaggle Challenge

Feedback Prize

Uniform random predictions for six writing dimensions.

The training function returns None and the prediction function samples random values for cohesion, syntax, vocabulary, phraseology, grammar, and conventions.

Text regressionNumPy scaffold

Open starter code

09 Implemented

Kaggle Challenge

Identify Contrails

A single 1×1 convolution for per-pixel labels.

Three false-colour satellite channels map directly to background and contrail logits. Training lasts one epoch with weighted cross-entropy.

Image segmentationPyTorch

Open starter code

10 Implemented

Recent Research

CLRS

A gated triplet message-passing graph network.

The encode-process-decode baseline targets Floyd–Warshall with 128 hidden units, learned triplet features, and encoded and decoded algorithmic hints.

Algorithmic graphsJAX / Haiku

Open starter code

11 Implemented

Recent Research

BabyLM

A GPT-2 causal language model trained from scratch.

It reuses the GPT-2 configuration and tokenizer but initializes new model weights, then trains for one epoch with batch size 16 and evaluates perplexity.

Language modelingHugging Face

Open starter code

12 Code baseline

Code Improvement

LLaMA inference

Unoptimized Hugging Face generation with LLaMA-7B.

The timer covers 100 batches of four 128-token contexts and one newly generated token on an A100. Accelerate is imported but not applied.

Text generationTransformers

Open starter code

13 Code baseline

Code Improvement

Vectorization

A deliberately slow four-loop 2-D convolution.

The baseline convolves random 32×64×64×3 input with eight 3×3 filters, stride two, padding two, and ReLU—entirely through nested Python loops.

Array computationNumPy

Open starter code

How the paper builds its tables

Three different things called “baseline”

The paper’s wording compresses three separate layers. Keeping them apart makes Tables 3 and 4—and the starter files—much easier to read.

Starter artifact

The code placed in the agent’s workspace: a runnable model, an incomplete scaffold, dummy predictions, or deliberately slow code.

Paper · Appendix B

Evaluation reference

The denominator used to calculate percentage improvement. Usually this is the mean score of unchanged control runs; six tasks are assigned fixed values in code.

See the exact computation

“Baseline” column

A no-LLM control policy. It executes train.py once and immediately submits, without proposing or testing an improvement.

Open the control policy

Reading the published numbers

Why can the baseline beat itself?

In Table 3, the “Baseline” column is the success rate of repeated no-LLM control runs—not a raw reference score. Most entries are 0%, but Identify Contrails is 40% and CLRS is 42.9%. Stochastic control runs can land more than 10% above the mean reference used as their denominator.

Table 4 reports average percentage improvement over that reference; its “Baseline” column is 0.0 throughout. The improvement formula also reverses the comparison for metrics where lower is better.

Table 3 · success rate ↗Table 4 · average improvement ↗Improvement formula ↗

Reference score ledger

What each task is compared against

“Measured mean” values come from the authors’ unchanged control runs and are not hard-coded in the repository. “Fixed in code” values are literal constants.

Task	Metric	Reference used in calculations	Source
CIFAR-10	Accuracy ↑	Mean final score of unchanged starter runsMeasured mean	plot.py mean ↗
IMDb	Accuracy ↑	0.5 accuracyFixed in code	plot.py constant ↗
ogbn-arxiv	Accuracy ↑	0.3134 accuracyFixed in code	plot.py constant ↗
House Prices	MAE ↓	1 × 10¹⁰ MAE · deliberately poor sentinelFixed in code	plot.py constant ↗
Spaceship Titanic	Accuracy ↑	0.5 accuracyFixed in code	plot.py constant ↗
Parkinson’s Disease	SMAPE ↓	Mean final score of unchanged starter runsMeasured mean	plot.py mean ↗
FathomNet	MAP@20 ↑	1 × 10⁻¹⁰ · effectively zeroFixed in code	plot.py constant ↗
Feedback Prize	MCRMSE ↓	Mean final score of unchanged starter runsMeasured mean	plot.py mean ↗
Identify Contrails	Dice ↑	Mean final score of unchanged starter runsMeasured mean	plot.py mean ↗
CLRS	Mean square error	Mean final score of unchanged starter runsMeasured mean	plot.py mean ↗
BabyLM	Perplexity ↓	Mean final score of unchanged starter runsMeasured mean	plot.py mean ↗
LLaMA inference	Wall-clock time ↓	Mean final time of unchanged code runsMeasured mean	plot.py mean ↗
Vectorization	Wall-clock time ↓	6.1742 secondsFixed in code	plot.py constant ↗

The official workflow explicitly runs the unchanged policy to create baseline evaluations. See the repository evaluation instructions ↗ and the paper’s note that IMDb, House Prices, Spaceship Titanic, and FathomNet have no self-running model baseline in their supplied code.