Materials WorkFlow Benchmark by Collinear AI

MatFlowBench

Can frontier AI do real materials research?

MatFlowBench evaluates how reliably and efficiently LLM agents carry out the materials science workflows of computational and experimental scientists. Agents run multi-step calculations, request experimental measurements, and interpret the results to reach scientific conclusions.

Talk to a researcherView leaderboardView workflows

45 tasks, private benchmarkRead the launch post →

01 / The leaderboardResults by track

Four models clear a 40% pass rate.
The other six stay under 30%.

MatFlowBench leaderboard: Total
#ModelHarnessPass rateRollouts passedPass@1Tasks solved
1GPT-6 AstraOpenAICodex60.8%79 / 13060.0%34 / 45
2Claude Opus 5.5AnthropicClaude Code53.8%70 / 13056.7%30 / 45
3Claude Sonnet 5.5AnthropicClaude Code47.7%62 / 13050.0%32 / 45
4GPT-6.1 SolOpenAICodex44.6%58 / 13045.0%30 / 45
5Grok 4.7xAITerminus 226.2%34 / 13022.8%17 / 45
6Gemini 3.8 FlashGoogleTerminus 225.4%33 / 13018.9%14 / 45
7Kimi K3Moonshot AITerminus 223.8%31 / 13022.2%16 / 45
8Tencent HY4 PreviewTencentTerminus 222.3%29 / 13020.0%16 / 45
9Muse Spark 1.3MetaTerminus 220.0%26 / 13016.7%15 / 45
10Qwen3.8-27BAlibabaTerminus 211.5%15 / 1308.3%9 / 45

Total: 45 tasks, all rollouts. Every model ran at high reasoning effort. Results from October 2026.

02 / CostPareto frontier by track

The most expensive model
is not the most accurate.

Pareto frontierOn the frontierOther models
0%10%20%30%40%50%60%70%$3$5$10$20$50$100Average cost per rollout (USD, log scale)Pass rateGPT-6 Astra: 58.0% pass rate, $71.26 per rolloutGPT-6 AstraGPT-6.1 Sol: 46.0% pass rate, $11.87 per rolloutGPT-6.1 SolGemini 3.8 Flash: 2.0% pass rate, $4.75 per rolloutGemini 3.8 FlashGrok 4.7: 14.0% pass rate, $30.25 per rolloutGrok 4.7Kimi K3: 18.0% pass rate, $11.52 per rolloutKimi K3Muse Spark 1.3: 8.0% pass rate, $51.33 per rolloutMuse Spark 1.3Qwen3.8-27B: 0.0% pass rate, $4.25 per rolloutQwen3.8-27BClaude Opus 5.5: 64.0% pass rate, $8.23 per rolloutClaude Opus 5.564.0%$8.23Claude Sonnet 5.5: 56.0% pass rate, $4.87 per rolloutClaude Sonnet 5.556.0%$4.87Tencent HY4 Preview: 14.0% pass rate, $4.37 per rolloutTencent HY4 Preview14.0%$4.370%10%20%30%40%50%60%70%$3$5$10$20$50$100Cost per rollout (USD, log scale)Pass rateGPT-6 Astra: 58.0% pass rate, $71.26 per rolloutGPT-6 AstraGPT-6.1 Sol: 46.0% pass rate, $11.87 per rolloutGPT-6.1 SolGemini 3.8 Flash: 2.0% pass rate, $4.75 per rolloutGemini FlashGrok 4.7: 14.0% pass rate, $30.25 per rolloutGrok 4.7Kimi K3: 18.0% pass rate, $11.52 per rolloutKimi K3Muse Spark 1.3: 8.0% pass rate, $51.33 per rolloutMuse SparkQwen3.8-27B: 0.0% pass rate, $4.25 per rolloutQwen3.8Claude Opus 5.5: 64.0% pass rate, $8.23 per rolloutOpus 5.564.0%$8.23Claude Sonnet 5.5: 56.0% pass rate, $4.87 per rolloutSonnet 5.556.0%$4.87Tencent HY4 Preview: 14.0% pass rate, $4.37 per rolloutHY414.0%$4.37

Computational: 25 tasks × 2 rollouts per model. Cost is model API spend plus estimated compute, averaged per rollout.

03 / The constructWhat a successful task establishes

Completing a
research workflow.

MatFlowBench measures whether an LLM agent can complete a specified materials research task using the available tools and data, within the task’s resource limits.

The verifier checks the evidence behind an answer. In DFT tasks it reads the agent’s calculation output files rather than the number the agent reports. In budgeted searches it checks which measurements the agent requested: a recommended material must have been measured, and a claim that no material qualifies must rest on measurements that rule out each candidate.

A pass means that the submitted result meets the task’s checks. It does not by itself establish new materials discovery, complete physical validation, or sound reasoning at every step.

01 Interpret the objective02 Use tools and data03 Submit the result04 Check the result
04 / The workflowsTwo evidence settings

Computational and
experimental workflows.

In silico25 tasks

Computational

Agents are given a scientific goal and a prepared environment with Quantum ESPRESSO, Wannier90 and Phonopy for density functional theory (DFT) and related calculations. They plan and run the calculations themselves, much as a computational materials scientist would, and must carry the work through to the property or conclusion the task asks for.

Task families

  • Electronic structure and topologyLocate band crossings and determine topological properties.
  • Spin and anomalous Hall transportCalculate spin Hall or anomalous Hall response for a specified electronic and magnetic state.
  • Magnetism and spin structureDetermine magnetic onset, spin splitting, or band crossings in a specified magnetic state.
  • Strain and composition-driven transitionsLocate changes in electronic structure as strain, doping, or composition varies.
  • Strained-film propertiesRelax constrained films and calculate polarization or conduction-band properties.
  • Wannier model constructionBuild a model that reproduces specified energy bands and their alignment.
  • Lattice dynamicsCalculate selected phonon frequencies and locate an acoustic-branch minimum.
From measured data20 tasks

Experimental

Agents receive a set of target properties and access to a simulated lab, where every experiment returns real measured data. Each sample can be characterized on several instruments, and each instrument reveals a different set of properties. With a finite budget to spend, the agent decides which samples to prepare and which measurements are worth running, using what it learns to narrow the search until it finds a material that meets the requirements or can show that none does.

Task families

  • Optical-film selectionFind coatings and filters that meet several spectral requirements within a measurement budget.
  • Catalyst selectionSelect oxygen-evolution catalysts that meet activity and conditioning requirements.
  • Feasibility triageDetermine which target specifications a materials library can meet.
  • Raw instrument-data reductionConvert spectrometer and electrochemical files into corrected measurements and candidate selections.
05 / EvaluationHow a rollout is scored

What the score means.

Each rollout passes or fails. The task’s executable verifier makes that call, and the leaderboard reports how often each model passes.

01

Numerical references

Results are compared with numerical references, using tolerances set by the underlying physics.

02

Physical constraints and evidence

Other checks cover physical constraints from well-established scientific workflows. Some tasks also require calculation files or measurement logs, and budgeted searches must finish within their measurement budget.

03

Pass rate and Pass@1

Each model ran with high reasoning effort, twice per computational task and four times per experimental task. Pass rate is the share of all rollouts that passed, and the leaderboard is ranked by it. Pass@1 is the share of passing rollouts on a task, averaged so that every task counts equally.

To request an evaluation, contact science@collinear.ai.

06 / TrainingRL environments in progress

Collinear is building RL environments
on these workflows.

MatFlowBench shows where frontier agents fall short in materials research. The environments, with verified trajectories, target three gaps in these results.

01

Multi-step DFT pipelines

No model outside the top four passes more than 18% of computational rollouts.

02

Reliability across attempts

Across the 10 models, only 38% of the tasks solved at least once are solved on every attempt.

03

Cost per run

On computational tasks the most accurate model averages $8.23 a rollout. Models with lower pass rates spend up to $71.26.

Talk to a researcher
07 / QuestionsFAQ and citation

Frequently asked
questions.

What is MatFlowBench?

MatFlowBench is a private benchmark from Collinear AI that measures whether LLM agents can carry out multi-step materials research workflows. It has 45 tasks: 25 computational tasks, in which agents run density functional theory (DFT) and related calculations with Quantum ESPRESSO, Wannier90 and Phonopy, and 20 experimental tasks, in which agents plan measurements, analyze archived instrument data and select materials.

Which model scores highest on MatFlowBench?

In the October 2026 results, GPT-6 Astra has the highest pass rate on all 45 tasks at 60.8%, followed by Claude Opus 5.5 at 53.8% and Claude Sonnet 5.5 at 47.7%. On the computational tasks Claude Opus 5.5 leads at 64.0%, and on the experimental tasks GPT-6 Astra leads at 62.5%.

How is a task scored?

Each rollout passes or fails, and the task’s executable verifier makes that call. For DFT tasks, the verifier reads the agent’s calculation output files rather than the number it reports. For budgeted searches, it checks which measurements the agent requested: a recommended material must have been measured, and an agent that reports that no material qualifies must have measured enough of the pool to rule each one out.

How were the models run and ranked?

Each model ran with high reasoning effort, twice per computational task and four times per experimental task. OpenAI models ran in Codex, Anthropic models in Claude Code, and all others in Terminus 2. The leaderboard ranks models by pass rate, the share of all rollouts that passed. Pass@1 averages each task’s share of passing rollouts, so that every task counts equally. Cost is model API spend plus estimated compute, averaged per rollout.

Can I download the tasks?

No. MatFlowBench is a private benchmark: its tasks, verifiers and model rollouts are not published.

Can I have my model evaluated?

Yes. To request an evaluation, contact science@collinear.ai.

Does Collinear offer training environments for these workflows?

Collinear is building RL environments and verified trajectories on these workflows. To talk about them, contact science@collinear.ai.

If you use MatFlowBench in your research, please cite:

@misc{matflowbench2026,
  title        = {MatFlowBench: Materials Workflow Benchmark},
  author       = {{Collinear AI}},
  year         = {2026},
  howpublished = {\url{https://matflowbench.com}},
  note         = {45 computational and experimental materials research workflows for LLM agents.}
}