MatterBench: can an agent deliver a whole legal matter end to end?

MatterBench is the first ultra-long-horizon legal benchmark that grades an agent on real whole legal matters: every deliverable of a real matter, in the order the client asked for it, from the files the lawyer actually had, against what the lawyer actually delivered and what the client and the regulator sent back. Frontier models attempted each matter end to end, four times over. One in six attempts at an ultra-long-horizon task met every check.

A matter is a client's whole case. Each matter is split into ultra-long-horizon (ULH) tasks: a set of deliverables the client asked for together, which the agent must produce in order, in one run. A deliverable is one piece of work, such as a memo, a table or an email. There are 8 ULH tasks, two to four per matter. Each matter is 100+ lawyer-hours of work. The other 8 tasks are short: one deliverable on its own, mostly taken from a ULH task and graded alone.

Matter 1: Regulatory licensing application15 deliverables · 103 checks · 22 files
Matter 2: Commodity supply9 deliverables · 82 checks · 6 files
  • ULHCrude supply engagementLongest run: GLM-5.36 deliverables · 67 checks · 6 files
    • ShortAdvice on an unopened letter of credit7 checks · 4 files
    • ShortDemurrage rebuttal after a delivery switch10 checks · 6 files
  • ULHDelivery-term switchLongest run: Gemini 4 Argon3 deliverables · 10 checks · 4 files
    • ShortEmail weighing the receiver's pricing position8 checks · 4 files
Matter 3: Corporate due diligence21 deliverables · 147 checks · 13 files
  • ULHSoftware contract chainLongest run: DeepSeek 4.1 Flash5 deliverables · 44 checks · 7 files
  • ULHAML gap analysis of payment agreementsLongest run: DeepSeek 4.1 Flash7 deliverables · 81 checks · 5 files
  • ULHCorporate due diligenceLongest run: GLM-5.35 deliverables · 13 checks · 1 file
  • ULHFinance and security due diligenceLongest run: Gemini 4 Argon4 deliverables · 9 checks · 7 files
    • ShortFinance and security flags table3 checks · 1 file
    • ShortFinance flags from registers and statements5 checks · 2 files

Matter checks count each deliverable once; a short task mostly repeats checks from its ULH task.

Last updated 13 November 2026.

Contents

1. Leaderboard

Loading the leaderboard.

Score is the share of attempts in which the model passed the task. An attempt passes only when every check is met. Partial credit is the mean reward per attempt. Scores a few attempts apart are a group, not a ranking.

The best model scores 53.1%: Claude Opus 5.5 passes 34 of its 64 attempts. Gemini 4 Argon scores 45.3%, Grok 4.7 43.8%, Claude Fable 5.2 40.6%. No model scores above 54%. On partial credit the top four sit between 90% and 87%. 35% of ULH task attempts met every critical check.

2. Why real matters

A test written by hand has the traps someone chose to set. The hardest and most valuable legal tasks are the ones nobody designed and delivered in the real world. These tasks are built from real matters.

Authored tasks have a ceiling. The author chooses the trap, so the trap has a shape, and models trained on enough shaped tasks learn the shape. Real matters do not cooperate. The conflict sits where the counterparty put it. The key document is a scanned multilingual PDF. The client changes position between the first email and the third. The regulator sends queries numbered past 120 and a consultancy edits the submission without redlines. Nobody designed any of that, and it is exactly the work a law firm is paid for.

MatterBench is built from work that lawyers delivered to clients. Each task is one piece of that work, as it was assigned, with the matter file as it stood on the day, graded against what the lawyer actually delivered and the feedback it actually received. Two properties follow from that, and neither of them can be bolted onto an authored task:

  1. The instruction is the partner's real request. "Prepare the email requested in the facts memo." "Draft the Regulatory Business Plan amendments for the MA broker-dealer application." The agent is told what an associate was told, no more.
  2. The rubric and its checks come from the matter's own record. They are drawn from the deliverable that went to the client and from what surrounded it: before delivery, the lawyer's reasoning, captured in their discussions with and instructions to our agents; after it, the client's feedback, the partner's redline, the client's instruction note and the regulator's comments.

3. What current legal evals miss

Every legal benchmark until now has been built from tasks someone wrote for the test and graded against answers that experts authored. That holds for Harvey's Legal Agent Benchmark (LAB), GDPx legal, GDPval and LegalBench, and for DeepFinance in finance.

In each case a person decided where the difficulty would be, wrote the documents to put it there, and wrote the answer. What is left for MatterBench to say is what none of them do: the unit is the ULH task, the reference answer was observed, and the files are the matter's own.

Where tasks come fromUnit of workWho wrote the answerWhat is public
Harvey LABreal client matters, broken into delegated tasks; documents generatedone delegated taskexpertsthe tasks, open source
GDPx legalassignments from the partners' own practice; files generatedone task, one hourpartnersone task with its runs
DeepFinance (Julius)real data, scrubbed; tasks composed by finance professionalsone task of 20 to 30 hoursexpertsone sample prompt
MatterBenchthe matter record itselfa deliverable, or a ULH taskthe lawyer's delivered work and the feedback it receivedfive tasks with checks and every attempt
Checks per task
DeepFinance (Julius), average84
MatterBench, largest ULH task81
GDPval, lawyer tasks (OpenAI), average60
Harvey LAB, average57

4. Making tasks for MatterBench

Tasks begin with the assignment as the supervising partner actually wrote it, on the day it was written, to the associate who did the work. The agent takes the associate's seat: it should understand its role, objective and expected deliverable from the prompt, and nothing in the prompt tells it where the difficulty is. The complexity lies in the matter file: interacting provisions across a supply agreement and two sets of general terms, a regulator's comments against a consultancy's silent edits, a client position that shifted between letters, and facts that need careful interpretation.

The environment is the matter folder as it stood when the piece of work was assigned: agreements and compared versions, regulators' rulebooks and written feedback, client instruction notes, and the law. The rubric the judge grades against is drafted from the delivered work product and the feedback it received, one atomic pass-or-fail check at a time. Each check is anchored to its source. Red herrings are written from the mistakes the lawyer avoided and the demands the client did not make. Critical checks mark the points the partner would reject the work for.

5. The data

The matters come from partner law firms under licence, with the consent of their clients, and only once the matter has been delivered and closed. Before a matter becomes a task it goes through clearance: people, companies, ports, vessels, addresses and numbers are replaced by placeholders that stay consistent within a task.

6. Results

An attempt passes only when every check in the task's rubric is met.

Partial credit is the share of checks met, reduced for each red herring the agent fell for and halved when a critical check is missed.

Short and ULH tasks, by model
Short and ULH tasks by model, in order of ULH attempts that met every critical check
ModelShort: passedULH: passedShort: partial creditULH: partial creditShort: every critical check metULH: every critical check met
All models55%17%83%79%86%35%
Claude Fable 5.150.0%28.1%86.2%87.4%96.9%62.5%
Claude Opus 5.568.8%37.5%91.1%89.1%100.0%59.4%
Claude Fable 5.246.9%34.4%85.3%88.3%96.9%59.4%
Grok 4.768.8%18.8%94.2%83.3%100.0%46.9%
Gemini 4 Argon75.0%15.6%96.3%82.9%100.0%34.4%
GPT-6 Astra46.9%12.5%74.1%80.1%71.9%25.0%
GLM-5.365.6%3.1%84.1%72.6%81.3%18.8%
DeepSeek 4.1 Flash34.4%3.1%62.9%69.6%56.3%12.5%
DeepSeek V4-Pro37.5%0%68.8%59.8%68.8%0%

All models: 158 of 288 short attempts passed and 49 of 288 ULH attempts; every critical check was met in 247 of 288 short attempts (86%) and 102 of 288 ULH attempts (35%).

Critical checks mark the points a partner would reject the work for.

Quantitative analysis

  1. One ULH attempt in six passes. On short tasks the best model passes 75% of its attempts; on ULH tasks 49 of 288 attempts pass (17%). Claude Opus 5.5 (90.1%), Gemini 4 Argon (89.6%) and Grok 4.7 (88.7%) lead on partial credit, and a paired bootstrap over tasks cannot separate them, or Claude Fable 5.2 (86.8%) from any of them. Claude Opus 5.5 is first on both.
  2. On ULH tasks, models miss the points a partner would reject the work for. 86% of short-task attempts met every critical check; on ULH tasks, 35% did. A ULH task carries four times as many critical checks, and missing one is enough. The Anthropic models did best: Claude Fable 5.1 cleared every critical check in 63% of its ULH attempts, and Claude Opus 5.5 and Claude Fable 5.2 in 59% each; no other model passed 47%.
  3. Gemini 4 Argon leads the Pareto frontier. It dominates Grok 4.7, Claude Fable 5.2, GLM-5.3, GPT-6 Astra and Claude Fable 5.1, costing less than each and earning more partial credit, and comes within 0.5 points of Claude Opus 5.5 at a third of its cost. Claude Fable 5.1 costs seven times as much as Gemini 4 Argon and two and a half times as much as Claude Opus 5.5, and ranks fifth. See the score-versus-cost chart on the overview.

Qualitative analysis: where models fail

  1. Agents ask where the lawyer advises. The regulator asked why it should rely on the parent company's funding. The file held two instruments: a parent commitment letter and a draft subordinated intra-group loan. The lawyer's direction was to tell the client that the loan may serve better than the letter. No attempt did. 32 of 36 put both instruments to the client as an open choice; 4 asked only for the signed letter. In the same questionnaire, 75% of attempts asked the client to confirm in writing a point that the policy on file already states. The lawyer's rule: if it is confirmed, do not ask.
  2. Agents treat sister companies as strangers. A software licence ran from an outside vendor through one group company to its sister company. The facts memo says the two are under common ownership and control. 94% of attempts still rated the gaps between the two sisters (no service-level buffer, no termination fee) as high-priority risks, in the same class as the exposure to the outside vendor. Two attempts made the distinction: one by Claude Opus 5.5, one by Claude Fable 5.1.
  3. The local statute book splits the field. The client asked if late-payment interest ever stops under the mandatory rules of the contract's governing law, a civil-law system. The lawyer's answer rests on two rules of the civil code: agreed interest cannot exceed 7% a year, and total interest cannot exceed the principal. The agents had no web access and the code was not in the files. Claude Opus 5.5, Claude Fable 5.1, Claude Fable 5.2 and Gemini 4 Argon stated both rules in all 16 attempts. GPT-6 Astra, DeepSeek 4.1 Flash, DeepSeek V4-Pro and GLM-5.3 stated the yearly cap in none of 16 and the principal cap in 1 of 16.
  4. Agents miss what is missing from the data room. In the finance review, two later documents refer to a facility addendum that is not in the data room. 44% of attempts in the ULH task and 42% in the short task did not flag it; some named the addendum and still did not say it was missing. The facilities had also expired five days before the report date with no renewal on file; 31% of ULH-task attempts did not say so. In the ULH task, Claude Opus 5.5, both Claude Fable models and GPT-6 Astra flagged the addendum every time. DeepSeek 4.1 Flash, DeepSeek V4-Pro and GLM-5.3 never did.

7. Explore the results

Two views of the results.

  • Behaviour. On the five public tasks, each attempt's Grading tab lists every check with the judge's verdict and its reason. See a public task's rollouts.
  • Efficiency. The overview page plots partial credit against cost per attempt, on a log scale, with the Pareto frontier and a toggle to output tokens. See the chart on the overview.