The mistakes a better model does not fix

We gave nine AI models real legal work from closed matters, and graded it against what the lawyer delivered. Some of their mistakes fade as the models get stronger. Others do not. On three points of professional judgment, the four strongest models got it wrong in 43 of 48 attempts, and the five others in 54 of 60.

Prysm research · 18 November 2026.

Contents

1. In brief

  • There are two kinds of mistake. We looked at the points of legal work that the models got wrong most often. In most of them, either the model lacks something (a rule of law, a document, a date), or it has everything it needs and still acts wrongly.
  • Stronger models mostly fix the first kind. The four strongest models, by overall score in the benchmark, made these mistakes in 13 of 128 attempts (10%). The five others made them in 106 of 160 (66%).
  • No model fixes the second kind. On three points, the four strongest models made these mistakes in 43 of 48 attempts (90%), and the five others in 54 of 60 (90%).
  • The second kind is the lawyer's own job. Take a position. Know who the parties are to each other. Do not waste the client's time.
Does a stronger model fix the mistake?

Type 1: yes, mostly

The five other models: 66%. The four strongest: 10%.

0%50%100%The five other models: 106 of 160 attempts got it wrongThe four strongest models: 13 of 128 attempts got it wrong66%10%Five other modelsFour strongest models

8 checks: law that is not in the file, missing documents, dates.

Type 2: no, it does not

The five other models: 90%. The four strongest: 90%.

0%50%100%The five other models: 54 of 60 attempts got it wrongThe four strongest models: 43 of 48 attempts got it wrong90%90%Five other modelsFour strongest models

3 checks: advise, know the parties, do not over-ask.

How often the models got the point wrong, as a share of their attempts. The four strongest are the models with the highest overall score. Each model did each task four times.

We call the first kind Type 1 and the second kind Type 2. The rest of this piece shows what each looks like in real work, and what a law firm can do about it.

2. Three mistakes no model fixes

These are the Type 2 mistakes. Nothing is missing. The facts are in the file, the model reads them, and often it repeats them. Then it does the reverse of what the lawyer did.

36 of 36attempts got it wrong

Advise. Do not ask.

In the file
Two ways to document a parent company's funding: a commitment letter and a draft subordinated intra-group loan.
The lawyer
Recommended the loan, and explained why it gives the regulator more comfort than the letter.
The agents
32 of 36 attempts put both instruments to the client as an open choice. 4 asked only for the signed letter. None recommended the loan.
34 of 36attempts got it wrong

Know who the parties are to each other.

In the file
A software licence that runs from an outside vendor through one group company to its sister. The facts memo says the two are under common ownership and control.
The lawyer
Rated the gaps between the sisters as low risk: an intra-group matter, unlikely to be litigated.
The agents
Rated the same gaps as high-priority risks, in the same class as the exposure to the outside vendor.
27 of 36attempts got it wrong

Do not ask what the file already answers.

In the file
A policy that already states what the monitoring vendor's system detects.
The lawyer
Struck the question from the list for the client: “if we understand it is confirmed, why waste time and why ask?”
The agents
Asked the client to confirm it in writing.

The three cases have one shape. Each time, the lawyer's move was to commit: to pick one instrument, to discount a risk, to strike a question. Each time, the model's move was to hedge: to offer both, to flag everything, to ask again.

This is not a gap in knowledge, and one pair of points shows it. The same fact, that the companies are in one group, matters twice in these matters. It makes the corporate-governance rules more relevant to a custody arrangement inside the group, and Claude Opus 5.5, Gemini 4 Argon, Claude Fable 5.2, Claude Fable 5.1 and GPT-6 Astra kept those rules in scope every time. It also makes a contract dispute between sister companies less likely, and 34 of 36 attempts missed that. Where the fact called for more caution, five of the nine models used it every time. Where it called for less, no model did so reliably.

3. Why this is the lawyer's role, not model quality

A lawyer is paid to take a position, to know who the parties are to each other, and to protect the client's time. In the three cases above the models did the reverse: they gave a menu, treated sister companies as strangers, and asked a question the file already answered.

The models are trained to be careful and to defer. In most settings that is a virtue. Here that habit is the mistake, and more intelligence does not remove it. Claude Opus 5.5, the model with the highest overall score in the benchmark, got these three points wrong in 11 of 12 attempts.

That changes what a firm should expect from the next model release. A new model will know more law and miss fewer gaps in a data room. On this evidence, it will still hand the decision back.

4. The mistakes a stronger model does fix

These are the Type 1 mistakes. Something is absent from the model: a rule of law it does not hold, a gap in the file it does not notice, a date it does not compare with today's.

Law that is not in the file. A client asked if late-payment interest ever stops under the mandatory rules of the contract's governing law, a civil-law system. The answer rests on two rules of the civil code, and the code was not in the files. No model had web access. Claude Opus 5.5, Gemini 4 Argon, Claude Fable 5.2 and Claude Fable 5.1 stated both rules in every attempt. GLM-5.3, GPT-6 Astra, DeepSeek 4.1 Flash and DeepSeek V4-Pro missed them 31 of 32 times. Some wrote that they could not confirm the position under that law.

What is absent. In a finance review, two later documents refer to a facility addendum that is not in the data room. Several attempts named the addendum and still did not say that it was missing. The facilities had also expired five days before the report date. GLM-5.3, DeepSeek 4.1 Flash and DeepSeek V4-Pro missed this in 11 of 12 attempts; some noted the expiry date and advised the client to diarise it.

Our reading. These are real failures, and for a firm they are the cheaper kind. They close with the choice of model. The split is also sharp, not gradual: on the two interest rules, four models are right every time and four are wrong almost every time. A firm should test a model on its own governing law before anything else.

5. Why other tests of legal AI cannot show this

To grade “the model asked where it should have advised”, you need to know what the lawyer did at that exact point.

In most tests of legal AI, an expert writes the task and the model answer in advance. That answer holds facts to find and rules to state. It does not hold the partner's correction to a draft, because no draft and no partner existed.

MatterBench has the correction, because its tasks come from the record of real matters: the files the lawyer had, what the lawyer delivered, and what was said along the way. The three mistakes above are not points that we thought a model should make. They are points on which a lawyer corrected the work, in a real matter, in writing.

6. What a law firm can do with this

Type 1 is a buying decision. Choose the model on evidence from your own kind of matter and your own governing law. On these points, some models are right every time and others almost never.

Type 2 is not a buying decision. No model in this benchmark does it reliably, so an upgrade alone will not supply it. When to advise and when to ask, how much weight a group relationship carries, which questions a client should never see: these are a firm's own standards. A system has to be given them, and then tested against the firm's own matters to show that it follows them.

We have not yet tested how far an instruction removes a Type 2 mistake. That is the next experiment.

7. How we did this

The work. MatterBench is built from three closed legal matters. Each model received the files the lawyer had and the request as it was written, and produced the work: memos, tables, emails. Nine models did each of the 16 tasks four times, so each task has 36 attempts. The benchmark write-up gives the full method.

The checks. A check is one point that the lawyer's own work made, or that the client or the regulator raised. A judge model reads the AI's work and decides if the point is there. A critical check is a point that a lawyer would treat as a must. There are 89.

The 14. Of the 89 critical checks, the models got 61 right almost every time: at most 3 of the 36 attempts got them wrong. We took the ones that at least 30% of attempts got wrong. There are 14, and they hold 68% of all the critical mistakes (271 of 401). Thirteen of them are in the long tasks, where the model must deliver several pieces of work in one run.

The two types. For each of the 14 we asked one question: does a stronger model get it wrong less often? For 8, yes: Type 1. For 3, no: Type 2. The other 3 are about precision and follow no order of models.

Choose a point to see which models got it wrong
Type 2: no model fixes itType 1: a stronger model mostly fixes itPrecision

Type 2 · check 1 of 14

Recommended one of two funding instruments to the client, and said why

36 of 36 attempts got it wrong

Claude Opus 5.54 of 4
Gemini 4 Argon4 of 4
Grok 4.74 of 4
Claude Fable 5.24 of 4
Claude Fable 5.14 of 4
GLM-5.34 of 4
GPT-6 Astra4 of 4
DeepSeek 4.1 Flash4 of 4
DeepSeek V4-Pro4 of 4
An attempt that got it wrongAn attempt that got it right
Show all 14 as a table
#What the lawyer didOpus 5.5Gemini 4Grok 4.7Fable 5.2Fable 5.1GLM-5.3GPT-6DS FlashDS V4-ProAll
Type 2
1Recommended one of two funding instruments to the client, and said why44444444436 of 36
2Rated the gaps between two sister companies as low risk34443444434 of 36
3Did not ask the client a question that the file already answers40444440327 of 36
Type 1
4Stated the yearly cap on interest under the governing law00200444418 of 36
5Stated that total interest cannot exceed the principal00200344417 of 36
6Named the statute under which an asset-freeze committee acts00100444417 of 36
7Flagged that an addendum, which two later documents refer to, is not in the data room03100404416 of 36
8Flagged the same missing addendum, as a short task00011334315 of 36
9Found that only two of eight agreements carry an anti-money-laundering clause00000144413 of 36
10Kept the corporate-governance rules in scope, because custody is outsourced inside the group00300304212 of 36
11Flagged that the bank facilities had expired five days before the report date00000304411 of 36
Precision (no order of models)
12Mapped all seven core laws and rulebooks to the first review cluster34301442122 of 36
13Cited the schedule of the service agreement that fixes daily backup and twice-yearly recovery tests00443031217 of 36
14Gave both halves of the payment-aggregator rule32043002216 of 36

Attempts that missed, of four. A stronger orange means more.

Each row is one model, strongest first by overall score. Each dot is one attempt.

The three precision checks. Checks 12, 13 and 14 are about the right content with the wrong pinpoint, or half of a rule. On the payment-aggregator rule, the three Claude models missed 10 of 12 attempts, because they gave the first half of the rule and left out the second, while Grok 4.7, GLM-5.3 and GPT-6 Astra missed none. We do not draw a conclusion from three checks, and one of them is an exact-text match that a correct answer in other words can fail.