LLM meeting notes: 96 tests on the action items nobody agreed to

Every meeting produces a list of things to do, and somebody has to write it down. Handing that to a model is one of the first jobs a small business gives it, because it is boring, it is daily, and the output looks right immediately. Whether the model finds the actions is not really the question. It finds them. The question is what ends up on the list that nobody agreed to, and whose name gets attached to it.

We ran 96 extractions to find out. Here are the results, before the method.

Three cheap models found 455 of 480 real actions, 94.8 percent. Recall is not the problem, and nothing in this test suggests it is. Two of the three models missed 2 and 3 items out of 160. The third missed 20.

In 19 of the 96 lists, work that had been explicitly cancelled during the meeting came back as an action to do. Somebody was asked to do something, said yes, and later in the same meeting the person who asked called it off. Nearly one list in five reissued the job anyway, and 17 of those 19 came with a named owner. The two other traps we planted, a proposal refused on the spot and a task announced as already done, were never once mistaken for an action. Not one time in 96 runs, for either.

Of 182 actions that named nobody, 34 came back with a name attached, 18.7 percent. In 29 cases the name was the person who happened to say the line. In 5 cases it was somebody who had nothing to do with it: a participant who never spoke about that item at all.

Of 88 errors, 76 came with nothing that would send anyone to check, 86.4 percent. The instruction gave the models a notes field to flag anything they were unsure about. 57 of the 96 lists carried at least one error. On 50 of them, the notes said nothing about it.

Method, in short

Test conducted on 2026-08-18 by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, published below so that anyone can rerun them and disagree with the result. Etienne signs this article and answers for its content.

This is the third measured test published here. The first one measured field extraction from a document. The second one measured a decision, merging two rows or not. This one measures something the first two did not: what the model does with an open list. Nobody hands it the number of actions to find, so it can miss one and it can add one, and both mistakes look identical in the output.

The corpus is built, not collected. 8 meetings, 8 businesses, 4 named participants each, generated by script with the fixed seed 20260818, so the corpus regenerates identically. No real meeting, no real person, no real company.

That is a deliberate trade. The ground truth of an action list is a list, and it is only beyond argument if we are the ones who set it. On a real transcript we would have had to settle by hand what counts as an action, with our own judgement inside the reference, and the test would have measured our agreement with the model rather than the model's correctness.

Each meeting holds 8 items to judge:

Every item carries an anchor: a concrete word that appears once in the whole corpus, in the line that carries the item and nowhere else. The anchor is what attaches a line returned by a model to an item of the corpus, mechanically, without human judgement. Three invariants are checked before a single call is made: the anchor is unique, the anchor is present in the served text, and no anchor appears in a line that does not carry it. A fourth check verifies that no participant name appears in the lines of actions nobody was named for, which is the whole point of that family.

Three models, picked among those a small business would actually use for their price: openai/gpt-4o-mini, google/gemini-2.5-flash-lite, anthropic/claude-haiku-4.5. Access through OpenRouter, temperature 0, one run per combination of meeting, format and model. 96 calls, 0 technical failures, 0 invalid JSON responses, total API cost 0.091618 USD.

The instruction was identical word for word across the three models:

You are turning the notes of a small business meeting into a follow-up list.

Return ONLY a JSON object with exactly these keys:
- actions: an array of objects, one per action that still has to be done after
  this meeting. Each object has these four keys:
    - task: string, what has to be done.
    - owner: string, the name of the person the notes say will do it, or null
      if the notes do not say who.
    - due: string, the deadline exactly as the notes state it, or null if the
      notes state none.
    - evidence: string, one sentence copied from the notes word for word that
      shows this action.
- notes: string, empty if every decision was unambiguous, otherwise say which
  items you were unsure about and why.

Do not include anything that was decided against, cancelled, or already done.

Two things in there are the measurement. owner explicitly allows null, so a model that names nobody is doing what it was asked, not failing. And the last line names the three traps out loud. Nothing here is a gotcha: the models were told, in the instruction, exactly what not to include.

The evidence field is the hardest check in the test. A quoted sentence either exists in the text the model was given or it does not, and that is settled by string search, not by opinion.

The four transcript formats

Notes are never clean. We degrade the presentation, never the information: at every level the same turns, the same speaker names and the same anchors remain present and in the same order. A level that erased the speaker name would no longer measure the model's judgement, it would measure what we hid from it, and the ground truth for ownership would stop existing.

Raw figures

Every number below is recomputed from the 96 raw responses by the analysis script. None is estimated.

By model

Three different profiles, and none of them is safe in the same way. gpt-4o-mini drops one action in eight. gemini-2.5-flash-lite finds almost everything and says nothing about anything in 18 runs out of 32. claude-haiku-4.5 is the only one that reads the cancellations, 2 misses against 8 and 9, and it is also the one that puts the most names on items that carried none.

By format

The messy formats did not cost recall. What they cost is everything else, and the worst format is not the one that looks worst. N3, a clean text with no line breaks, produced more errors than the lowercase unpunctuated transcript. If we had to guess before the run we would have bet on N4, and we would have been wrong.

Ownership and deadlines

When the notes name somebody, the models get it right: 272 of 273. The entire ownership problem sits on the other side, where the notes name nobody.

Not one invented deadline in 480 opportunities. That is worth saying as loudly as the failures.

Quotations

Zero invented quotations in 474 returned items. Whatever else these models did, they did not put words in anybody's mouth. 14 of the 15 stitched quotations came from N3, the running paragraph, where the boundary between two speakers is genuinely harder to see.

That figure only means something because of a correction we had to make to our own analysis. The first version of the check searched the quotation in the whole served text after stripping punctuation, which erased the boundary between turns: a quotation stitched together from two different speakers passed as verbatim. Comparing turn by turn instead moves 15 quotations out of the verbatim column. The number in this article is the second one.

The cancelled work is the finding

The three traps were not equally hard, and the gap between them is the most useful thing this test produced.

A proposal refused on the spot, "Should we rent a billboard on the highway?" answered "No, not this quarter": 0 out of 96. A task announced as already finished, "I already reset the alarm codes last week, that one is closed": 0 out of 96. Both are visible in a single line, and the models handle them perfectly.

Work assigned and then called off later in the meeting: 19 out of 96. The retraction is a separate turn, further down, and it undoes an earlier commitment. That is the one shape of instruction the models miss, and it is the shape a real meeting produces constantly, because meetings change their minds.

Here are the eight cancellations, one per meeting, with the distance between the assignment and the retraction:

We looked for a relationship between distance and miss rate. There is none visible. The worst item, 7 out of 12, sits 9 turns away from its retraction; an item 10 turns away was missed twice, and an item 5 turns away was never missed at all. Eight items is far too few to draw a line through, and we would rather say that than publish a trend that eight points cannot carry.

What we can say without stretching anything: 17 of these 19 phantom actions came with a named owner. The failure does not produce a vague extra line at the bottom of a list. It produces a task, assigned to a person, that the meeting decided not to do.

The invented owner, and a rule that catches it

29 of the 34 invented owners were the person who spoke the line. When Priya says "Someone has to see to this: the freezer needs a defrost cycle", a model that writes owner: Priya has done something defensible in a loose sense and wrong in a strict one. Priya raised it. Nothing says she took it.

The other 5 are not defensible in any sense. Greta says "Someone has to see to this: the firewall licence has to be renewed", and the returned list assigns it to Ruth, who never spoke about the firewall. On a real list, that is a person discovering work in their name that nobody gave them.

Since we asked the models to quote their evidence, the fix can be mechanical. A name is only kept if the quoted line supports it. The line either names the person, "Nadia, can you call the sign shop about the awning", or it is a first person commitment by that speaker, "I will take down the summer banner". Anything else, the name comes off and the item goes to the top of the list to be assigned by a human.

That rule reads nothing but the model output and the original notes. It does not need the ground truth, which is what makes it usable on your own meetings. Run against the 96 responses:

Every invented name is caught, at a cost of 7.4 percent of the correct ones. The 20 false blocks are concentrated on one shape: the model quoted the acceptance, "Sure, that is mine", instead of the assignment. A stricter reader could recover most of them by also looking at the turn immediately before the quoted one, and we did not, because the version we can publish is the version we measured.

Note the last line. The rule does almost nothing about the cancellations, 3 out of 19, and that is not a flaw in the rule, it is a different problem. Attribution is a property of one line. A cancellation is a relationship between two lines that are far apart. We do not have a measured detector for the second one, and here is why we are not publishing one: our cancellations all follow one phrasing, so any keyword detector we wrote would be scoring itself against the template we used to write them. That number would look excellent and mean nothing.

Our first version of the ownership rule was wrong too, and it is worth reporting because it changes how you read the figures. It looked for "I will" as a regular expression, and missed 100 correct attributions out of 272, because level N2 inserts a hesitation between the subject and the verb: "I er, will update the pricing". The models were right and our rule was wrong. The published version allows up to five words between the two.

What we would do with this

Do not let a model produce the follow-up list. Let it produce a draft. The recall is on its side, 94.8 percent, and the risk is on yours: a task nobody agreed to, in somebody's name.

Take the names off anything the quoted line does not support. It costs one script, it caught 34 out of 34 here, and an item with no owner is visible and gets assigned. An item with the wrong owner is invisible and gets ignored.

Read the cancellations yourself. This is the one thing no rule in this article covers. If a meeting called something off, that decision is the most fragile piece of information in the transcript, and it is the piece that produces work when it is lost.

Do not read an empty notes field as a clean run. gemini-2.5-flash-lite returned an empty notes field on 18 runs out of 32 and made 18 errors overall. 50 of the 57 flawed lists carried no relevant warning at all. A notes field is a comment, not a control.

If you want the broader version of this reasoning, we wrote about the tasks where AI fails without saying so and about the failure modes of automated workflows. This test is one measured instance of both.

Limits, stated plainly

• 8 meetings, 64 items, 96 runs. A test on 8 meetings is a test on 8 meetings.
• The corpus is built by script, not collected. It carries the kinds of mess we thought to invent, and a real transcript carries kinds we did not.
• Meetings of about 18 turns, entirely inside the model's context. An hour long transcript is a different problem this test does not touch.
• One run per combination, at temperature 0. Models are not perfectly deterministic even so. Gaps of a few points should not be read as significant. The gap between 2 and 20 missed actions, and above all the shape of the errors, always the same trap family, are.
• Three entry level models. Nothing here says what more expensive ones would do. That is an open question and it deserves its own test.
• English only, first names only, Latin script only. Meetings held in two languages, or with people who share a first name, are a known hard case this corpus does not contain.
• Attaching a returned line to an item by an anchor word is our rule. It attached 474 lines out of 474 with no ambiguity and no leftovers, which is the best evidence we have that it is not distorting anything, but it remains our rule.
• One judgement in this test is genuinely arguable: whether the person who raises an unowned task implicitly owns it. We counted that as an error and we report it separately, 29 of the 34, so that a reader who disagrees can subtract it and keep the other 5.

Rerun it

Everything this test produced is here, not summarised: the script that generated the 8 meetings and the one that degraded them, the ground truth with every anchor and every family, the 96 raw responses exactly as the models returned them, and the analysis that produces every figure above.

The generator uses a fixed seed, so the corpus you produce is the corpus we used.

python3 generer-corpus.py --graine 20260818 --sortie corpus.json
python3 conduire-test.py --corpus corpus.json --sortie resultats-bruts.json \
    --niveaux 1 2 3 4 \
    --modeles openai/gpt-4o-mini google/gemini-2.5-flash-lite anthropic/claude-haiku-4.5
python3 analyser.py --corpus corpus.json --resultats resultats-bruts.json \
    --sortie chiffres.json

The API key is read from an environment variable and appears nowhere in these files.

Download the dataset (zip, with the full inventory and the checksum). The scripts and the comments inside them are in French, which is the working language behind this site.

If you rerun this and get something different, we want to know. There is a contact form, and corrections get published on the article itself, dated, as the editorial rules of this site say.