THINK AI FIRST

Dataset

Meeting action item extraction by three low cost LLMs

96 raw model responses, 8 script generated meetings with their ground truth, four transcript formats. Test conducted on 2026-08-18.

This page describes one dataset, published in full. It is the raw material behind our measurement of what lands on a meeting follow-up list that nobody agreed to: 8 meetings, four transcript formats, three low cost models, and the 96 responses those models actually returned.

Everything on this page comes from that one run. Nothing is estimated.

What it measures, and what it does not prove

It measures two things precisely. First, what a model adds to a follow-up list: work that was cancelled during the meeting, and owner names attached to items that named nobody. Second, how often it does so without anything that would send anyone to check.

An action list is an open list. Nobody hands the model the number of items to find, so a missed item and an invented item look identical in the output. That is what makes this different from field extraction, where the fields are known in advance.

What it does not prove, stated plainly, because a dataset quoted without its limits does more harm than no dataset at all:

  • The meetings run about 18 turns and fit entirely inside the model's context. An hour long transcript is a different problem, and none of it is measured here.
  • The corpus is built by script, not collected. No real meeting, no real person, no real company.
  • 8 meetings, 64 items. A test on 8 meetings is a test on 8 meetings.
  • One run per combination, at temperature 0. Gaps of a few points between models are not significant here. The shape of the errors is what holds.
  • English only, first names only, Latin script only. Meetings held in two languages, or with two people sharing a first name, are a known hard case this corpus does not contain.
  • Three entry level models. Nothing here says what more expensive ones would do.

The corpus is built, and here is what that means

The 8 meetings were generated by script with the fixed seed 20260818. That is a deliberate trade, and it cuts both ways.

What it buys: the ground truth of an action list is a list, and it is only beyond argument if we are the ones who set it. Anyone can regenerate the exact same corpus and disagree with our reading of it. What it costs: these are not real meetings, and a real transcript carries kinds of mess we did not think to invent.

Each meeting holds 3 actions with a named owner, 2 actions nobody was named for, 1 action assigned and then cancelled later in the same meeting, 1 proposal refused on the spot, and 1 task announced as already done. Across the 8 meetings: 40 real actions and 24 traps.

Every item carries an anchor, a concrete word unique in the whole corpus, present in the line that carries the item and nowhere else. The anchor is what attaches a returned line to an item, mechanically. Four invariants are checked before a single call is made: the anchors are unique, each is present in the served text, none appears in a line that does not carry it, and no participant name appears in the lines of actions nobody was named for.

Test conducted on 2026-08-18. This page was published on 2026-08-18. The extractions were run by the automated testing pipeline of this site: the corpus, the model calls and the analysis are scripts, and all of them are in the archive below.

Raw figures

Actions found, and what came back with them

Model Actions found Owner invented Cancelled work reissued Errors of which silent
openai/gpt-4o-mini 140/160 12 9/32 51 49
google/gemini-2.5-flash-lite 158/160 8 8/32 18 15
anthropic/claude-haiku-4.5 157/160 14 2/32 19 12
All models 455/480 34 19/96 88 76

By transcript format

Format Actions found Owner invented Cancelled work reissued Errors
N1 clean notes 112/120 7 2/24 18
N2 case and hesitations 115/120 9 4/24 22
N3 running paragraph 115/120 11 8/24 26
N4 raw transcript 113/120 7 5/24 22

The three traps

Trap What it looks like in the notes Returned as an action
Assigned, then cancelled later one turn gives the task, a later turn calls it off 19/96
Proposal refused on the spot "Should we ...?" answered "No, not this quarter" 0/96
Task announced as already done "I already did that last week, that one is closed" 0/96

The two traps that fit in a single line are never mistaken. The one that spans two distant turns is missed by nearly one list in five, and 17 of those 19 phantom actions came with a named owner.

Ownership

On the 273 found actions that named an owner Count
Owner correct 272
Owner wrong participant 0
Owner missing 1
On the 182 found actions that named nobody Count
Correctly left empty 147
Set to the person who said the line 29
Set to somebody else entirely 5
Set to a collective, "the team" 1

Deadlines

State Count
Correct 207
Wrong 10
Dropped when one existed 15
Invented when none existed 0

Quotations

Each returned item had to quote the line of the notes that supports it. The quotation is compared turn by turn against the text actually served at that level.

State Count
Verbatim inside one turn 459
Stitched across turns from real fragments 15, of which 14 at N3
Containing words nobody said 0

Comparing against the whole served text instead of turn by turn moves 15 quotations into the verbatim column, because normalisation erases the boundary between speakers. The figures above use the turn by turn rule. Both are computable from the archive.

The anchored owner rule

Keep a name only if the quoted line supports it: either the line names the person, or it is a first person commitment by that speaker.

Quantity Value
Invented owners blocked 34 of 34
Correct owners wrongly blocked 20 of 272, 7.4 percent
Cancelled work blocked 3 of 19

The rule is about attribution, not about cancellation, and the last line says so. We publish no cancellation detector: our cancellations all follow one phrasing, so any keyword detector would be scoring itself against the template that wrote them.

Run totals

Quantity Value
Extractions 96
Action lines returned 474
Lines attached to no item, or to two 0 and 0
Technical failures 0
Invalid JSON responses 0
Total API cost 0.091618 USD
Lists carrying at least one error 57 of 96
of which nothing warns about it 50

What the archive contains

Seven files, 743151 bytes uncompressed. The inventory is here so you know what you are downloading before you download it.

File What it holds Size
generer-corpus.py the corpus generator, fixed seed 20260818, and its four invariants 16.9 KiB
corpus.json the 8 meetings, their ground truth, every anchor and every family 33.0 KiB
degrader.py the four transcript formats, and the turn by turn view used to grade quotations 4.8 KiB
conduire-test.py the model calls, with the prompt in clear 6.4 KiB
resultats-bruts.json the 96 responses, raw, with the prompt used 269.4 KiB
analyser.py the classification and the computation of every figure on this page 18.7 KiB
chiffres.json the computed figures, plus one line per item judged and per line returned 376.5 KiB

The scripts and their comments are in French, which is the working language behind this site. The API key is read from an environment variable and appears in none of these files.

Rerun it

The generator uses a fixed seed, so the corpus you produce is the corpus we used.

bash
unzip 2026-08-meeting-action-items.zip
cd 2026-08-meeting-action-items
python3 generer-corpus.py --graine 20260818 --sortie corpus.json
python3 conduire-test.py --corpus corpus.json --sortie resultats-bruts.json \
    --niveaux 1 2 3 4 \
    --modeles openai/gpt-4o-mini google/gemini-2.5-flash-lite anthropic/claude-haiku-4.5
python3 analyser.py --corpus corpus.json --resultats resultats-bruts.json \
    --sortie chiffres.json

Each raw response is one entry of resultats-bruts.json, exactly as the model returned it, before any classification:

json
{"modele": "openai/gpt-4o-mini", "reunion": "M1", "niveau": 4,
 "json_valide": true,
 "reponse": {"actions": [
   {"task": "Update the pricing on the kiosk", "owner": "rafael",
    "due": "by wednesday",
    "evidence": "rafael i er will update the pricing on the kiosk by wednesday"},
   {"task": "Call the sign shop about the awning", "owner": "nadia",
    "due": null, "evidence": "nadia sure um that is mine"}],
  "notes": "I was unsure about who is responsible for emptying the mailbox and rotating the gate keypad code, as the notes did not specify an owner for these tasks."}}

Verify what you downloaded before you trust it:

bash
shasum -a 256 2026-08-meeting-action-items.zip
# f90dc30daf2059f37824912ca53ab9fe356345fb1fe201d3926c0544f4da2ee8

How to cite this dataset

Copy one of the two forms below. The reference points at this page, which is the stable address of this dataset; the archive is what that address serves.

citation
Think AI First (2026). Meeting action item extraction by three low cost LLMs, 96 raw responses.
Think AI First. Test conducted 2026-08-18, published 2026-08-18.
https://think-ai-first.com/datasets/meeting-action-items-2026-08. Licensed CC BY 4.0.
bibtex
@dataset{thinkaifirst2026actions,
  author    = {{Think AI First}},
  title     = {Meeting action item extraction by three low cost LLMs, 96 raw responses},
  year      = {2026},
  publisher = {Think AI First},
  url       = {https://think-ai-first.com/datasets/meeting-action-items-2026-08},
  note      = {Test conducted 2026-08-18. Licensed CC BY 4.0. Archive SHA-256 f90dc30daf2059f37824912ca53ab9fe356345fb1fe201d3926c0544f4da2ee8}
}

Licence and reuse

Published under CC BY 4.0. Use it, quote it, contradict it. If you rerun this and get something different, tell us through the contact form: corrections are published on the article itself, dated.

Download the dataset (zip, 59.6 KiB, 7 files)

Other datasets published here

Each one is a separate test, with its own corpus, its own raw responses and its own archive.