Grounded extraction QA

Every figure the AI reports, ticked back to the line that proves it.

Auditors trace each number in a working paper to its source and tick it. This tool does the same for AI that reads a practice's financial statements during an acquisition. A figure is accepted only when the line it cites really shows it. Anything else becomes an exception, even when the number happens to be right.

Working paper A-1

Prepared by
Checked by this tool

Figures reported, with the result of tracing each to its source. Select a row to see its source line.
ItemReported Tick Source
Turn on JavaScript to work through the schedule.
Source document A clean statement. Everything adds up.
NORTHGATE CLINICAL GROUP
Consolidated Balance Sheet
As at 31 December 2024
Amounts in US dollars

ASSETS
Cash and cash equivalents                   1,204,500
Trade and other receivables                 3,410,000
Total current assets                        4,614,500
Property, plant and equipment               8,890,000
Total assets                                13,504,500

LIABILITIES
Trade and other payables                    1,120,400
Long-term borrowings                        4,200,000
Total liabilities                           5,320,400

EQUITY
Share capital                               1,000,000
Retained earnings                           7,184,100
Total equity                                8,184,100

Total liabilities and equity                13,504,500

CONSOLIDATED STATEMENT OF OPERATIONS
Revenue                                     24,880,000
Cost of sales                               (14,920,000)
Gross profit                                9,960,000
Net income                                  2,415,000
  • Agreed to source. Right, and the cited line shows it.
  • No source given. Right, but a reviewer must find it again by hand.
  • Exception. Wrong, or the cited line does not back it up.
  • Not reported. Left blank.

Review summary

6 statements, 8 figures each: 48 figures per preparer. Only figures agreed to source are counted as usable, because they are the only ones a deal team can rely on without checking them again.

Prepared byAgreed to sourceNo source givenExceptionNot reported DistributionUsable
Current versionFinds each figure, applies the statement's units, and points to the exact line.4800048 of 48
Previous versionPoints to its sources, but confuses lines with similar names.20022620 of 48
First draftGrabs the number after a label and never says where it came from.5118245 of 48
Test: no sourcesBuilt to fail. Every figure correct, and not one source given.048000 of 48
Test: fabricatorBuilt to fail. Invents believable figures and cites the row label, so each one looks sourced.0036120 of 48

The "no sources" test gets every figure right and is still usable on none of them. Scoring correctness alone would have rated it perfect.

It also catches a statement that does not add up

Every figure on Northgate Clinical Group's statement was read and agreed to source. The statement still does not foot: liabilities plus equity should equal total assets, and here they fall short. A tool that only copied numbers would carry the gap straight into the deal summary.

Total liabilities5,320,400
Total equity8,084,100
Liabilities and equity13,404,500
Total assets13,504,500
Difference Exception(100,000)

Does not foot by 100,000. Flagged before it reaches the deal team.

No update ships if it makes a single figure worse

Every change to the reader, including a change to an AI prompt, is worked through the same statements before it ships. If any one figure gets worse, sign-off is refused, even when the overall score goes up.

Previous version to current version

Figures fixed
28
Figures made worse
0

Approved to ship

The same change, rolled backwards

Figures fixed
0
Figures made worse
28

Refused

The same discipline for the data room

A sample diligence database, seeded on purpose with one problem of each kind, so every check should find something. The test suite also proves each check finds nothing once its problem is removed, so a check cannot pass by always raising the alarm.

Planted problem Result
Figures with no document behind themFound 1 record
Records that point at things that do not existFound 2 records
The same file uploaded twice for one practiceFound 1 record
Checklist items with nothing filed against themFound 1 record
Totals from two sources that disagreeFound 1 record
A marketing asset carrying another brand's nameFound 1 record
Supporting schedules for engineers

Grounding means the cited span is sliced out of the document and re-read: a citation that does not contain its own value fails, and the offending text is recorded. Grounded recall is the share of true values returned correct and grounded, and it is the headline metric.

ExtractorRecallPrecisionCitingGroundingGrounded recall
rules:v133.3%66.7%25.0%25.0%10.4%
rules:v262.5%71.4%100.0%71.4%41.7%
rules:v3100.0%100.0%100.0%100.0%100.0%
adversarial:uncited100.0%100.0%0.0%0.0%0.0%
adversarial:fabricating0.0%0.0%100.0%0.0%0.0%

Release gate: rules:v2 to rules:v3, PASS

Metric BaselineCandidateDelta
micro_recall62.5%100.0%+37.5%
micro_precision71.4%100.0%+28.6%
micro_f166.7%100.0%+33.3%
grounding_rate71.4%100.0%+28.6%
grounded_recall41.7%100.0%+58.3%
citation_rate100.0%100.0%0.0%

The gate fails on any regressed cell, any drop in a headline metric, any change to the golden set itself, and any arithmetic check that stops reconciling.

Live model on this instance: deepseek-v4-pro via https://api.deepseek.com.

make install && make ci runs the linter, the test suite, every extractor, the regression gate and the SQL checks. make compare exits non-zero on any cell that got worse. The documents are synthetic and every figure is fictional; the golden set is generated by tools/make_golden_corpus.py so it cannot drift.