Evaluation
The full pipeline run on every golden-set filing, each extracted figure graded against the SEC's XBRL value for that exact filing and period. Model gpt-6-luna@none, prompt v4, run 2026-09-28.
figure accuracy
99.8%
544 of 545 exact, sign included
cost per filing
$0.0006
$0.03 for all 54
time per filing
2.7s
three agents in parallel
misread figures
0
0 of 54 filings retried once
by field
| Total revenue | 50/51 | |
| Operating income | 33/33 | |
| Net income attributable to the company | 53/53 | |
| Diluted EPS | 54/54 | |
| Cash and cash equivalents | 53/53 | |
| Total assets | 54/54 | |
| Total liabilities | 37/37 | |
| Total shareholders' equity attributable to the company | 48/48 | |
| Net cash from operating activities | 54/54 | |
| Net cash from investing activities | 54/54 | |
| Net cash from financing activities | 54/54 |
by sector
| Technology | 107/107 | |
| Financials | 100/100 | |
| Healthcare | 98/98 | |
| Consumer | 103/103 | |
| Energy | 19/19 | |
| Industrials | 80/81 | |
| Telecom & Utilities | 37/37 |
reconciliation checks, after retries
| assets = liabilities + equity | 54/54 |
| net income before preferred dividends | 54/54 |
| diluted EPS is for total net income | 54/54 |
| cash flows tie to the change in cash | 39/44 |
| equity excludes minority interests | 31/31 |
| net income excludes minority interests | 28/28 |
The cash flow tie breaks on legitimate extra lines (discontinued operations, restricted cash), so it's reported but never triggers a retry.
restatement detection
For each filing the pipeline also reads last year's 10-K, and code compares last year's figures as reprinted now against as first reported. Graded against the SEC's XBRL for both filings.
precision
100.0%
12 of 12 flags were real
recall
100.0%
12 of 12 revisions caught
false alarms
0
of 527 unrevised figures
amounts exact
12/12
both sides of each caught revision
- BAC accounting change: The Corporation retrospectively changed accounting methods for certain tax-related equity investments. The change reclassifies income-statement items, including netting tax credits and benefits against investment expense, which explains the revised revenue presentation. quote verified
- BAC accounting change: The Corporation retrospectively changed accounting methods for certain tax-related equity investments, including a cumulative adjustment to retained earnings. The report says the changes had an insignificant impact on annualized net income, but does not specify how they account for each of these listed revisions. quote verified
- C unexplained: The passages do not explain why the reprinted total revenue is lower than originally reported. no verified quote
- USB unexplained: The passages do not explain the changes to these cash flow figures. no verified quote
- LLY unexplained: The passage discusses a future disclosure standard and does not explain the change in shareholders’ equity. no verified quote
- HON discontinued operations: The Advanced Materials business was spun off, and its results are presented as discontinued operations for all periods. Removing those results from the reprinted comparative figures explains the lower revenue. quote verified
- HON discontinued operations: The Advanced Materials business was spun off, and its financial position is presented as discontinued operations for all periods. Removing its financial position from the reprinted comparative figures explains the lower cash balance. quote verified
models, same pipeline and checks
| model | accuracy | cost / filing | time / filing | retried |
|---|---|---|---|---|
| gpt-6-luna@none | 99.8% 544/545 | $0.0006 | 2.7s | 0/54 |
| gpt-6-luna | 99.8% 544/545 | $0.0007 | 3.4s | 0/54 |
| gpt-4o-mini | 99.3% 541/545 | $0.0010 | 3.2s | 16/54 |
| gpt-6-sol | 99.8% 544/545 | $0.0128 | 4.1s | 0/54 |
Once code does everything except choosing the row, a model about 20 times the price doesn't choose rows any better. Filings made after a model's training cutoff score as well as those before it, so the figures are being read, not recalled. Full table with the cutoff split: eval/models.md.
management's claims vs the statements
claims found
348
333 quotes verified in the filing
checkable
90
company-wide GAAP figures only
consistent
100.0%
90 of 90
flagged
0
the checker catches 966 of 968 planted false claims
every miss
| filing | field | what happened | SEC | got | row picked / correct row |
|---|---|---|---|---|---|
| DE | Total revenue | wrong row | $45.7B | $38.9B | Net sales / Total |