How SEC evidence changes model accuracy

Model proposes; program decides

Single-shot answers without tools. The agent retrieves SEC facts and cites evidence IDs; program rules independently parse the claim's numbers, precision and comparisons to determine the final verdict.

1. ProgramSplit assertionsCheck each numeric assertion separately.
2. ModelRetrieve SEC factsCall lookup_fact, compute_change, Receive numbered evidence E1, E2…
3. ProgramParse original claim“$143.1B”→ Value and precision ±$0.05B; “More than” implies an inequality; the program checks wording independently.
4. ProgramValidate evidenceRequire valid evidence IDs and check fiscal/calendar interpretations.
5. ProgramDetermine verdictReturn supported, contradicted or insufficient evidence, with filing strength and derivation labels.

Accuracy and 95% confidence intervals

Accuracy = Predictions matching reference labels ÷ 357; Failed and budget-stopped runs count as wrong. Black lines show Wilson 95% confidence intervals. Differences are paired on the same claims (10,000 bootstrap samples).

Which claim families are hardest?

Each cell shows correct / total. Families cover approximation, inequalities, growth, period and unit traps, multiple numbers, record claims, unavailable evidence and ambiguous fiscal periods.

Inspect the filing facts behind each verdict

Select a claim on the left. Numbers are highlightedin yellow, Periodin blue, Compare six systems and inspect the agent's claim values, SEC values, tolerances andthe decisive check. Evidence cards link to the original SEC filings.

Reference verdicts for news claims

Counts use reference labels, independently of verifier predictions. Sites with fewer than 20 claims are grouped by source type. Samples are small and intervals wide.

Methods and limitations