FinAgent
Multi-agent financial research with an evaluation harness
100% numerical accuracy on the 14 checkable answers of a 22-question evaluation, zero runtime errors
The problem
A financial research question usually needs several sources, some arithmetic and the judgement to say "not enough yet". Most agent loops hide that control flow and declare success early, which makes their answers hard to trust and impossible to grade.
What I built
An explicit state graph with four nodes. The planner reads the question and everything gathered so far and emits a typed decision: call one tool, with arguments and a stated reason, or finish. The tool executor runs it and records the call; failures are captured, never raised. The reflect node is deliberately sceptical and sends the planner back when information is missing, with a hard iteration cap. The finalize node forces the answer through a Pydantic schema with key numbers, citations and a confidence that drops on any tool error or forced stop.
| Tool | What it does |
|---|---|
| sec_edgar | Official figures from the SEC XBRL API, with concept aliasing and an on-disk fixture cache for reproducible runs |
| calculator | Arithmetic through a sandboxed AST evaluator with an operator whitelist; no eval() |
| web_search | News and context outside filings, with a stub for offline runs |
| rag_retrieve | Passages from indexed 10-K PDFs in ChromaDB, returned with citations |
- A provider abstraction (Anthropic, OpenAI, or a deterministic mock) so the whole graph, including the reflection loop, runs in CI with no API keys
- A FastAPI endpoint and a CLI
- 30 offline unit and graph tests; CI runs lint, tests and the offline evaluation on every push
Results
The evaluation set has 22 questions: single-tool lookups, multi-tool calculations, cross-company comparisons, RAG, web, and one deliberately ambiguous item. The run below used Claude.
| Metric | Value |
|---|---|
| Numerical accuracy, within tolerance | 100% (14 of 14 checkable) |
| Answer-content checks passed | 90% (n = 10) |
| Reflection loop triggered | 54.5% (12 of 22) |
| Reflection recall on questions designed to need it | 100% (n = 5) |
| Tool selection, exact set match | 36.4% |
| Tool selection, mean Jaccard | 0.674 |
| Planner iterations per question, average | 1.14 |
| Confidence distribution | 14 high, 3 medium, 5 low |
| Runtime errors | 0 |
On the ambiguous question, which of two companies is more profitable and by how much, the reflect node flagged that profitability needs a margin comparison and that one company's figures were missing, looped back four times across five SEC look-ups, answered with both readings, and reported low confidence because it had hit its step budget.
Limitations
- Exact tool-set match is moderate at 36%: on easy derived values the model sometimes computes inline instead of calling the calculator, and RAG questions fall back to other tools when retrieval is unavailable. Mean Jaccard of 0.67 shows the primary tool is almost always right
- The ticker-to-CIK map is static and covers five companies, enough for the evaluation, not for production
- RAG runs locally only; in the sandboxed CI the embedding endpoints are network-gated
- 22 questions is a smoke-test-sized set. It checks routing, arithmetic and the reflection loop, not long-horizon research
What this means for your project
The harness is the part that transfers. For your assistant we write the question set and the pass criteria first, then build until it passes, and you keep the harness to re-run after every change.