Retrieval-augmented generation, or RAG, is the usual way teams point a language model at filings, earnings releases and policy statements: search a document store for passages that look relevant, put them in the prompt, and ask the model to answer from them. RAG for financial documents is attractive because the source text sits next to the answer. The published research is less reassuring about how often that answer is right, and the ways it goes wrong are specific enough to test for.

What the benchmarks found

The term comes from a 2020 paper by Patrick Lewis and colleagues, which paired a pretrained generator with a neural retriever over a dense index of Wikipedia. The authors reported state-of-the-art results on three open-domain question-answering tasks and more specific, factual language than a model answering from its parameters alone (Lewis et al.). Financial filings have proved a harder corpus than Wikipedia.

FinanceBench, released in November 2023, assembled 10,231 questions about publicly traded companies, each with an answer and an evidence string. Its authors called the questions clear-cut, intended as a minimum performance standard. They tested 16 model configurations on a 150-question sample and manually reviewed 2,400 answers. GPT-4-Turbo used with a retrieval system answered incorrectly or refused 81% of the questions. Feeding the relevant evidence through a longer context window helped, but the authors judged that unrealistic for enterprise use because of latency and because it cannot cover larger documents (Islam et al.). Those figures describe the models of late 2023, not today’s; what has aged less is the list of failure types.

DocFinQA, published in January 2024, took 7,437 questions from the earlier FinQA dataset and attached the full source document to each, raising the average context from under 700 words to 123,000. Retrieval pipelines and long-context models both found it hard, and the longest documents were the hardest (Reddy et al.).

Where the pipeline breaks

Tables. Financial reports mix text and tables, and many answers need arithmetic across both. TAT-QA, built from real financial reports, measured its best model at 58.0% F1 against 90.8% for expert humans (Zhu et al.). The FinQA authors, whose questions were written by financial experts, found that large pretrained models fell far short of experts at multi-step numerical reasoning (Chen et al.). A chunker that separates a table from its header row, or a figure from the footnote that defines it, hands the model numbers without their meaning.

Position. Adding more passages to the prompt is not a cure. The “Lost in the Middle” study found that performance was often highest when the relevant information sat at the beginning or end of the context and degraded significantly when it sat in the middle, even for models built for long contexts (Liu et al.).

Dates and versions. A financial corpus is full of documents that look alike: quarterly reports with the same section headings, an amended filing beside the original, a policy statement and the minutes of the same meeting. Similarity search ranks passages by wording, not by which document is authoritative for the date in the question. The Federal Reserve, for example, releases the minutes of a regularly scheduled meeting three weeks after the policy decision (Federal Reserve), so “what did the Committee decide” and “what did members discuss” point to different documents published on different days. Our explainer on the FOMC statement vs minutes sets out what each one contains.

Confident wrong answers. NIST’s Generative AI Profile calls this confabulation: erroneous or false content presented with confidence, which can include confabulated logic or citations that purport to justify the answer (NIST AI 600-1). In a RAG system, a citation that points to a real passage does not prove the passage supports the sentence attached to it.

Checks that catch these failures

None of these failures is exotic, and each has a test that can run on every answer:

  1. Keep tables whole. Parse tables into rows that carry their headers, units and reporting period, and index them separately from narrative text.
  2. Filter before you search. Store company, document type, fiscal period, filing date and amendment status with every chunk, and restrict retrieval to the period and document the question names.
  3. Recompute figures in code. When an answer derives a number from others, such as a margin or a year-over-year change, redo the arithmetic from the retrieved cells instead of trusting the model’s arithmetic.
  4. Match each citation to its sentence. Confirm that every cited passage actually contains the figure or claim it is attached to; for numbers, an exact match is a cheap first gate.
  5. Make “not found” an acceptable answer. A refusal is recoverable; a plausible wrong number in a report is not.
  6. Build your own test set. Follow FinanceBench’s design with questions, answers and evidence strings drawn from your own documents, and rerun it whenever the chunker, embedding model, prompt or model changes.

What bank supervisors have said

US bank model risk guidance was rewritten this year. On April 17, 2026, the Federal Reserve, the OCC and the FDIC issued revised guidance on model risk management that supersedes SR 11-7; the Fed’s letter says it is most relevant to banking organizations with more than $30 billion in assets (SR 26-2). The guidance places generative and agentic AI models outside its scope, while saying that a bank’s own risk management and governance practices should guide controls for tools it does not cover (revised guidance).

Two of its principles still translate well to a RAG system: “effective challenge,” meaning critical analysis by objective experts across the model lifecycle, and outcomes analysis, which compares model outputs with real-world outcomes. For a question-answering system over filings, the real-world outcome is the figure in the filing itself. New York-regulated firms should also read our guide to NYDFS AI cybersecurity guidance, and teams whose assistant calls tools rather than only reading documents can start with LLM function calling explained.

Sources

Sources reviewed October 12, 2026.