A bank’s support assistant reads a customer’s uploaded statement. A wealth app summarizes a company’s annual report. An accounts-payable tool extracts line items from supplier invoices. Each of these products feeds text written by someone other than the user into a language model, and each can be steered by instructions hidden in that text. That is prompt injection, and the security references from OWASP and NIST agree on an uncomfortable point: there is no reliable way to filter it out.

The OWASP Top 10 for LLM Applications 2025 ranks prompt injection first, as LLM01. It says that because of the stochastic way models work, “it is unclear if there are fool-proof methods of prevention.” NIST’s AI 100-2 E2025, its taxonomy of adversarial machine learning published March 24, 2025, reaches the same view: current mitigations “do not offer full protection against all attacker techniques.” The practical question for a team handling financial data is therefore which defenses limit the damage when an injection gets through, not which filter stops it.

Direct and indirect injection

OWASP separates two types. A direct injection comes from the user’s own prompt, deliberately or by accident. An indirect injection arrives through content the model reads from outside: a web page, a file, an email, a retrieved document.

NIST explains why indirect injection works. Generative models “combine the data and instruction channels,” so anyone who controls a resource the system reads can, in effect, write instructions into it. The attacker is a third party, and NIST notes that the person harmed is often the application’s own user.

For financial products the indirect route is the bigger exposure. Statements, invoices, filings, support tickets and emails are untrusted input by nature, and they are exactly what these tools are built to read.

What an injection can do

NIST groups indirect injection outcomes into three attacker goals.

  • Integrity. Researchers have shown injections that produce wrong summaries, insert attacker-chosen information or hide sources. NIST also lists knowledge-base poisoning, where one crafted document in a retrieval system steers answers to particular questions.
  • Privacy. A model can be told to leak data it can reach. NIST cites demonstrations of models forwarding emails to an attacker and of exfiltration through a request to an attacker-controlled URL, including through markdown image rendering.
  • Availability. An injected instruction can make the model loop on a slow task, refuse to use a tool or corrupt its output format.

OWASP’s list of impacts adds unauthorized access to functions available to the model, arbitrary commands in connected systems and manipulation of decision-making. Its example scenarios include a support chatbot told to query private data stores and send emails, and a modified document in a retrieval index that changes answers for everyone who retrieves it.

NIST adds that hiding is routine: injections placed in invisible parts of a page, split across stages or encoded in Base64. OWASP describes the same with multiple languages and emojis. Keyword filters cannot keep up with that.

Defenses that hold, and defenses that only help

OWASP lists seven mitigations. Read against the NIST findings, they fall into two groups.

Controls that keep working when the model is fooled:

  1. Least privilege in code. OWASP says the application should hold its own API tokens and handle functions “in code rather than providing them to the model,” with the model’s access limited to the minimum its task needs. A summarizer that has no tool to move money, export a customer list or send email cannot be talked into doing any of those things.
  2. Human approval for high-risk actions. OWASP recommends human-in-the-loop controls for privileged operations. In finance that means a person, or a deterministic rule, approves any payment, limit change or data export the model proposes. Our explainer on AI agent payment permissions covers how that approval step can be scoped.
  3. Deterministic output validation. OWASP says to specify the expected output format and use “deterministic code” to check that the model followed it. If an extraction step must return an invoice number, an amount and a date, ordinary code can reject anything else, including URLs or instructions the model was tricked into adding.
  4. Architectural separation. NIST describes designs that assume injection is possible whenever a model sees untrusted input: several models with different permissions, or untrusted data reached only through narrow, well-defined interfaces. A model that reads the document should not be the model that holds the tools.

Measures that lower the odds but can be bypassed:

  • System-prompt instructions to stay on task and ignore attempts to change the rules.
  • Input and output filtering, which OWASP pairs with semantic checks for relevance and groundedness.
  • Marking untrusted content so the model can tell it apart, which NIST calls spotlighting.
  • Model-level training against injection, and separate detection classifiers.

NIST lists all of these among its mitigations and still concludes they are incomplete. They are worth having as layers, but a design that depends on them alone is betting that the attacker never finds a phrasing the filter missed.

Testing

OWASP’s seventh measure is regular adversarial testing that treats the model “as an untrusted user” to probe trust boundaries and access controls. NIST names public tools for this, including AgentDojo, a framework that measures how often an agent is hijacked by data returned from its tools. A test plan for a financial assistant should include poisoned documents in the retrieval index, invoices with hidden text, and attempts to make the model emit links or images that carry customer data out.

The regulatory frame

In New York, the Department of Financial Services’ October 16, 2024 guidance on AI cybersecurity risks says covered entities should address, in their Part 500 risk assessments, vulnerabilities from AI applications that could threaten the confidentiality, integrity and availability of their systems or nonpublic information. It adds no new requirements beyond Part 500. Our explainer on NYDFS AI cybersecurity guidance covers that letter and the ones that followed. For retrieval systems in particular, our analysis of RAG for financial documents describes the accuracy limits that sit alongside the security ones.

Sources