Put LLM Agent Security in the Tool Layer, Not the Prompt
A system prompt lowers how often an agent misbehaves; it can’t stop a prompt injection. Where controls actually hold: authorization in every tool, limits once untrusted content is in context, and checks before consequential actions.
in AI Systems & Architecture6 min read
An agent’s system prompt says to touch only the current user’s records and never send data anywhere else. Then the agent reads a support ticket, a web page or a tool result that says otherwise, and which instruction wins is a matter of probability. This is for engineers building agents that call real APIs on real data, and it argues for putting the controls that must hold somewhere the model can’t talk its way past.
What a system prompt can and can’t do
OWASP’s entry on prompt injection is blunt about the state of the art: “it is unclear if there are fool-proof methods of prevention for prompt injection”, because of how generative models work (OWASP LLM01:2025 (opens in a new tab)). OpenAI’s agent safety guide reaches the same place from the vendor side: “even with these mitigations, agents won’t be perfect and can still make mistakes or be tricked” (OpenAI: Safety in building agents (opens in a new tab)).
That doesn’t make the prompt useless. A clear role, a defined task and explicit limits lower how often an agent does the wrong thing, and OWASP lists constraining model behaviour in the system prompt as its first mitigation. What the prompt can’t do is set a ceiling. It changes the distribution of what the model does; an attacker only needs the tail.
The distinction is the same one web developers already make between a disabled button and a server-side permission check. The button tells honest users what they’re meant to do. The check is what stops everyone else. A system prompt is the disabled button.
Treat every tool call as a request from an untrusted client
The useful shift is to stop thinking of the model as part of your backend. It’s closer to a client you don’t control: it sends requests (tool calls) with arguments it chose, and some of what shaped those choices came from text an attacker wrote. Each tool handler should then do what any API endpoint does with a request from a client, which is to decide in code whether this caller may do this thing.
OWASP’s entry on excessive agency names the two halves of that. Tools should run in the individual user’s context with the least privilege they need, and authorization belongs in the downstream systems rather than in the LLM’s judgment (OWASP LLM06:2025 (opens in a new tab)).
The failure this prevents is common and easy to miss in review. A tool takes userId or accountId as an argument, the model fills it in from context, and the handler trusts it. Nothing goes wrong until a document in that context says the relevant account is someone else’s. The fix is structural: the caller’s identity comes from the authenticated request your application already has, never from the model’s arguments. The model can choose which record to open; the handler decides whether this user may open it.
MCP’s security guidance applies the same rule to servers. A server “MUST NOT accept any tokens that were not explicitly issued for the MCP server”, it “MUST NOT treat possession of a state handle as authentication”, and the handle should be bound to the user derived from the verified token rather than supplied by the client. Its list of common mistakes includes “Treating claimed scopes in token as sufficient without server-side authorization logic” (MCP security best practices (opens in a new tab)).
Prompt injection is often compared to SQL injection, and the comparison holds for the cause: instructions and data share one channel. It breaks down at the fix. SQL got parameterised queries, which separate the two by construction. Natural language has no equivalent yet, and that’s exactly why the enforcement has to move out of the model and into the code around it.
Limit what the agent can do after it reads untrusted content
Authorization stops an agent from acting outside the user’s permissions. It doesn’t stop an injected agent from misusing the permissions the user legitimately has, such as forwarding the user’s own data to an address the attacker supplied. That needs a second control: limiting what an agent can do once untrusted text is in its context.
A 2025 paper by researchers from IBM, Google, Microsoft, ETH Zurich and others puts the principle precisely: “once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions” (Beurer-Kellner et al. (opens in a new tab)). The paper proposes six design patterns that achieve this by construction rather than detection, and it’s candid that detection-based defenses “remain fundamentally heuristic and cannot guarantee prevention of all attacks” (full text (opens in a new tab)). Four of the patterns cover most production agents:
|
Pattern |
How it contains injection |
What it costs |
|---|---|---|
|
Action-selector |
The model maps a request to predefined tool calls, and no tool output feeds back into it |
The agent can’t adapt to what a tool returns |
|
Plan-then-execute |
The plan is fixed before any untrusted data is read, so that data can’t add steps |
Tasks that need to change course mid-way don’t fit |
|
Dual LLM |
A quarantined model reads untrusted content and has no tools; a privileged model has tools and never sees that content |
Two models to orchestrate, and passing results between them takes care |
|
Context-minimization |
Content that’s no longer needed is removed from the context between steps |
The agent loses detail it may later want |
OpenAI’s guide makes a related point about the data passed between steps. Structured outputs such as “enums, fixed schemas, required field names” between nodes “eliminate freeform channels that attackers can exploit to smuggle instructions or data”, and untrusted input placed in developer messages “gives attackers the highest degree of control”. If a step only needs a category and an amount, extract a category and an amount, not a paragraph.
These patterns trade capability for safety, and the paper says so. A research agent whose job is to read the open web and decide what to do next can’t be fully plan-then-execute. When the pattern doesn’t fit, shrink what the agent can reach instead: read-only tools, no outbound send, nothing that touches other users. OWASP singles out open-ended tools such as shell commands and URL fetches as the ones to avoid; when an agent genuinely needs a shell, run it in a container with no secrets and a restricted network, as covered in Docker sandboxes for AI agents.
Put deterministic checks in front of consequential actions
Some actions should not depend on the model getting it right even once. Sending money, deleting data, emailing someone outside the organisation, changing permissions: these need a check that runs in code before the action, whatever the model decided.
Human approval is the obvious one, and OWASP recommends it for high-risk actions. OpenAI goes further for MCP tools and says to “always enable tool approvals so end users can review and confirm every operation, including reads and writes”. For tools you build yourself, approval fatigue is the practical limit: ask people to confirm every read and they stop reading the prompts. Reserve approval for actions that are hard to reverse or visible outside the user’s account, and make the approval show the actual arguments, not the model’s summary of them.
Below approval sit checks that need no human. Validate the model’s output against a schema and reject anything outside it. Cap amounts, recipients and record counts per action. OWASP’s damage-limitation advice for when prevention fails is to “log and monitor the activity of LLM extensions” and “implement rate-limiting to reduce the number of undesirable actions”, which turns a successful injection from an incident into a bounded one.
What still belongs in the prompt
Keep writing good system prompts. Mark external content clearly as data, separate from instructions, which OWASP lists as its own mitigation. State the task narrowly. Tell the model what to do when content asks it to do something else.
Then test it as an attack surface. Injection succeeds some fraction of the time, so a single passing run says little; measuring with pass^k applies just as well to attack success rates. And when you change models, rerun the attacks along with everything else, because a prompt that held on one model can fail on the next; pin your prompts to a regression suite covers the mechanics.
The rule to apply to every tool before it ships: write down what happens if the model calls it with the worst arguments an attacker could choose. If the answer depends on the model refusing, the control is in the wrong place. Move it into the tool.
Sources
- LLM01:2025 Prompt Injection (opens in a new tab), OWASP GenAI Security Project
- LLM06:2025 Excessive Agency (opens in a new tab), OWASP GenAI Security Project
- Design Patterns for Securing LLM Agents against Prompt Injections (opens in a new tab), Beurer-Kellner et al., 2025 (full text (opens in a new tab))
- Safety in building agents (opens in a new tab), OpenAI
- Security Best Practices (opens in a new tab), Model Context Protocol
