How to Penetration Test an AI Agent

Most of the AI pentests I've seen are a prompt-injection checklist run against a chat window. Somebody fires fifty jailbreaks at the model, counts how many get blocked, and writes up a "penetration test."

That tests the demo. It does not test the agent.

An agent is not a model. An agent is a system: a model wired to tools, with memory, running inside an orchestration loop, acting under some identity, usually with credentials. The model is the part you talk to. The tools are the part that does damage. Every AI pentest that stops at the chatbox is testing the lock on the front door while the loading dock stands open.

Start with what it can do, not what it says

Conventional pentest methodology starts with recon, and that instinct is right. But recon on an agent looks different from recon on a network. You are not mapping hosts. You are mapping capabilities.

Before a single adversarial prompt, I want the capability inventory: every tool the agent can call, every API it can reach, every credential it holds, every system it can write to. Tool manifests, MCP server listings, function schemas, the permission scopes on its service accounts. This is the attack surface. Everything else is commentary.

This is also where most engagements go wrong, and it is a measurement problem. The industry wants a single number, so it measures prompt-injection block rate: we threw 200 attacks, 187 were refused, 93.5% secure. That number measures the model's manners. It tells you nothing about what happens when attack 188 lands and the agent has a database write tool with no confirmation step.

The wrong unit of measure is refusal rate. The right one is blast radius per compromised component. If I turn the planner, what can it touch? If I poison one tool's output, where does the lie travel? Measure that and the jailbreak scoreboard becomes what it is: a hygiene metric, not a security verdict.

The five layers

I work the agent as five layers. Each one gets tested at the class level, because individual payloads are infinite and classes are not.

1. The model layer. This is the prompt-injection checklist everyone already runs: direct injection, indirect injection through tool output and retrieved documents, multi-turn manipulation, instruction smuggling in file contents. Run it. It is table stakes, not the engagement. Note what the model refuses and what it merely rephrases, because rephrasal is where the interesting failures live.

2. The tool layer. This is where agents actually break. Tool poisoning, where a malicious or compromised tool description steers the agent. Permission overreach, where the agent holds scopes it never needs for its stated job. Missing confirmation gates on destructive or exfiltrating actions. I built a deliberately vulnerable MCP server once, as a false-negative oracle for my own scanner: a tool server guaranteed to be findable so I could prove the scanner missed nothing. Then I pointed it at the internet. The telemetry on what people actually try against agent tool servers looks nothing like the threat model in the vendor decks. Tool servers are the new open S3 bucket. Treat them that way.

3. The memory layer. Agents that persist state across sessions can be poisoned across sessions. A fact planted in long-term memory in March becomes a trusted premise in June. Test whether memory writes are validated, whether the agent distinguishes user-stated facts from tool-returned facts in its store, and whether there is any review path for what got remembered. Most implementations I have seen have none.

4. The orchestration layer. The planner, the delegation chain, the loop itself. Can a sub-agent be convinced its parent authorized something the parent never said? I once built a multi-agent workforce over eleven days, got a three-hop delegation chain working, and then shelved the whole thing with a post-mortem on exactly where autonomous delegation breaks. Delegation boundaries are where agents fail in ways no single-model test will ever catch. Test the handoffs.

5. The identity layer. What the agent authenticates as, and whether downstream systems can tell the agent apart from the user who invoked it. The confused deputy problem is forty years old and agents reinvent it weekly: a tool that trusts "the agent asked" the way old systems trusted "the request came from localhost." If the agent's actions are logged under the user's identity with no agent attribution, your IR team will have a bad quarter.

Chain it or it didn't happen

A finding in one layer is a vulnerability. A path across three layers is a pentest. The deliverable that matters is the chain: poisoned tool output becomes a false premise in memory, which the planner acts on two sessions later using an over-scoped credential, with the whole thing attributed to the user who clicked first.

Single-layer findings get patched as bugs. Chains get treated as architectural problems, which is what they are. If your engagement report reads like a list of blocked jailbreaks, you ran an eval. If it reads like an attack path, you ran a pentest.

Rules of engagement are engineering controls

Testing agents that can take real actions needs real guardrails, and I treat them as engineering, not paperwork. Depth gating: the engagement plan declares how many layers deep the test goes, and the harness enforces it. Scan pacing: rate limits on tool invocation so a runaway loop cannot burn the client's API budget or trigger their fraud controls. A single audited chokepoint for privileged actions, so nothing irreversible happens without a human in a defined seat. The guardrail that is not executable is not a guardrail. Write it in code or do not claim it.

What the scanners are good for

The automated side deserves its concession. LLM-as-judge evals and agent security scanners have a real job: regression testing at scale. Once a human has mapped the capability inventory and found the chains, automation is how you make sure next month's model update did not reopen them. Fast, cheap, repeatable. That is a genuine contribution.

The sin is selling the eval as the pentest. A scanner tests the model against known classes. A pentest maps the system against unknown ones. One of those finds the novel chain. It is not the scanner.

How I'd know I'm wrong

Here is the falsification turn, because a methodology without one is a sales pitch. This approach is wrong if agent architectures converge on a standard, sandboxed tool layer where capability inventories are trivially enumerable and permission models are actually enforced. If the ecosystem grows up and every tool call goes through a real authorization check with a real identity, then layer-by-layer testing collapses into configuration review and I will be the first to say so.

I do not see that ecosystem. I see a gold rush of tool servers with README-grade security and agents holding credentials they cannot justify. Until that changes, test the system.

The checklist version

For the buyer reading this before an engagement, here is what to demand:

1. A capability inventory before testing starts. No inventory, no engagement.
2. Findings reported as chains across layers, not as blocked-prompt counts.
3. Written ROE with depth gating, pacing, and a privileged-action chokepoint.
4. Cover the memory and delegation layers alongside the chat interface.
5. Identity attribution: whose credentials acted, and can IR tell?

If the vendor's sample report leads with a jailbreak scoreboard, you are buying an eval. Price it like one.

An agent is a system with a chat interface on top. Pentest the system. The chatbot will survive the experience. Whether the infrastructure around it does is the actual question.