15. Worked example: threat-modeling an AI agent

This part applies everything above to one realistic system, using Shostack’s four questions from Part 1. The verdict comes first: the biggest risk is an outsider’s email steering the assistant into sending confidential documents out, and the fix is architectural - break the lethal trifecta on the send path - not a better prompt.

1. What are we working on?

A company knowledge assistant. Employees chat with it in a web app. It answers from the internal wiki and shared drive (RAG), reads the user’s email and calendar, drafts and sends emails, and creates support tickets. It calls a third-party LLM API and logs every conversation.

The diagram shows the components and the trust boundaries (TB1–TB6) that matter. Note where untrusted content enters: incoming email from anyone on the internet, and wiki pages that many people can edit.

[embedded content: knowledge assistant · 6 trust boundaries]

Untrusted text crosses TB3 into the agent’s context; the register below traces where it could go next.

2 and 3. What can go wrong, and what do we do about it?

Walking each boundary with STRIDE (Part 1), the OWASP lists (Parts 10–11) and the lethal trifecta gives this register, ordered by risk:

#	Threat	Entry point	Property	Main mitigations
1	An external email contains hidden instructions; the agent emails confidential files to the attacker	Incoming email (TB3) → send tool (TB4)	Confidentiality	Break the trifecta on sending: external recipients need human approval showing the raw email; recipients must come from the user’s own request, never from retrieved text (a data-flow policy)
2	Injected text makes the chat UI load an image whose URL carries private data	Any retrieved content → UI (TB5)	Confidentiality	No external image loading; sanitize markdown; strict content security policy
3	Retrieval returns HR or finance documents to employees who may not see them	Vector index (TB6)	Confidentiality	Carry document permissions into chunk metadata and filter by the user’s identity at query time; separate namespaces per department; permission tests in CI
4	A tampered wiki page states that supplier bank details have changed	Wiki → index (TB3)	Integrity	Show sources with every answer; restrict editing of pages used for policy answers; flag recently edited sources
5	OAuth scopes allow full mailbox access, send-as and delete, magnifying any injection	Connector tokens	Elevation of privilege	Minimum scopes (read, create drafts); short-lived per-user delegated tokens; sending as a separate, gated capability
6	Prompts and documents are retained by the model provider or used for training	Backend → LLM API (TB2)	Privacy	Data processing agreement, EU region, limited retention; DPIA; send only the chunks needed
7	A leaked API key or prompt flooding runs up the bill	Keys; chat endpoint (TB1)	Availability / cost	Secrets manager; per-user token budgets; spend alerts; server-side generation limits
8	Conversation logs full of sensitive data are readable by many engineers	Log store	Confidentiality	Access control on logs; redaction; retention limits
9	Injected content creates hundreds of junk tickets	Ticket tool (TB4)	Availability	Rate limits and deduplication on tool calls; circuit breaker on agent loops
10	Nobody can tell which email the agent sent, or why	Tool layer	Repudiation	Tamper-evident log of each tool call with the context that triggered it

The pattern in the mitigations is the lesson of the whole guide. Almost none of them depend on the model resisting injection. They limit what an injected model could do: deterministic policies on data flow, narrow credentials, safe sinks, and human approval exactly where an action is irreversible and crosses the company boundary.

4. Did we do a good job?

• Red-team plan. Build an indirect-injection suite: malicious emails and wiki pages aimed at threats 1–4 and 9. Run each case many times, and measure the attack success rate before and after mitigations.
• Permission tests. Automated queries from users in each department try to retrieve documents from the others.
• Canary tokens in restricted documents, alerting if they ever appear in outputs or outbound traffic.
• Exfiltration tests of the rendering path with image and link payloads.
• Regression. All of the above runs on every change to the model, prompts, tools or index.
• Residual risk, accepted explicitly. Injection can still distort summaries and drafts (integrity). Users review drafts before sending, and a named owner signs off on that remaining risk.

What to take from the example

• Start from assets and boundaries, not from attack names.
• Ask, for every boundary, “if the model obeyed an attacker here, what could it do?”
• Prefer removing a capability over detecting misuse of it.
• Write down residual risk and who accepted it - this is what makes the exercise real.