Cybersecurity for AI Systems from First PrinciplesPost 5 of 19 · 19 of 19 written
- How to use this guide
- 01What security actually is
- 02How computers fail
- 03Networks
- 04Cryptography
- 05Identity and access
- 06Web and application security
- 07Cloud, infrastructure, supply chain, detection
- 08The bridge: why AI systems are different
- 09Adversarial machine learning
- 10LLM and generative AI security
- 11Agentic systems
- 12The ML supply chain and MLOps
- 13Privacy, governance and regulation
- 14Red teaming, evaluation, and AI for defense
- 15Worked example: threat-modeling an AI agent
- 16The question toolkit
- 17Learning path, practice, reading list
- 18Glossary
- 19Sources
5. Identity and access
Every request raises two separate questions: who are you (authentication), and are you allowed to do this (authorization)? Most serious breaches are failures of the second question, and AI agents make it harder than ever.
Authentication
The classic factors are something you know (password), something you have (phone, hardware key), and something you are (biometric). Multi-factor authentication (MFA) combines them. But not all MFA is equal.
Phishing resistance is the property that matters. SMS codes and authenticator-app codes can be phished: a fake login page asks for the code and relays it to the real site in real time. Passkeys / WebAuthn cryptographically bind the credential to the real website’s domain. A fake site simply cannot obtain a valid signature, whatever the user does. That is the main reason passkeys are a big deal.
Machines authenticate too:
- API keys are bearer secrets - whoever holds the key is the service. Leaked key, full impersonation.
- mTLS certificates prove identity during the TLS handshake.
- Workload identity has the platform issue short-lived credentials to a running workload automatically, with nothing stored on disk.
The direction of travel: short-lived, automatically issued, narrowly scoped credentials instead of long-lived static keys.
Sessions and tokens
After login, the server issues either a session (a random ID in a cookie, with state kept on the server) or a token (a self-contained signed statement, often a JWT). Sessions are easy to revoke. Self-contained tokens are easy to scale but hard to revoke before they expire.
Common JWT mistakes: accepting the none algorithm, mixing up algorithms, not checking issuer, audience and expiry, and very long lifetimes. For cookies, set HttpOnly (scripts cannot read it), Secure (HTTPS only), and SameSite (limits cross-site sending).
OAuth 2.0 and OpenID Connect
OAuth 2.0 is delegated authorization. A user lets an app act on their behalf, with limited scopes, without handing over their password. The roles: the user (resource owner), the app (client), the authorization server that issues tokens, and the API (resource server) that accepts them.
OpenID Connect (OIDC) adds authentication on top: an ID token that says who the user is. “Sign in with Google” is OIDC.
The usual flows are authorization code with PKCE for apps acting for users, and client credentials for service-to-service calls. Typical mistakes: scopes far broader than needed, sloppy redirect-URI checks, tokens leaking into logs or URLs, and refresh tokens stored carelessly.
AI link. OAuth is how agents should reach user data, and the Model Context Protocol (MCP) uses OAuth for authorizing remote servers. Scopes are your main least-privilege lever for agents: an agent that only needs to read a calendar should never hold a token that can delete email.
Authorization models
| Model | Idea | Good for | Watch out for |
|---|---|---|---|
| Access control list (ACL) | Each resource lists who may do what | Files, simple systems | Hard to audit at scale |
| Role-based (RBAC) | Permissions grouped into roles; users get roles | Organizations | Role explosion; roles that grant too much |
| Attribute-based (ABAC) | Policies over attributes (department, sensitivity, time) | Fine-grained rules | Policies get complex and hard to test |
| Relationship-based (ReBAC) | Permission follows relationships (“editor of the folder containing this doc”) | Document sharing, multi-tenant apps | Needed for permission-aware RAG; requires a relationship store |
| Capabilities | Holding an unforgeable token is the permission | Least privilege by construction | Tokens must not leak or be forwarded freely |
Policy engines such as OPA and Cedar keep authorization rules separate from application code, so they can be reviewed and tested.
The most common real bug is broken object-level authorization (also called IDOR). The API checks that you are logged in, but not that object 123 belongs to you. Change the ID in the URL and you read someone else’s data. Broken access control tops the OWASP web risk list for a reason.
The confused deputy
This 1988 idea is the key to agent security, so it is worth knowing well. A compiler service had permission to write a billing file. A user asked it to write its output to a path - which happened to be the billing file. The compiler used its own authority on behalf of a user who had none, and overwrote the billing records.
The pattern: a deputy (a program holding authority) is tricked into using that authority for a party that should not have it. The root cause is ambient authority - permissions attached to the program, not to the specific request. The fix is to make authority travel with the request (capabilities), so the deputy acts with the requester’s rights, not its own.
AI link. An LLM agent is a deputy. It holds your credentials and takes instructions from a mix of sources: you, plus every document, email and web page it reads. Indirect prompt injection is a confused-deputy attack: an outsider’s text makes the agent use your authority for them. A guiding principle for Part 11: an action should run with no more authority than the least-trusted input that influenced it.
Secrets management
Secrets include API keys, database passwords, OAuth tokens and signing keys. Keep them out of source code, notebooks, committed .env files, container images, logs, and prompts. A system prompt is not a safe place for a key; assume prompts leak.
Good practice: store secrets in a secrets manager, inject them at runtime, scope each one narrowly, rotate regularly, and scan repositories for leaks automatically. Bots scan public code hosts for keys continuously, so assume a leaked key is abused within minutes. When one leaks: revoke first, investigate second.
AI link. LLM and cloud API keys are prized targets. Stolen credentials are used to resell model access or run up huge bills on the victim’s account, a pattern often called “LLMjacking”.
Questions to ask
- How does each user and each service prove its identity? Are credentials short-lived?
- Is our MFA phishing-resistant?
- Is authorization checked server-side on every request, for every object?
- Whose authority does this component act with - its own, or the requester’s?
- Could a less-privileged party get this component to use its privileges on their behalf?
- What can each token do, and is that the minimum the task needs?
- Where are secrets stored, who can read them, and how fast can we rotate them?