Cybersecurity for AI Systems from First PrinciplesPost 6 of 19 · 19 of 19 written
- How to use this guide
- 01What security actually is
- 02How computers fail
- 03Networks
- 04Cryptography
- 05Identity and access
- 06Web and application security
- 07Cloud, infrastructure, supply chain, detection
- 08The bridge: why AI systems are different
- 09Adversarial machine learning
- 10LLM and generative AI security
- 11Agentic systems
- 12The ML supply chain and MLOps
- 13Privacy, governance and regulation
- 14Red teaming, evaluation, and AI for defense
- 15Worked example: threat-modeling an AI agent
- 16The question toolkit
- 17Learning path, practice, reading list
- 18Glossary
- 19Sources
6. Web and application security
Nearly every web vulnerability is either an injection (data treated as code) or a missing authorization check. The browser runs code from many sources side by side, and the server receives requests it can never fully trust. AI features inherit all of these bugs and add new ways to trigger them.
The browser security model
- Same-origin policy (SOP). An origin is scheme + host + port. A script from one origin cannot read data belonging to another. This is the browser’s core isolation boundary.
- Cookies travel automatically with requests to their site, even when another site triggered the request. This is the root of CSRF.
- CORS lets a server deliberately allow other origins to read its responses. Misconfigured CORS (for example, trusting any origin while allowing credentials) quietly removes the SOP’s protection.
- Content Security Policy (CSP) restricts which scripts may run on a page. It is a strong second layer against XSS.
The classic vulnerabilities, and their AI-era versions
| Vulnerability | Root cause | Primary fix | AI-era version |
|---|---|---|---|
| SQL / command / template injection | User input concatenated into a query or command | Parameterized queries; no shell strings; sandboxed templates | Model output placed into SQL or shell; text-to-SQL agents with write access |
| Cross-site scripting (XSS) | Attacker’s script runs in your site’s origin | Context-aware output encoding; HTML sanitization; CSP | Chat UIs rendering model-generated HTML or markdown |
| Cross-site request forgery (CSRF) | Browser sends authenticated requests another site triggered | SameSite cookies; anti-CSRF tokens; Origin checks |
Agents with browser sessions visiting attacker pages |
| Server-side request forgery (SSRF) | Server fetches a URL the attacker chose | Destination allowlists; block internal ranges; egress proxy | “Fetch this URL” tools, link summarizers, image-URL loaders |
| Broken access control / IDOR | Missing per-object authorization | Check ownership on every object, server-side | RAG returning documents the user may not see |
| Insecure deserialization | Formats that build arbitrary objects from data | Safe formats; never deserialize untrusted data | Loading pickled models and datasets |
| Security misconfiguration | Debug mode, defaults, open buckets | Hardened defaults; config review | Unauthenticated notebooks and model servers |
| Vulnerable components | Outdated dependencies with known bugs | Dependency scanning and updates | Large, fast-moving ML dependency trees |
| Business logic flaws | Rules that can be bent (negative quantities, races) | Careful design and testing | Agents talked into refunds or discounts |
Two of these deserve a closer look because they matter most for AI products.
XSS. If an attacker gets script running in your origin, they can do anything the logged-in user can do. Modern frameworks such as React escape output by default, and the danger comes from escape hatches (dangerouslySetInnerHTML, raw HTML in markdown renderers). The AI rule is simple: model output is untrusted input. The model may be repeating text an attacker planted in a document it read. Render it with the same care as any user-submitted content.
SSRF. When your server fetches a URL, it does so from inside your network, with your network’s access. An attacker can point it at internal admin panels, or at the cloud metadata endpoint (a special internal address where cloud machines fetch their own credentials) and steal those credentials. The 2019 Capital One breach involved this pattern. Defenses: allowlist destinations; resolve DNS first, then block private and link-local addresses (watch for DNS tricks that change the answer between check and use); require the hardened metadata service versions; and route fetches through an egress proxy. Every “browse” or “fetch” tool you give an agent is an SSRF surface.
APIs
- Authenticate and authorize every endpoint, checking both the function (“may this user call delete?”) and the object (“may they delete this record?”).
- Rate limits and quotas on everything. For LLM endpoints, cost is part of availability.
- Size limits on requests, uploads and prompts.
- Avoid mass assignment: binding request JSON straight onto database models lets users set fields like
is_admin. - Return only what the client needs; extra fields leak data.
The OWASP API Security Top 10 is the standard checklist here.
The secure development lifecycle
Security is cheapest at design time and most expensive in production. A mature workflow layers several checks:
- Threat model at design (Part 1).
- Secure defaults from frameworks; avoid their escape hatches.
- Code review with security in mind.
- SAST (static analysis such as Semgrep, CodeQL, Bandit for Python) finds known bug patterns in code.
- SCA (dependency scanning such as pip-audit and Dependabot) flags vulnerable libraries.
- Secret scanning catches committed keys.
- DAST (dynamic scanning such as OWASP ZAP) probes the running application.
- Fuzzing feeds generated inputs to parsers and APIs to find crashes (Atheris for Python).
Vulnerabilities are tracked publicly as CVEs and rated with CVSS scores from 0 to 10. A CVSS score measures generic severity, not your risk; a critical bug in a component you never expose may matter less than a medium one on your login page.
AI-written code. If you delegate implementation to AI coding tools, treat their output like a contribution from a fast, well-read stranger. It can contain every classic bug above, use outdated patterns, disable security checks to make an error disappear, or import packages that do not exist (an attacker may register that name; Part 12). Owning the architecture includes owning a security review of what gets generated.
Questions to ask
- Where does user-controlled or model-generated text get rendered, executed or used in a query?
- Does the server fetch any URL a user or model can influence?
- Is every endpoint, and every object, authorization-checked server-side?
- Are there size, rate and cost limits on every endpoint?
- What runs in CI to catch known bug classes, leaked secrets and vulnerable dependencies?
- Which business rule would an attacker most like to bend?