What security actually is · Cybersecurity

A system is secure when it keeps doing what it should, and nothing else, even while an intelligent adversary is actively trying to make it misbehave. The adversary is the whole difference between security and ordinary engineering.

Security, reliability, safety

Reliability is about random faults. Nature is not trying to break your disk; failures follow a distribution.
Security is about an adaptive opponent. The attacker picks the worst possible input, on purpose, and changes tactics when blocked.
Safety (in the AI sense) is about the system not causing harm even when nobody is attacking it - a model giving dangerous advice to an honest user.

The consequence is profound: average-case testing proves almost nothing about security. An attacker does not sample from your test distribution; they search for the worst case. This is exactly why 99% test accuracy tells you nothing about adversarial robustness (Part 9). Safety and security overlap: a jailbreak is a security failure whose impact is a safety failure.

The vocabulary of risk

Term	Meaning	AI example
Asset	Something of value you must protect	Model weights, training data, user chats, API keys
Threat actor	Who might attack, and why	Competitor stealing a model, fraudster abusing a chatbot
Threat	A possible harmful event	Exfiltration of user data via the assistant
Vulnerability	A weakness that makes the threat possible	Agent can read email and send HTTP requests freely
Exploit	A concrete technique that uses the vulnerability	A crafted email containing hidden instructions
Attack surface	All points where an attacker can interact	Chat input, uploaded files, retrieved web pages, tool outputs
Control	A measure that reduces risk	Egress allowlist, human approval, output filtering
Risk	Likelihood × impact	High: likely to be tried, damaging if it works

Security is risk management, not risk elimination. Zero risk is impossible at any price. The real questions are which residual risks remain, whether they are acceptable, and who formally accepts them.

The properties you protect: CIA and beyond

The classic triad names three properties. Every threat you discuss should map to at least one. If you cannot say which property a threat violates, you do not understand the threat yet.

Confidentiality - only authorized parties can read. In AI: training data, model weights, system prompts, conversation logs, embeddings (which can leak the text they encode).
Integrity - only authorized parties can change. In AI: training data (poisoning), weights (backdoors), retrieved documents, and the model’s actions.
Availability - it works when needed. In AI: GPU exhaustion, very long prompts, and “denial of wallet” (an attacker running up your token bill).

Useful extensions: authenticity (who really sent this?), non-repudiation (the sender cannot later deny it), accountability (actions trace to an actor), and privacy. Privacy is wider than confidentiality: it covers what can be inferred, not only what is read. A model that reveals whether a person was in its training set breaks privacy without leaking any record directly.

Adversary models: “secure against whom?”

“Secure” with no adversary named is meaningless. Practitioners describe an adversary along five axes:

Goal - steal, corrupt, disrupt, embarrass, profit.
Knowledge - what they know about the system. In ML: white-box (has weights and architecture), grey-box (partial), black-box (only query access).
Capabilities - what they can do: send inputs, edit a web page your agent will read, contribute to an open dataset, insert code into a dependency.
Resources - time, money, compute, skill. A bored teenager and a state intelligence agency are different threat models.
Access - outsider, authenticated user, insider, someone in your supply chain.

Kerckhoffs’s principle: assume the attacker knows how your system works; only keys stay secret. Hiding the design (“security through obscurity”) can add cost to an attacker, but it can never be the foundation. A secret system prompt is obscurity, not a control.

The asymmetry. A defender must cover every path; an attacker needs one. Defenders win back ground in two ways. Layered controls mean one hole is not enough. And detection means the attacker must stay invisible through every step of an attack, while the defender only needs to notice one.

Economics. Most attackers are rational. You rarely make an attack impossible; you make it cost more than it is worth. Good security raises attacker cost cheaply.

Trust and trust boundaries

In security, “trusted” does not mean “good”. It means “able to break your security if it misbehaves”. Everything you trust is something that can hurt you. The trusted computing base (TCB) is the set of components whose correctness your security depends on. A core design goal is to make the TCB as small as possible.

A trust boundary is any line where data or control passes between parties with different levels of trust. Examples: browser to server, user input to a SQL query, an internet document into an LLM’s context, a model’s output into a shell command. Most vulnerabilities live exactly at trust boundaries: data from the less-trusted side gets handled with the privileges of the more-trusted side.

A habit worth building: draw your system, put a box around everything with the same trust level, and examine every arrow that crosses a box.

Threat modeling

Threat modeling is structured thinking about what can go wrong, done at design time when fixes are cheap. Adam Shostack’s four questions are the core:

What are we working on?
What can go wrong?
What are we going to do about it?
Did we do a good job?

The usual method: draw a data-flow diagram (external entities, processes, data stores, data flows, trust boundaries). Then walk each element through STRIDE, which maps neatly onto the properties above:

Threat	Property violated	AI example
Spoofing	Authenticity	An email pretends to come from the CEO and the agent obeys it
Tampering	Integrity	Poisoned documents in the RAG index
Repudiation	Non-repudiation	No log of which tool calls the agent made, or why
Information disclosure	Confidentiality	Model reveals another tenant’s data or its system prompt
Denial of service	Availability	Prompts engineered to maximize tokens and GPU time
Elevation of privilege	Authorization	Injected text makes the agent call an admin-only tool

Attack trees are the complementary tool. The root is the attacker’s goal (“read another user’s chats”). Children are alternative ways to reach it (OR) or steps that must all succeed (AND). Leaves are concrete actions you can estimate cost for. They force you to think like the attacker, path by path.

For AI systems, you add domain checklists on top: MITRE ATLAS and the OWASP lists covered in Parts 10–11.

Design principles that have lasted 50 years

Saltzer and Schroeder published these in 1975. They still explain most modern failures, including AI ones.

Principle	Meaning	AI application
Least privilege	Every component gets only the permissions its task needs	A summarizing agent gets read-only email, no send
Fail-safe defaults	Default is deny; access is granted explicitly	Tools are allowlisted, not blocklisted
Complete mediation	Check every access, every time	RAG checks the user’s document permissions on every retrieval
Economy of mechanism	Keep security mechanisms small and simple	One clear policy layer beats ten clever prompt rules
Open design	Security must not rely on hiding the design	Assume the system prompt will leak
Separation of privilege	Require more than one condition for sensitive actions	Payment needs both a policy check and human approval
Least common mechanism	Minimize state shared between users	No shared agent memory across tenants
Psychological acceptability	Security people bypass is no security	Approval prompts on every action train users to click “allow”

Modern additions: defense in depth (layers, so one failure is not fatal), zero trust (network location grants nothing; authenticate and authorize every request), secure by default, minimize attack surface, and assume breach (design so a compromise is contained and detectable).

Questions to ask

What exactly are we protecting, and what happens if its confidentiality, integrity or availability fails?
Who would attack this, what do they want, and what can they see and touch?
Where are the trust boundaries, and what crosses each one?
What does this component trust, and what happens if that trust is misplaced?
If this component were fully compromised, what is the blast radius?
How would we know an attack happened, and how fast?
Which assumption, if false, breaks everything?