2. Representing data: bytes, types, schemas, formats

All data is bytes plus an agreement about how to read them. Choosing a representation is choosing which operations will be cheap and which will be expensive.

Bytes and primitive types

• Integers have fixed widths. An int32 holds roughly ±2.1 billion; IDs and counters outgrow that more often than people expect, so use int64.
• Floats (IEEE 754) are approximations. 0.1 + 0.2 is not exactly 0.3, so never compare floats for equality and never store money as float; use a decimal type.
• Low-precision floats matter in AI. float32 is the classic default, while float16 and bfloat16 halve memory for weights, activations and stored embeddings. The cost is precision, so know which one a pipeline writes.
• Booleans and enums look trivial, but an enum is a contract: adding a new category upstream can break every consumer that assumed the list was closed.

Text is the hardest primitive

Unicode assigns a number (a code point) to every character. UTF-8 encodes each code point in 1 to 4 bytes, so string length in characters and in bytes differ.

• Mojibake (“ü” instead of “ü”) means bytes written in one encoding were read as another. With German text you will meet this constantly.
• Normalization matters for AI. “é” can be one code point or “e” plus a combining accent. Two strings can look identical and hash differently, which silently breaks deduplication. Normalize (NFC or NFKC) before hashing or comparing.
• Invisible characters: byte-order marks, zero-width spaces, non-breaking spaces and mixed line endings all survive copy-paste and poison tokenizers and joins.

Time

Timestamps are where most subtle data bugs live.

• Store instants in UTC and convert only for display. Germany switches between UTC+1 and UTC+2, so local time has a missing hour every spring and a repeated hour every autumn.
• Know whether a column is a timestamp with a time zone (an instant) or without (a wall-clock reading that means different instants in different places).
• Know the precision: seconds, milliseconds, microseconds or nanoseconds. Joining a millisecond column to a microsecond column silently matches nothing.
• Use ISO 8601 strings when text is unavoidable: 2026-10-30T14:05:00Z.

Null is not one thing

Null, empty string, zero, “N/A” and a missing key can each mean “unknown”, “not applicable”, “not collected yet” or “genuinely zero”. A model cannot tell them apart unless the pipeline does. Decide what each means and document it in the schema.

Schemas: the contract

A schema is field names, types, nullability, constraints and, most importantly, meaning (“price in EUR cents, including VAT”).

• Schema-on-write: the system rejects data that does not match (databases, Avro, Parquet). Errors surface early, at the producer.
• Schema-on-read: store anything, interpret it when reading (raw JSON in a data lake). Flexible, but errors surface late, at the consumer, often in a model.
• Schema evolution is how a schema changes safely. Adding an optional field is usually safe. Renaming, removing, or changing a type is breaking.
• Backward compatible means new readers can read old data. Forward compatible means old readers can read new data. Streaming systems use schema registries to enforce both.

Structured, semi-structured, unstructured

• Structured: fixed columns and types. Tables.
• Semi-structured: self-describing nested records whose shape can vary. JSON, logs, API responses.
• Unstructured: no machine-readable structure. Text, PDFs, images, audio, video.

Most AI data is unstructured, and a big part of AI data engineering is wrapping it in structure. The standard pattern: store the blob (an image, a PDF) in object storage, and store its metadata (source, license, language, hash, labels, embedding) as a row in a table with a pointer to the blob.

Row versus columnar layout

This single idea explains most of modern analytics performance.

• Row layout stores all fields of one record together. Reading or writing one whole record is cheap. This suits transactional systems: “fetch order 4711”.
• Columnar layout stores all values of one column together. Reading a few columns across billions of rows is cheap, because you skip every other column entirely.
• Columns also compress far better, since neighbouring values are similar, and they allow vectorized execution: the CPU applies one instruction to many values at once.

Worked example: a table has 1 billion rows and 100 equally sized columns. A query that averages one column reads about 1% of the bytes from a columnar file, and 100% from a row file.

File formats you will meet

Format	Layout	Schema	Typical use	Watch out for
CSV	Row, text	None	Exchange with humans and legacy systems	No types, quoting and delimiter bugs, encoding ambiguity
JSON / JSONL	Row, text, nested	Implicit	APIs, logs, LLM datasets (one JSON object per line)	Large and slow to parse; types drift between records
Parquet	Columnar, binary	Embedded	Analytics, lakehouse tables, training data	Files are immutable; too many small files hurt
ORC	Columnar, binary	Embedded	Hive and older Hadoop stacks	Less common outside that ecosystem
Avro	Row, binary	Embedded, evolvable	Kafka messages, write-heavy pipelines	Needs the schema, usually via a registry
Protobuf	Row, binary	Separate .proto file	Service-to-service messages, gRPC	Not self-describing without the schema
Arrow	Columnar, in memory	Embedded	Passing data between tools without copying	An in-memory standard, not an archive format
WebDataset (tar shards)	Sequential archive	By convention	Streaming images, audio and text into training	Random access is poor by design
safetensors	Tensor blobs	Header	Model weights	Weights, not datasets
Lance	Columnar, random access	Embedded	Multimodal AI data, vectors	Younger ecosystem

Parquet, in enough depth to reason about it

A Parquet file is split into row groups (often 128 MB to 1 GB of rows). Inside each, every column is a column chunk, split into pages. The footer holds the schema and statistics such as each chunk’s min and max values.

That structure buys two optimizations. Column pruning reads only the columns a query needs. Predicate pushdown uses the min/max stats to skip whole row groups: if a row group’s dates run from March to April, a query for May never reads it.

Values are first encoded (dictionary encoding for repeated strings, run-length encoding for repeats, delta encoding for sorted numbers), then compressed with a codec such as Snappy or Zstandard. Sorting data before writing improves both stats and compression.

Apache Arrow and why zero-copy matters

Every time data moves between tools, someone may serialize and deserialize it, which can cost more than the actual computation. Arrow is a shared in-memory columnar layout. pandas 2, Polars, DuckDB, Spark and PyTorch-adjacent libraries can hand Arrow buffers to each other without copying.

Questions to ask

• What is the schema, who owns it, and how is it allowed to change?
• What units, what time zone, what text encoding, what precision?
• What does null mean in each column?
• Is the format row or columnar, and does that match how it will be read?
• Can I read only the rows and columns I need, or must I read the whole thing?