Data and serialization · Python

Choose data formats by who reads them and whether the source can be trusted: text formats for interchange, columnar formats for analytics, safe tensor formats for weights. Never deserialize a pickle you did not create.

Formats at a glance

Format	Good for	Watch out for
JSON	API payloads, configs, LLM input and output	no NaN or infinity in strict JSON; tuples come back as lists
JSONL	datasets, logs, eval results (one record per line)	none serious; stream it and append safely
CSV	quick tables, spreadsheet exchange	no types; quoting and encoding problems
Parquet	large typed tables, columnar and compressed	binary; read with pyarrow, polars or pandas
YAML, TOML	human-edited configuration	YAML needs `yaml.safe_load`, never plain `load`
pickle	Python objects between trusted Python processes	runs code when loaded: never load untrusted files
safetensors	model weights	none serious; safe, fast and memory-mappable
`.pt` / `.bin` from `torch.save`	PyTorch checkpoints	pickle underneath; load with `weights_only=True`
`.npy` / `.npz`	NumPy arrays	none serious; simple and fast

Why pickle is dangerous

Unpickling can call arbitrary functions while reconstructing objects. A malicious file can therefore run code on your machine the moment you load it. That is why model hubs moved to safetensors and why recent PyTorch versions default torch.load to weights_only=True.

pandas, polars and DuckDB

pandas: a DataFrame is a set of named columns, each backed by an array, aligned on an index. Use column operations; df.apply(fn, axis=1) is a Python loop in disguise.
pandas assignment: chained assignment warnings come from the view-versus-copy question (Part 13). Write df.loc[mask, "col"] = value, and know that pandas is moving to copy-on-write semantics.
polars: a Rust engine on Arrow memory, multi-threaded, with a lazy query optimizer: pl.scan_parquet(path).filter(...).collect(). It is often much faster on large data and uses an expression API instead of an index.
DuckDB: in-process SQL that queries Parquet, CSV and DataFrames directly. Excellent for exploration.
Hugging Face datasets: Arrow-backed, memory-mapped datasets with map and a streaming mode for corpora too large for disk.

Files and paths

Use pathlib.Path: Path("data") / "train.jsonl", .exists(), .glob("*.parquet"), .read_text(encoding="utf-8").
Write important outputs atomically: write to a temporary file, then os.replace(tmp, final). A crash then never leaves a half-written checkpoint behind.
Memory-map huge arrays with np.load(path, mmap_mode="r") so only the parts you touch are read.

JSON in practice

import json

record = {"id": 1, "text": "Grüße aus Bonn", "score": 0.93}
line = json.dumps(record, ensure_ascii=False)   # keeps umlauts readable
back = json.loads(line)

The standard json module cannot serialize datetime, NumPy numbers or sets. Convert them first, or use Pydantic’s model_dump_json().
Model output that “is JSON” often arrives wrapped in prose or code fences. Prefer the API’s structured-output feature over regex extraction, and always validate (Part 10).

Questions to ask:

Can this file be trusted enough to deserialize?
Is this DataFrame operation vectorized, or a hidden Python loop?
What happens if the job dies halfway through writing?