Data engineering for LLMs · Data Engineering

LLM training is the largest data engineering problem in AI. Pretraining turns petabytes of messy web pages, code and documents into a clean, deduplicated, filtered, mixed and tokenized stream; post-training turns small, carefully curated example sets into behaviour. In both, data decisions move model quality as much as architecture does.

The kinds of LLM data

Stage	Data	Scale	What matters most
Pretraining	Web text, code, books, papers	Trillions of tokens	Coverage, deduplication, quality filtering, mixture
Continued pretraining	Domain or long-context text	Billions of tokens	Domain relevance without forgetting general skills
Supervised fine-tuning (SFT)	Prompt and ideal response pairs	Thousands to millions of examples	Correctness, diversity, consistent format
Preference data	A prompt with a preferred and a rejected response	Thousands to millions of pairs	Clear guidelines, annotator agreement
Reinforcement learning tasks	Problems with checkable answers	Varies	Reliable verifiers, difficulty spread
Evaluation	Held-out benchmarks and app-specific test sets	Hundreds to thousands of items	Never leaking into training, stable versions

The pretraining pipeline, step by step

Acquisition. Web crawls such as Common Crawl, which publishes raw WARC files and extracted WET text; licensed code; books and papers; curated sources. Respect robots.txt and opt-outs, and store provenance with every document.
Extraction. Turn HTML into main text by removing navigation, ads and boilerplate (tools: trafilatura, resiliparse). Turn PDFs into text with layout-aware parsers or OCR, keeping headings, tables and math where possible. Extraction quality caps everything downstream.
Language identification. Classify each document’s language, typically with fastText-based models, and route it to language-specific rules.
Heuristic filters. Rules that remove obvious junk: too short, too many symbols, repeated lines, missing common words, placeholder text, blocklisted domains. Published rule sets from datasets such as C4 and Gopher are good references.
Model-based quality filters. A classifier scores how useful or educational a document is, often trained on labels produced by a strong LLM. FineWeb-Edu is a well-known example. Perplexity under a small reference model is another signal.
Safety and privacy. Toxicity filtering, removal of illegal content, and detection and redaction of personal data such as emails, phone numbers and IP addresses.
Deduplication. Exact, near-duplicate and semantic (explained below).
Decontamination. Remove documents that overlap with evaluation benchmarks, usually by long n-gram matching, so test scores stay honest.
Mixing. Decide the share of web, code, math, books and each language, and how many times each source is repeated. Teams tune mixtures with small-scale training experiments before committing large compute.
Tokenization and packing. Convert text to token IDs, pack them into fixed-length training sequences, and write shuffled shards with a manifest.

Every step should record what it removed and why. A filter that silently drops 40% of German text is a bias you will only find if you measure it per slice.

Deduplication, and why it matters so much

Duplicates waste compute, make models memorize and regurgitate text, raise privacy risk, and leak test data into training. Three levels:

Exact: hash normalized text (lowercased, Unicode-normalized, whitespace collapsed) and keep one copy per hash.
Near-duplicate: catch documents that differ only in a date, a footer or a few words. MinHash with locality-sensitive hashing is the standard method.
Semantic: embed documents and remove clusters of paraphrases. More expensive, used selectively.

How MinHash works. Split each document into overlapping word n-grams called shingles. Two documents’ similarity is their Jaccard similarity: shared shingles divided by total distinct shingles. Comparing every pair directly is impossible at web scale, so apply k hash functions to the shingles and keep the minimum value for each. The resulting k numbers are the document’s signature. The chance that two signatures agree at any position equals the documents’ Jaccard similarity.

LSH avoids comparing all pairs. Split each signature into b bands of r values. Two documents become candidates if any band matches exactly. For two documents with similarity s:

P(\text{candidate}) = 1 - \left(1 - s^{r}\right)^{b}

This curve is steep: choosing b and r sets a threshold, below which pairs are almost never compared and above which they almost always are. Candidates are then verified and grouped, and one document per group is kept.

Tokenization from the data side

A tokenizer, usually byte-pair encoding (BPE) or SentencePiece, is itself trained on data. Its training sample decides how efficiently each language and domain is encoded. German compound words, code and math may cost noticeably more tokens per word than English prose if they were under-represented, which makes training and inference more expensive for them.

After tokenization, sequences are packed into fixed lengths, and stored in formats built for fast streaming: memory-mapped arrays of token IDs, MosaicML’s MDS shards, or WebDataset tar shards. Section 15 explains why.

Synthetic data

LLM-generated data is now central: rephrased web text, generated instructions and answers, distilled reasoning traces, and code with tests.

Verify what you can. Run generated code against tests, check math answers, validate structured output against schemas.
Filter and deduplicate. Generators repeat themselves; diversity collapses quickly without checks.
Track provenance. Record the generator model, prompt template, sampling settings and the terms of use that apply.
Mix with real data. Repeatedly training models only on their own outputs degrades them over generations, an effect called model collapse.

Fine-tuning data

SFT data is small and every example counts. A typical JSONL record:

{"messages": [
  {"role": "system", "content": "You answer questions about German tax forms."},
  {"role": "user", "content": "What is a Steuernummer?"},
  {"role": "assistant", "content": "A Steuernummer is the tax number your local Finanzamt assigns..."}
],
 "source": "expert_written_v2", "license": "internal"}

Things that silently go wrong:

Chat template mismatch: training with one template and serving with another degrades the model without any error.
Loss masking: usually only assistant tokens should contribute to the loss; check that the training code does this.
Inconsistent style or quality: the model learns the average of its examples, flaws included.
Overlap with evaluation prompts: deduplicate against every eval set.

Evaluation data

Treat eval sets as the most protected data you own. Version them, never train on them, check contamination before every run, and break results down by slice. Keep a frozen core set for comparability and a growing set built from real production failures. When an LLM judges outputs, version the judge model and its rubric as part of the dataset.

Multimodal data

Image-text pairs, audio with transcripts, and video follow the same pipeline with different tools: perceptual hashes for image deduplication, image-text similarity scores to filter mismatched captions, and sharded storage because the files are large. Store blobs in object storage and metadata in tables, as in section 2.

Tooling

Open-source pipelines worth studying include Hugging Face’s datatrove, NVIDIA’s NeMo Curator and the Allen Institute for AI’s Dolma toolkit. They run the steps above on Spark, Ray or Slurm clusters. Published dataset write-ups such as FineWeb, Dolma and RedPajama explain their filtering decisions and ablations, and are the best textbook on this topic.

Questions to ask

What fraction of data did each filter remove, per language and per source?
How was deduplication done, at what threshold, and across which splits?
Has every evaluation set been checked for contamination against this training data?
Which model, prompt and license produced each synthetic example?
Does the chat template used in training match the one used in serving?