Building an LLM: The Complete Guide

🧊 Part 1 — The Bigger Picture: Why Weights Are Just the Tip

Open any “how LLMs work” article and you’ll see the same story: weights, weights, weights. A matrix here, a gradient there, and boom — intelligence. It’s a clean narrative. It’s also misleading.

Weights are the artifact of training, not the training itself. They’re the final checkpoint file you download — but they encode the cumulative decisions of hundreds of engineers over months or years. To treat weights as “the model” is to confuse the score with the symphony.

This post covers the whole iceberg. We’ll start from the very first decision (how do we chop text into tokens?) and end with the last (how do we serve this thing to a million users without going bankrupt?). Along the way, you’ll understand why two teams with identical weights can produce wildly different products — and why the “secret sauce” of frontier labs is almost never in the weights themselves.

🔤 Part 2 — Tokenization: The First Decision

The Simple Version

Before a model can learn from text, the text has to be chopped into pieces the model can handle. These pieces are called tokens. A token might be a whole word (“hello”), part of a word (“un-” and “believ” and “able”), a character, or even a byte. The tokenizer is the translator between human text and model math.

The Technical Version

A tokenizer defines a vocabulary V of |V| tokens (typically 32,000 to 200,000) and a deterministic function that maps any Unicode string to a sequence of token IDs in {0, 1, ..., |V|-1}. Modern LLMs almost universally use subword tokenizers:

• BPE (Byte-Pair Encoding) — GPT family, Llama. Iteratively merges the most frequent adjacent pair of tokens.
• Unigram — SentencePiece (used by Llama, T5). Learns a probabilistic subword vocabulary.
• WordPiece — BERT, older models. Similar to BPE but optimizes likelihood.
• Byte-level BPE — GPT-2/3/4, Llama 3. Handles any Unicode without an UNK token.

Why This Matters More Than You Think

Tokenizer choice has downstream effects on everything:

• Context efficiency: GPT-2’s tokenizer needed ~2× more tokens than English to represent Chinese. Modern multilingual tokenizers (Llama 3’s 128K vocab) fixed this.
• Embedding size: Bigger vocab = bigger embedding matrix = more parameters.
• Code handling: Code has lots of symbols ({ } [ ] = ==). Tokenizers trained mostly on English prose tokenize code inefficiently. Specialized code tokenizers help.
• Special tokens: Models need tokens for <|begin_of_text|>, <|tool_call|>, <|image|>, etc. These shape what the model can do.

🏗️ Part 3 — Architecture: The Transformer Family

“Transformer” isn’t a single design — it’s a family. The original 2017 paper described an encoder-decoder architecture for translation. Modern LLMs use only the decoder half, and even that has evolved dramatically.

The Simple Version

Think of the architecture as the blueprint of a factory. The weights are the machines inside. You can put the same machines in two different factory layouts and get very different products. The layout matters.

Key Architectural Decisions

These choices compound. A model with SwiGLU + RMSNorm + RoPE + GQA + MoE (the 2025-2026 “standard recipe”) is fundamentally different from GPT-3’s ReLU + LayerNorm + learned positions + dense design — even at the same parameter count.

🍲 Part 4 — Data: The Real Secret Sauce

If architecture is the factory blueprint and weights are the machines, data is the raw material. And in 2026, every serious lab will tell you the same thing: data quality matters more than data quantity.

The Simple Version

You can build the most beautiful kitchen in the world, but if you cook with rotten ingredients, you’ll serve rotten food. The same is true of LLMs. A 7B model trained on pristine, carefully curated data will often beat a 70B model trained on garbage.

The Technical Version

Pre-training data pipelines in 2026 typically involve:

The Synthetic Data Revolution

By 2025-2026, synthetic data has become a first-class citizen. Microsoft’s Phi models, NVIDIA’s Nemotron, and many others demonstrated that carefully generated synthetic textbooks, coding problems, and reasoning traces can match or beat real data for specific capabilities. The key insight: synthetic data lets you control the distribution exactly — no more hoping the web has enough good math problems.

📈 Part 5 — Scaling Laws: The Math of Getting Bigger

In 2020, OpenAI published a paper showing that model performance scales predictably with compute. In 2022, DeepMind’s Chinchilla paper flipped the industry on its head by showing that most models were undertrained — they had too many parameters for the amount of data they’d seen.

The Simple Version

There’s a formula for how good a model will be, given how big it is and how much data it’s seen. You can use this formula to plan a training run before you spend $100M. Ignoring scaling laws is how you waste a year of work.

The Technical Version

The Kaplan/Chinchilla scaling laws say that loss L scales as a power law with model size N, dataset size D, and compute C:

L(N) ∝ N^(-α)      L(D) ∝ D^(-β)      L(C) ∝ C^(-γ)

The Chinchilla-optimal ratio is roughly 20 tokens per parameter. So a 70B model should train on ~1.4T tokens. A 400B model should train on ~8T tokens. Llama 3 405B trained on 15T tokens — well beyond Chinchilla-optimal — because Meta could afford it and because more data always helps.

Modern labs also use small-scale proxy runs to predict large-scale performance — training a 1B model on 1/100th the data and using scaling laws to forecast what the 100B version will look like. This saves enormous money.

⚖️ Part 6 — What Is a Weight?

Now that we’ve covered the context, let’s zoom into the weights themselves.

The Simple Version

Imagine a giant light board with billions of dimmer switches. Each switch has a value between roughly -1 and +1. When a word comes in, the signal flows through the board, and every switch it passes through nudges the signal a little — amplifying some meanings, suppressing others, rotating concepts into new directions. By the time the signal reaches the end, it has been shaped into a prediction. Each dimmer switch is a weight.

The Technical Version

A weight is a single scalar floating-point number (typically float32, bfloat16, or float8) stored in a multi-dimensional tensor. Weights are organized into matrices that perform linear transformations on activation vectors.

For a single linear layer with input dimension d_in and output dimension d_out, the weight matrix W has shape [d_out, d_in] and there is usually a bias vector b of shape [d_out]. The computation is:

y = W·x + b

In a modern 70B-parameter model, these matrices are enormous. A single attention projection matrix might be [8192, 8192] — that’s 67 million weights in one matrix, and there are several per layer, across 80 layers.

🗺️ Part 7 — Where Do the Weights Live?

The weights are organized into named modules. Here’s the anatomy of a single transformer block:

📊 Part 8 — The Complete Weight Reference Table

Below is a scrollable reference of every major weight group in a modern decoder-only transformer. The header stays pinned as you scroll.

🎯 Part 9 — How Are the Weights Set?

We don’t program the weights — we grow them.

The Simple Version

Imagine you’re blindfolded on a hilly landscape, trying to reach the lowest valley. You feel the slope under your feet and step downhill. Repeat a trillion times. That’s training — the landscape is the loss function, the slope is the gradient, and your feet are the optimizer.

The Technical Version

🧪 Part 10 — The Training Recipe: Hyperparameters

This is where labs’ “secret sauce” mostly lives. The same architecture and data can produce wildly different models based on hyperparameters.

Finding the right recipe requires hundreds of small proxy runs at 1B scale before committing to a 70B+ run. Labs like Meta, Google, and Anthropic publish papers describing these recipe choices — but the exact values are often kept private.

🌐 Part 11 — Distributed Systems: Training at Scale

A 70B model in bf16 is ~140 GB. That doesn’t fit on one GPU. A 405B model is ~810 GB. That doesn’t fit on 8 GPUs. Training these models is first and foremost a distributed systems problem.

The Simple Version

Imagine trying to read a 10,000-page encyclopedia with 1,000 friends. You can’t each read the whole thing — you have to split it up. But you also need to stay in sync, share notes, and make sure nobody’s pages are missing. That’s distributed training.

Parallelism Strategies

Modern frontier training uses 3D or 4D parallelism — combining DP + TP + PP + (SP or EP) simultaneously. The Llama 3 405B run used a complex hybrid across 16,000 H100s.

The Engineering Reality

• Fault tolerance: In a 16,000-GPU cluster, a GPU fails every few hours. Training must continue via automatic checkpointing and node replacement.
• Network topology: NVLink within a node (900 GB/s), InfiniBand/RoCE between nodes (400-800 Gbps). Network design matters as much as GPU count.
• Straggler detection: One slow node drags down the whole cluster. Systems must detect and replace stragglers automatically.
• Checkpointing: Saving 1+ TB of state every few hundred steps without stopping training. Async checkpointing to fast storage is critical.

🧬 Part 12 — The Three Phases of Making an LLM Useful

Raw pre-training produces a model that can complete text but won’t chat with you. Three phases turn it into an assistant.

🛡️ Part 13 — Alignment & Safety Infrastructure

Alignment isn’t a single step — it’s an infrastructure that spans the entire lifecycle.

The Layers of Safety

• Data-level: Filtering harmful content from pre-training data. Refusals baked into SFT examples.
• Training-level: RLHF with human reviewers; Constitutional AI (Anthropic) where the model critiques its own outputs against a set of principles; GRPO (DeepSeek) for reasoning-focused alignment.
• Mechanistic-level: Interpretability research identifying “refusal neurons” and “deception circuits” in the model. Anthropic, Google DeepMind, and OpenAI’s super-alignment teams publish on this regularly.
• System-level: Input/output guardrails (separate classifiers that filter prompts and responses), rate limiting, monitoring, incident response.
• Red-teaming: Thousands of adversarial humans (and AI red-teamers) trying to break the model before release. Findings feed back into training.
• Evals: Dangerous capability evaluations (bioweapons, cyber, persuasion) run before every release.

🎨 Part 14 — Training Specific Characteristics & Specialties

Want a model that’s great at Python? Or speaks like a pirate? Or diagnoses rare diseases? Here’s how each is achieved.

14.1 — Specialty: Code Generation

The training mix is re-weighted to 30-50% code tokens (GitHub, StackOverflow, documentation). SFT on curated (instruction, code) pairs with unit tests. Models like DeepSeek-Coder and CodeLlama show that even a small amount of high-quality code SFT dramatically lifts pass@k scores.

14.2 — Specialty: Medical / Legal / Financial

Continued pre-training on domain-specific corpora (PubMed, case law, SEC filings) at low learning rate (~1e-5) to avoid catastrophic forgetting. SFT on expert-written Q&A pairs. RLHF with expert reviewers for high-stakes outputs.

14.3 — Personality & Tone

Personality is almost entirely a product of SFT data style and RLHF preferences. The underlying “intelligence” weights barely change — only the top-layer stylistic projections shift. This is why a single base model can produce both a terse coding assistant and a warm therapy bot with different fine-tunes.

14.4 — Safety & Refusals

Trained via red-teaming: adversarial humans try to elicit harmful outputs, and those conversations are added to RLHF with “refusal is preferred” labels. The model learns a refusal direction in activation space.

👁️ Part 15 — Adding Modalities: Vision, Audio, Video

A pure LLM only sees text. To make it multimodal, we add new encoder modules whose outputs are projected into the LLM’s embedding space.

The key insight of modern multimodal architectures (Flamingo, LLaVA, GPT-4o, Gemini) is that the LLM backbone stays the same. We just add encoders that translate other modalities into “tokens” the LLM already understands.

📏 Part 16 — Evaluation: Measuring What Matters

You can’t improve what you can’t measure. Evaluation infrastructure is as important as training infrastructure — and often just as expensive.

The Benchmark Ecosystem

The Contamination Problem

If your training data contains the test questions, your benchmark scores are meaningless. Labs now use:

• Canary strings embedded in benchmarks to detect leakage
• Contamination detectors that compare training data against benchmark text
• Evergreen benchmarks (LiveBench, LiveCodeBench) that update regularly
• Held-out private evals that never get released

⚡ Part 17 — Inference: Making It Usable

A model is useless if it’s too slow or expensive to serve. Inference optimization is a massive subfield — and in many ways harder than training, because you have to serve millions of concurrent users at low latency.

The Inference Stack

The best inference stacks in 2026 combine all of these. A single served request might go through FlashAttention → PagedAttention → continuous batching → INT4 quantization → speculative decoding. The result: frontier-model quality at consumer-hardware prices.

🛠️ Part 18 — How to Actually Train Your Own (Practical Guide)

You probably aren’t pre-training a 70B model from scratch. But here’s what’s realistic in 2026:

The Practical Stack in 2026

• Framework: PyTorch + torch.distributed + FSDP2, or DeepSpeed ZeRO-3
• Tokenizers: SentencePiece or tiktoken, vocab ~100K–200K
• Training loop: Hugging Face TRL, Nanotron, litgpt, or torchtitan for larger runs
• Data: Dolma, RedPajama, StarCoderData, DCLM for pre-training; OpenHermes, Magpie, UltraChat for SFT
• Alignment: TRL’s DPOTrainer, Open-Instruct, or TRL’s GRPOTrainer
• Evaluation: lm-evaluation-harness, MMLU, GPQA, LiveBench, Chatbot Arena
• Serving: vLLM, SGLang, TensorRT-LLM

💰 Part 19 — Cost & Economics

Let’s talk money. Building and running LLMs is expensive, and the economics drive many of the technical decisions.

The Cost Stack

The Economics Shape the Tech

Many technical decisions are driven by cost:

• MoE exists because inference cost matters more than training cost at scale
• Quantization is driven by serving economics — every GB of memory saved is money
• Distillation lets you serve a 7B model that behaves like a 70B
• Speculative decoding cuts GPU hours, which cuts dollars
• Open weights (Llama, Mistral, Qwen) exist because the training cost is so high that only a few players can afford it — but everyone can benefit from the outputs

🧠 Part 20 — Where the Weights “Store” Different Kinds of Knowledge

Recent work in mechanistic interpretability has mapped what different parts of the network actually do:

• Early layers (1–10): Surface-level features — parts of speech, simple syntax, local context
• Middle layers (10–50): The “knowledge layers.” Fact recall, entity disambiguation, induction heads. This is where most factual knowledge lives as sparse features.
• Late layers (50–80): Task-specific computation, formatting, style. The “last mile” that converts internal representations into next-token distributions.
• Attention heads: Some specialize in previous-token (local copying), some in induction (completing patterns), some in previous noun (coreference), some in doc-start (attention to context beginning).

🔮 Part 21 — The Frontier: What’s Next

As of mid-2026, the field is moving fast on several fronts:

• Test-time compute scaling: Models like o3, Claude 4’s extended thinking, and DeepSeek-R1 spend more inference FLOPs on harder problems. The weights are the same — only the inference path changes.
• Mixture of Experts: Now the default for frontier models. Llama 4, Grok-3, and others use MoE to get 70B+ quality at 10-20B active-parameter inference cost.
• State-space models: Mamba-2, Jamba, and hybrids offer linear-time inference. Still catching up on quality but improving fast.
• Continuous training: Models that never stop learning — updating weights on a rolling stream of new data without forgetting. The holy grail of lifelong AI.
• Weight merging: Taking two fine-tunes of the same base model and averaging their weights to combine capabilities. Surprisingly effective.
• Agentic capabilities: Tool use, planning, memory, multi-step reasoning. Models that can browse the web, write and execute code, and coordinate with other models.
• Long context: 1M+ token contexts are now standard. 10M+ is being researched. This requires architectural innovations like ring attention and YaRN.
• Process reward models: Instead of just judging the final answer, judging each step of reasoning. Enables better math and coding.
• Multimodal unification: Single models that handle text, images, audio, video, and actions in one conversation. GPT-4o, Gemini 2.5, and Claude 4 are all heading this way.

🔄 Part 22 — The Full Lifecycle: From Idea to Production

Let’s zoom out and see how all these pieces fit together in a real project.

A frontier model goes through all nine phases over 12-24 months. Each phase has its own team, its own challenges, and its own failures. The weights you download at the end are just the final artifact of this enormous effort.

✅ Summary

Building an LLM is not just about weights. It’s about:

1. Tokenization — how you chop text into pieces
2. Architecture — the blueprint of the factory
3. Data — the raw material (and the real secret sauce)
4. Scaling laws — the math of getting bigger
5. Weights — the machines inside the factory
6. Training recipe — the hyperparameters and optimizer
7. Distributed systems — how you train across 16,000 GPUs
8. Post-training — SFT, RLHF, alignment
9. Safety — red-teaming, guardrails, interpretability
10. Specialties — code, medical, personality, modalities
11. Evaluation — measuring what matters
12. Inference — making it fast and cheap to serve
13. Economics — the cost reality that shapes everything
14. Operations — monitoring, feedback, continuous improvement

The weights are the model — but they’re also the output of an enormous, multi-year engineering effort. Understand the weights, and you understand the intelligence. Understand everything else, and you understand how to build it.