Training an LLM

Creating a modern, production-ready Large Language Model is not a single training step. It is a multi-stage engineering lifecycle that transforms raw internet text into a helpful, instruction-following assistant.

The three foundational stages of LLM training are:

1. Pre-training: Learning world knowledge, grammar, and reasoning patterns from trillions of raw tokens (creates a Base Model).
2. Supervised Fine-Tuning (SFT): Teaching the base model how to respond to user instructions and engage in dialogue (creates an Instruct Model).
3. Preference Alignment (RLHF / DPO): Guiding the model to be helpful, honest, concise, and safe based on human preferences (creates a Production Model).

#Why does this matter?

As an AI engineer or systems architect, understanding the training pipeline enables you to make critical architectural decisions:

• Should you train a model from scratch, fine-tune an open-weights model (like Llama 3), or use in-context prompting / RAG?
• Why do base models output unformatted text while instruct models provide clean conversational responses?
• What are the compute, memory, and cost requirements for training and fine-tuning?

#Stage 1: Pre-training (The Heavy Compute Phase)

Pre-training accounts for 98%+ of the total compute cost of building an LLM.

Trillions of Tokens (Web, Code, Books) ──► Massive GPU Cluster (Thousands of H100s) ──► Base Model

#1. The Dataset

Pre-training datasets span petabytes of data:

• Common Crawl: Filtered, cleaned, and deduplicated web pages.
• Code Repositories: High-quality open-source GitHub repositories (Python, TypeScript, Rust, C++, SQL).
• Academic Papers: ArXiv, PubMed, and scientific journals.
• Books & Encyclopedias: High-density conceptual literature and Wikipedia.

#2. The Self-Supervised Objective: Next-Token Prediction

Pre-training does not require expensive manual human labeling. The training objective is self-supervised: the text itself provides the ground-truth answers.

Given an excerpt:

$$\text{"Docker containers share the host OS ____"}$$

The model makes an initial guess across its vocabulary. The true target token is "kernel".

#3. The Pre-training Training Loop

┌─────────────────────────────────────────────────────────────┐
│ 1. Sample a batch of text sequences                         │
│ 2. Tokenize and project through Transformer layers          │
│ 3. Compute Softmax probability distribution for next token  │
│ 4. Calculate Cross-Entropy Loss against actual target token │
│ 5. Backpropagate error gradients through all layers         │
│ 6. Update billions of weights via AdamW optimizer           │
│ 7. Repeat over 15+ Trillion tokens                          │
└─────────────────────────────────────────────────────────────┘

At the end of pre-training, the resulting Base Model is an extraordinary pattern completion engine—but it does not yet act like a chatbot. If you prompt a base model with "What is the capital of France?", it might simply respond with "What is the capital of Germany? What is the capital of Spain?" because it treats your prompt as a list in a geography quiz.

#Stage 2: Supervised Fine-Tuning (SFT / Instruction Tuning)

To turn a base completion engine into an assistant, the model undergoes Supervised Fine-Tuning (SFT):

Base Model + 100,000 High-Quality (Prompt, Response) Demonstrations ──► Instruct Model

In this stage, the model is trained on curated examples written by expert humans or generated by frontier teacher models:

{
  "prompt": "Write a Python function to reverse a linked list.",
  "response": "Here is an optimal iterative implementation in Python:\n\n```python\ndef reverse_list(head):\n    prev = None\n    current = head\n    while current:\n        next_node = current.next\n        current.next = prev\n        prev = current\n        current = next_node\n    return prev\n```"
}

Through SFT, the model learns formatting standards, conversational tone, code structure, and the convention of answering questions directly.

#Stage 3: Preference Alignment (RLHF & DPO)

Even after SFT, models can hallucinate, produce biased text, or generate verbose, unhelpful responses.

Alignment teaches the model which types of answers humans prefer:

Prompt: "Explain recursion simply."
Response A: [ Highly academic 5-paragraph mathematical proof ]  (Human: 👎)
Response B: [ Clear, intuitive Russian-doll analogy + brief code ] (Human: 👍)

#Methods of Alignment:

1. Reinforcement Learning from Human Feedback (RLHF): A separate Reward Model is trained on human preference ratings to score responses. The LLM is optimized via reinforcement learning (PPO) to maximize the reward score.
2. Direct Preference Optimization (DPO): A modern, computationally efficient alternative that directly optimizes the LLM parameters on pairs of accepted/rejected responses without training a separate reward model.

#The Scale of Compute and Precision

Training frontier models requires specialized high-performance computing clusters:

• Hardware: Thousands of interconnected GPUs (e.g., NVIDIA H100/H200, B200) networked using high-bandwidth InfiniBand (up to 3.2 Tbps per node).
• Numerical Precision: Models are trained using 16-bit floating point formats (BF16 / FP16) or mixed 8-bit precision (FP8) to conserve GPU memory bandwidth while maintaining numerical stability.
• Cost: A frontier model pre-training run typically costs between $10,000,000 to $100,000,000+ in pure compute energy and cluster time.

#Common Misconceptions

• Misconception: Fine-tuning is used to teach a model large amounts of new factual world knowledge. Reality: Pre-training teaches world knowledge; fine-tuning teaches behavior, style, format, and task alignment. Adding extensive new facts during fine-tuning often leads to catastrophic forgetting or hallucinations.
• Misconception: Once trained, an LLM continues learning from your chat conversations in real time. Reality: Model parameters are frozen in production inference. The model does not update its weights during standard chat interactions unless the engineering team collects logs and explicitly schedules a new fine-tuning run.

#Key Takeaways

[!IMPORTANT]

• Pre-training is the massive self-supervised phase where a base model learns language and reasoning by predicting next tokens across trillions of web tokens.
• Supervised Fine-Tuning (SFT) uses curated prompt-response pairs to teach the base model to act as an assistant.
• Alignment (RLHF / DPO) uses human preference data to ensure helpful, concise, and safe outputs.
• Production LLM weights are frozen during inference; learning during conversation is an illusion created by in-context memory.

#What comes next?

Now that we have a trained, aligned model with billions of frozen parameters, what actually happens when an end-user sends a prompt over an API?

In Lesson 07, we explore Inference and Generation—from the prompt prefill phase to autoregressive token sampling and streaming.