Kishor.Academy
Log in
← All courses

Online course · advanced

Post-Training: Fine-Tuning, RLHF & DPO

Learn how supervised fine-tuning, RLHF, DPO, and LoRA turn a pre-trained model into an assistant, audit preference data, and decide when fine-tuning is worth it. Offline labs; no GPU required.

With Admin

advanced9 lessons · 3 sections8–12 hours of reading and lab work, plus 4–6 hours for the project; estimates exclude video.

THE TAKEAWAYS

What you’ll learn

  • Explain what post-training changes in a pre-trained model and what it does not.
  • Prepare a supervised fine-tuning dataset with provenance, deduplication, and a held-out set.
  • Choose between prompting, retrieval, and fine-tuning using a metric and threshold.
  • Audit preference data for invalid pairs, duplicates, annotator conflicts, leakage, and length bias.
  • Describe RLHF's reward model and KL penalty, and calculate a penalised reward.
  • Compute the DPO loss for a preference pair and explain what it does not prove.
  • Calculate LoRA trainable parameters and explain how QLoRA reduces memory.
  • Design before-and-after evaluations that catch capability and safety regressions.
  • Deliver a fine-tuning decision memo with a data audit and evaluation plan.

Is this for you?

Developers and analysts who can prompt an LLM and write basic Python, and who need to decide whether and how to customise a model.

Before you start

  • Complete LLMs: From Prompts to Practice, or demonstrate prompt design and basic evaluation.
  • Python 3.10 or newer; comfort with functions, dictionaries, and reading JSON.
  • No GPU, paid API, or model download required. Running a real fine-tune is optional and must be labelled with the base model, data, settings, and date.

PUT IT TO WORK

What you’ll build

Fine-tuning decision memo. Client scenario: a team asks whether to fine-tune a model so its tutor always cites lesson IDs. Deliverable: a decision memo, preference-data audit, cost sketch, and evaluation plan.

Follow a model from pre-training through supervised fine-tuning and preference tuning. Compare prompting, retrieval, and fine-tuning against measured requirements. Audit a hand-written preference dataset for duplicates, conflicts, leakage, and length bias, then compute the DPO loss, a KL-penalised reward, and LoRA parameter counts in an offline lab. Finish with a fine-tuning decision memo that includes a data audit and a before-and-after evaluation plan.

Course curriculum

SECTION 1How post-training shapes a model3 lessons
  1. From pre-trained model to assistantReading
  2. Supervised fine-tuning and data qualityReading
  3. Prompt, retrieve, or fine-tune?Reading
SECTION 2Learning from preferences: RLHF and DPO3 lessons
  1. Preference data and its pitfallsReading
  2. RLHF: reward models and the KL penaltyReading
  3. Direct preference optimisationReading
SECTION 3Efficient fine-tuning, evaluation, and a decision3 lessons
  1. Parameter-efficient fine-tuning: LoRA and QLoRAReading
  2. Distillation, evaluation, and regressionsReading
  3. Capstone: a fine-tuning decision and data auditReading