LLM Engineering: RAG, Evals & Multimodal Apps
Build a transparent retrieval baseline, validate citations and access, design multimodal workflows, and evaluate an evidence-based tutor prototype.
By Admin
Free courseView course
Measure · Intermediate
Evaluation and reliability
Benchmarks, task-specific evals, LLM-as-judge and red-teaming, plus why leaderboards mislead.
Why it matters
You cannot improve what you cannot measure, and many published scores do not measure what they claim. Evals are the core skill of AI engineering.
Key ideas
- Benchmark design and contamination
- Task-specific eval sets
- LLM-as-judge and its biases
- Red-teaming
- Regression testing for prompts
You will be able to
- Build an eval set for your app
- Spot a contaminated benchmark
- Track quality across releases
Primary reading
- Holistic Evaluation of Language Models (HELM) (opens in a new tab)Liang et al. · 2022
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (opens in a new tab)Jimenez et al. · 2023
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (opens in a new tab)Zheng et al. · 2023
Learn it here
Applied Machine Learning: Build, Evaluate, Improve
Fit a real regression model, compare a baseline, calculate meaningful metrics, investigate errors, and write an honest model card.
By Admin
Free courseView course