LLMOps Course
LLMOps is the practice of shipping, monitoring, evaluating and iterating on LLM systems in production. This course covers deployment patterns, versioning of prompts and models, offline and online evaluation, observability, cost and latency budgets, and the incident playbooks that decide whether a model change is safe to roll out.
LLMOps is the practice of shipping, monitoring, evaluating and iterating on LLM systems in production. This course covers deployment patterns, versioning of prompts and models, offline and online evaluation, observability, cost and latency budgets, and the incident playbooks that decide whether a model change is safe to roll out.
- Evaluation before every release
- Prompt and model versioning
- Latency and cost budgets
- Shadow and canary rollouts
- Modules
- 8
- Format
- Online
- Mentorship
- Weekly 1:1
- Project
- Production
Why this track
What you get that a general AI course does not cover.
The demo-to-production gap is where projects die
A prompt that works on your machine fails on real traffic for boring reasons: rate limits, long inputs, retries stacking, and context nobody controls. This course is about those reasons specifically.
Evaluation is a gate, not a report
We wire evaluation into the release path so a change cannot ship without a score. A dashboard nobody checks does not prevent a regression.
Cost and latency as budgets you enforce
Tokens per request, cache hit rate and p95 latency become explicit limits with alerts, not a surprise on the invoice after a marketing push.
Safe rollout of model changes
Model upgrades and prompt edits change behaviour in ways unit tests miss. Shadow traffic, canaries and instant rollback are how you find out before users do.
Curriculum
Eight modules, each ending in something you can show.
- M01
What breaks in production
A catalogue of real failure modes: nondeterminism, token blowups, provider outages, rate limits and silent quality drift.
- Failure taxonomy
- Nondeterminism
- Token blowups
- Provider failure
Tools
- Python
- M02
Offline evaluation
Building representative test sets, scoring with LLM judges where they help, and calibrating human evaluation against them.
- Test set design
- LLM-as-judge
- Human calibration
- Metrics
Tools
- promptfoo
- Ragas
- LangSmith
- M03
Online evaluation and monitoring
Sampling production traffic for scoring, user feedback signals, drift detection and alerting on quality signals.
- Traffic sampling
- Drift detection
- Feedback signals
- Quality alerts
Tools
- Langfuse
- LangSmith
- OpenTelemetry
- M04
Versioning prompts and models
Treating prompts as code: registries, diffs, lineage from prompt to evaluation result, and reproducing a bad output exactly.
- Prompt registries
- Lineage
- Reproduction
- Model pinning
Tools
- Langfuse
- Git
- M05
Serving patterns
Batching, caching, streaming, queueing and concurrency control. What to do when the provider is the bottleneck.
- Caching layers
- Batching
- Queues
- Concurrency limits
Tools
- vLLM
- Redis
- Celery
- M06
Cost and latency engineering
Per-request cost attribution, model routing by difficulty, prompt caching, and enforcing budgets with hard limits.
- Cost attribution
- Model routing
- Prompt caching
- Budgets
Tools
- LiteLLM
- OpenTelemetry
- M07
Reliability and safe rollout
Retries with jitter, circuit breakers, fallbacks across providers, shadow traffic, canary releases and rollback.
- Circuit breakers
- Fallbacks
- Shadow traffic
- Canary release
Tools
- Envoy
- Kubernetes
- M08
Security and incident response
Prompt injection in deployed systems, secret and tenant isolation, audit trails, and the playbook for a bad-model-change incident.
- Prompt injection
- Tenant isolation
- Audit logs
- Incident playbook
Tools
- Vault
- OpenTelemetry
Roles this maps to
Where this track lines up with real job descriptions.
MLOps Engineer
Extend your MLOps practice to language models, where evaluation and cost behave differently from classical training.
AI Platform Engineer
Own the shared runtime, evaluation gates and observability that every team on the platform depends on.
Senior LLM Engineer
Take responsibility for a production LLM feature's reliability, cost and safe change process.
Site Reliability Engineer
Bring SLOs, incident response and safe rollout discipline to systems whose failures are qualitative.
What you can do afterwards
Concrete capabilities, not a certificate.
- Design an evaluation suite that gates releases and catches regressions
- Trace any bad production output back to an exact prompt and model version
- Set and enforce per-request cost and p95 latency budgets
- Ship model and prompt changes through shadow and canary stages with rollback
- Run a cross-provider fallback that survives a provider outage
Questions
Is this different from MLOps?
Considerably. Classical MLOps revolves around retraining and reproducible training runs. LLMOps is dominated by prompt and model changes you cannot retrain, qualitative evaluation, and per-token cost.
Do I need Kubernetes?
Not to start. Local processes and a simple server cover the concepts. The deployment patterns transfer to whatever you run, and we name the pieces you will meet in Kubernetes without requiring it.
Which observability tool should I use?
The concepts are tool-independent. We demonstrate with Langfuse and OpenTelemetry, but the traces, metrics and spans are the same whichever backend you pick.
What if I have no production traffic yet?
Then you generate it. Replaying a captured dataset and running shadow traffic gives you the same signals, which is also the right way to rehearse before real users arrive.
Related tracks
Ready when you are
Applications for the next cohort are open. You pick your track during the first month, so you can change if the material does not suit you.