tiny-llm is a hands-on course that teaches LLM inference system design to systems engineers by building a minimal serving system for the Qwen model on Apple Silicon.
The course addresses the gap between understanding LLM fundamentals and grasping the systems-level details of production inference. Rather than using high-level abstractions, students implement operators and serving machinery from first principles using MLX arrays and Metal kernels. The learning path progresses from basic matrix operations and model components like attention and RoPE through KV-cache optimization, continuous batching, and paged KV layouts, culminating in a coding agent that demonstrates practical serving patterns. By keeping implementations small and readable, the course connects mathematical operations directly to memory traffic, kernel occupancy, and scheduling decisions.
This course suits systems engineers who want to understand inference end to end without relying on expensive GPU infrastructure. Apple Silicon provides a practical local environment with shared memory and direct Metal access, while Qwen3-4B is large enough to expose real performance constraints yet small enough to iterate on a single machine. The course structure mirrors educational projects like CMU's Needle, positioning itself as a serving-focused counterpart to foundational ML systems courses.
Development activity shows steady progression through a structured curriculum. The course is published incrementally with reviewed checkpoints rather than as a complete release, with early weeks covering core inference mechanics and later weeks introducing scheduling and agent patterns. Implementation is split between student exercises in the main package and reference solutions that serve as correctness oracles and performance baselines. The project maintains a published book alongside code, with chapter readiness tracked explicitly.
