An educational open-source project is offering developers a guided route through the mechanics of large-language-model inference by combining a small working engine with more than six hours of video instruction and over 80 pages of handwritten notes. The repository, called samir-vllm, focuses on what happens between receiving a prompt and producing tokens on a GPU.
The engine is written in C++ and CUDA and implements an inference path for the one-billion-parameter Llama 3.2 model. Its materials cover loading model weights, executing transformer components, processing prompts and generating subsequent tokens. Later lessons move into operational concerns including prefill, decode, key-value caching, static batching and continuous batching.
The project is deliberately constrained as a teaching system rather than presented as a drop-in replacement for a production inference platform. Its configured model dimensions and 2,048-token context are fixed. That bounded design gives learners a concrete codebase in which memory management and scheduling decisions can be inspected without the wider complexity of a general-purpose serving system.
One detailed part of the implementation is its paged key-value cache. The cache defaults to 128 MiB but can be changed through a command-line option. Admission control reserves enough capacity for each request’s maximum processed prefix, using 16-token pages across the model’s 16 layers. Physical pages are assigned only when needed, while separate admission credits ensure an active request has enough budget to complete.
Requests wait in first-in, first-out order when their declared budgets cannot fit beside work already running. A request whose budget exceeds the entire pool is rejected. When a request finishes early, both assigned pages and unused credits are released. The policy avoids preemption and recomputation, though the repository notes that a large requested output allowance can reduce concurrency even when generation ultimately stops early.
The engine supports static and continuous batching. Prompts are prefilled one at a time between decode iterations, while each decode step can place one token from every active sequence into shared buffers. Continuous batching admits waiting work as capacity becomes available; static batching waits for the current group to finish. The documentation also cautions that BF16 rounding may make greedy output differ at different batch sizes when vocabulary scores are nearly tied.
Licensing is split by material. The engine source, tests and build files are offered under the MIT License. The notes, scans, illustrations and PDFs remain all rights reserved, with personal reading and study permitted but redistribution requiring permission. The bundled tokenizer and separately obtained model weights remain subject to Meta’s Llama 3.2 terms, while third-party components keep their own licenses.
For developers studying inference, the main contribution is the connection between explanation and implementation: scheduling, caching and batching are not only described, but exposed in a compact engine that can be built, inspected and tested.
