paged-attention

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

a community oriented 1:1, vLLM-alike (Continuous batching, paged KV) engine in C++ with additional features (GGUF, RadixAttention, Cache-aware scheduling, ...)

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching

A high-throughput LLM serving engine with non-uniform KV cache compression, built on vLLM

⚡ High-throughput LLM serving runtime to run frontier 70B–753B models on a single 24GB GPU (RTX 4090/L4). Features 75% KV compression, 57% subspace activation pruning, and dual OpenAI + Anthropic APIs.

Implementation of PagedAttention from vLLM paper - a breakthrough attention algorithm that treats KV cache like virtual memory. Eliminates memory fragmentation, increases batch sizes, and dramatically improves LLM serving throughput.

Rust + cuTile research prototype for paged latent-cache LLM decode attention, validated on an RTX 4060.

SILKern: sparse-index localization kernels for context-parallel decode — deterministic, allocation-free, CUDA-graph-safe

A from scratch LLM inference engine build in PyTorch with custom GPT2 transformers, kv cache, paged kv cache, continuous batching and A100 benchmarks

An AI inference engine from scratch. Like nanoGPT, but for serving.

A tested from-scratch Qwen inference engine: KV caching, continuous scheduling, paged KV, SSE, and Triton on Modal.

🌱 A tiny, readable LLM serving engine with vLLM/SGLang-style features.

Measurement harness for the sliding window attention premium in the vLLM TPU Ragged Paged Attention v3 kernel: per layer decode cost, block size control, throughput, and goodput for Gemma 4 31B on TPU v6e.

High-performance On-Device MoA (Mixture of Agents) Engine in C++. Optimized for CPU inference with RadixCache & PagedAttention. (Tiny-MoA Native)

An animated, click-through companion to IOP Systems' Inference Infrastructure blog series. Eleven modules take you from a single attention head all the way to a production serving stack — every concept rendered as a hands-on animation you can play, pause, and step through at your own pace.

A mini-vLLM: a from-scratch LLM inference server with a paged KV cache and continuous batching (GPT-2, OpenAI-compatible streaming API).

Local-first LLM stack on a single RTX 5090: QLoRA fine-tuning, exact speculative decoding, paged KV-cache, and continuous batching — served via FastAPI with a live React dashboard.

Discrete-tick simulator for KV-cache memory compaction policies in LLM inference servers. Compares NoCompaction, GreedyCompaction and ThresholdCompaction via 2D parameter sweep, Pareto frontier analysis and latency impact (P95/P99). C++20 + Python.