A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

Kubernetes enhancements for Network Topology Aware Gang Scheduling & Autoscaling

CUDA Templates and Python DSLs for High-Performance Linear Algebra

cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.

NVIDIA Inference Xfer Library (NIXL)

A programmable Mixture-of-Models router for heterogeneous LLM inference

A Datacenter Scale Distributed Inference Serving Framework

Building the Virtuous Cycle for AI-driven LLM Systems

A Datacenter Scale Distributed Inference Serving Framework

FlashInfer: Kernel Library for LLM Serving

slime is an LLM post-training framework for RL Scaling.

FlashInfer: Kernel Library for LLM Serving

An LLM post-training framework with vLLM for RL Scaling

SGLang is a high-performance serving framework for large language models and multimodal models.

A high-throughput and memory-efficient inference and serving engine for LLMs

A high-throughput and memory-efficient inference and serving engine for LLMs

Fork of the classic

Development repository for the Triton language and compiler

Burn is a next generation tensor library and Deep Learning Framework that doesn't compromise on flexibility, efficiency and portability.

LLM inference in C/C++