Build software better, together

ChriStingo / Huffman-coding

Simple Huffman coding

encoding statistics cpp huffman probability inference huffman-coding

Updated Jul 23, 2021
C++

AdityaGuhaa / BenchForge

A local-first, automated LLM benchmarking tool that runs llama-bench against your GGUF models across CPU & GPU and gives you an interactive comparison dashboard.

windows macos linux benchmarking machine-learning cpp inference quantization llama-cpp local-llm gguf llm-benchmarking llm-benchmark

Updated May 17, 2026
C++

Ardynai / llama.cpp

Star

LLM inference in C/C++

local models inference

Updated May 5, 2026
C++

sahan974 / qwen-3.5-cuda

Star

A C++/CUDA runtime for quantized Qwen3.5 MoE inference on consumer NVIDIA GPUs.

cuda inference gguf qwen3-5

Updated Aug 21, 2026
C++

rafaelzaydh-jpg / MiniMaxBrain

Star

Run 35B MoE models on a laptop with only 16 GB of RAM

python machine-learning deep-learning zero-copy inference moe memory-management granite mixture-of-experts memory-paging llm qwen gguf ai-infrastructure

Updated Aug 27, 2026
C++

Grifel79 / vision_docker

Star

Docker / C++ solution for CNN inference with LibTorch

opencv cmake inference imagenet libtorch

Updated Apr 27, 2023
C++

vizvasanlya / offlineai

Star

Offline AI inference engine with C++/CMake core and multi-platform support

machine-learning ai offline cpp inference

Updated Dec 12, 2025
C++

waqasm86 / cuda-nvidia-systems-engg

Star

Production-grade C++20/CUDA distributed LLM inference system with TCP networking, MPI scheduling, and content-addressed storage. Features comprehensive benchmarking (p50/p95/p99 latencies), epoll async I/O, and optimizations for low-VRAM GPUs (1GB+). 2500+ LOC demonstrating systems engineering expertise.

benchmarking performance networking tcp cpp storage gpu mpi distributed-computing cuda inference epoll quantization systems-programming llm

Updated Dec 27, 2025
C++

sanmathigb / nrvna

Star

Asynchronous Inference Primitive.

asynchronous inference edge job-queue edge-computing edge-ai local-first llama-cpp local-llm local-ai

Updated Aug 12, 2026
C++

tettou771 / tcxOnnx

Star

ONNX Runtime wrapper addon for TrussC

machine-learning creative-coding cpp inference onnx onnxruntime trussc-addon trussc

Updated Jun 30, 2026
C++

nexgus / nextfodie

Star

A RESTful API server to perform inference based on Intel OpenVINO 2018 R5

inference cpprestsdk intel-openvino

Updated Feb 1, 2019
C++

HexaForge-1 / highperf-ai-ml-inference

Star

High-perf C++ AI/ML inference engine with ONNX Runtime & LibTorch. CPU default, GPU opt-in. CLI + REST API, cross-platform via CMake/Docker. Auto-fetch models/assets, CUDA runtime check, CI merge gating, future multi-model & GPU training support.

docker cli machine-learning cmake ai deep-learning cross-platform cpp gpu rest-api high-performance cuda inference pytorch model-serving onnx libtorch onnx-runtime

Updated Aug 16, 2025
C++

dbjwhs / inference-systems-lab

Star

C++17 inference platform: belief propagation, Mamba SSM, MoE, neuro-symbolic logic, SIMD containers

machine-learning cpp inference simd cpp17

Updated Mar 14, 2026
C++

JohnScheuer / tensor-memory-allocator

Star

Simulates and compares GPU tensor memory allocation strategies (FreeList, Buddy, Slab, LifetimeAware) for LLM inference. 108-run sweep across 9 strategies, 4 pool sizes, 3 batch sizes. Key finding: simple free-list heuristics beat buddy/slab for continuous tensor size distributions.

simulation gpu inference memory-allocator systems-programming cpp20 fragmentation buddy-allocator slab-allocator llm

Updated Jul 10, 2026
C++

waqasm86 / local-llama-cuda

Star

Custom CUDA implementation for LLM inference with MPI-based distributed computing. Memory-efficient layer offloading, multi-rank coordination, and GPU optimization for constrained hardware (1GB VRAM).

cpp gpu mpi distributed-computing cuda inference memory-optimization llm