GGUF

Local LLM inference engine supporting GGUF models with hardware acceleration on Metal, CUDA, ANE, WebGPU. Features Flash Attention, MicroLoRA, RoPE, quantization (Q4-Q8, π-Quantization), MoE routing, and streaming tokens for browser and edge deployment.

Activation-aware Weight Quantization method that preserves model accuracy at 4-bit quantization by identifying and skipping important weights. Maintains 99%+ of original performance with moderate inference speed improvements.

Post-training quantization method for 4-bit weight compression that focuses on GPU inference performance. First quantization method to compress LLMs to 4-bit range while maintaining accuracy, minimizing mean squared error to weights.

A commutative vector quantization method for KV cache compression that reduces FP16 cache size by 87.5% with 2-bit quantization and enables 1-bit quantization, allowing LLaMA-3.1 8B to run with 128K context on a single RTX 4090 GPU.

Optimized variant of Faiss with faster ADC quantization for GPU-accelerated vector search via CUDA, achieving higher throughput and lower latency than CPU Faiss on large-scale similarity tasks. Designed for real-time AI applications, CV inference, and high-QPS workloads requiring NVIDIA hardware acceleration. Outperforms standard CPU Faiss and baselines like Annoy in GPU environments.

High-performance vector search library with product quantization.