Skip to content
Hacker News · AI·· 23 h agoAI score36

Show HN: Vantage-V-AI – 2-bit ternary KV-cache with attention sinks

Show HN: Vantage-V-AI – 2-bit ternary KV-cache with attention sinks

AI summary

Nguyen Vo Anh Khoa released Vantage V-AI v8.2.0, an open-source bounded-memory inference runtime that packs KV-cache keys and values into 2-bit ternary states, claiming 7.97x storage reduction over FP16. On 2x NVIDIA Tesla T4 GPUs with Qwen2.5-3B-Instruct, its FP8 streaming mode matched lossless output at 64/64 tokens, and attention sinks held average retrieval coherence at 0.988 across 512–4,096 token contexts. It integrates with Hugging Face AutoModelForCausalLM via a one-line `vantage.enable()` call under the MIT License.

Source:Hacker News · AI · github.com