Graphcore and Arm fit an 11B Llama 3.2 Vision model on an Android phone at 3.7 GB
From research to reality: Fitting an 11B VLM on a phone
Graphcore and Arm researchers compressed the 11B Llama 3.2 Vision Instruct VLM from 21.3 GB of bfloat16 weights to a 3.7 GB implementation that runs on an Android phone CPU. Their S3D8 format stores three weights per byte at 2.7 bits/weight using 3D vector quantisation and a small lookup table, and quantisation-aware training with sampled prompts recovers downstream task performance that direct casting destroys. The team reports S3D8 reaching 33.8 GMAC/s versus 26.5 GMAC/s for INT8 on a 4096 x 14336 matrix-vector product, and about 22% extra compression over block-scaled INT at equal task performance; they note the 11B model is still too heavy for mainstream mobile use and would benefit from Vulkan kernels for prefill.
Source:Hacker News · AI · graphcore.ai