Skip to content
Hot eventDeveloping

Graphcore and Arm compress 11B VLM to 3.7 GB on phone

1 reports1 reporting sourcesUpdated 23 h ago

Start here

AI digest

Graphcore and Arm researchers compressed the 11B Llama 3.2 Vision Instruct model from 21.3 GB of bfloat16 weights to a 3.7 GB implementation that runs on an Android phone CPU. Their S3D8 format stores three weights per byte at 2.7 bits per weight using 3D vector quantisation and a small lookup table, and quantisation-aware training with sampled prompts recovers downstream task performance that direct casting loses. The team reports S3D8 reaching 33.8 GMAC/s versus 26.5 GMAC/s for INT8 on a 4096 x 14336 matrix-vector product, and about 22% extra compression over block-scaled INT at equal task performance. They say the 11B model is still too heavy for mainstream mobile use and that the implementation would benefit from Vulkan kernels to improve prefill time; they released it to support future development.

Generated from reports · updated 23 h ago

Report timeline

Follow the reports to see the event from different sides.

November 15
  1. Hacker News · AI
    Graphcore and Arm fit an 11B Llama 3.2 Vision model on an Android phone at 3.7 GB

    Graphcore and Arm researchers compressed the 11B Llama 3.2 Vision Instruct VLM from 21.3 GB of bfloat16 weights to a 3.7 GB implementation that runs on an Android phone CPU. Their S3D8 format stores three weights per byte at 2.7 bits/weight using 3D vector quantisation and a small lookup table, and quantisation-aware training with sampled prompts recovers downstream task performance that direct casting destroys. The team reports S3D8 reaching 33.8 GMAC/s versus 26.5 GMAC/s for INT8 on a 4096 x 14336 matrix-vector product, and about 22% extra compression over block-scaled INT at equal task performance; they note the 11B model is still too heavy for mainstream mobile use and would benefit from Vulkan kernels for prefill.

Heat for this event

Current, comparable range
5
Peak, comparable range
10November 15 18:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.