Skip to content
Hacker News · AI·· 2 h agoAI score58

SHARD launches open-source engine for distributed LLM inference across LAN machines

Show HN: Shard – Distributed AI Inference Across Multiple Computers

AI summary

SHARD (shard-engine) is an open-source distributed local LLM inference engine that pools GPU VRAM and system RAM across LAN machines, using PyTorch, FastAPI, llama.cpp RPC and native C++20/MASM AVX2 kernels. A master node discovers workers over UDP port 8001, splits model layers across nodes, and serves OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) compatible endpoints on port 8000 for tools such as Cursor, Claude Code and Aider. The project is at v1.8.1 under the MIT license and installs via pip install shard-engine, with a shard fit command that estimates whether a model fits a single GPU, a hybrid CPU+GPU setup or a pooled LAN cluster before download.

Source:Hacker News · AI · github.com