Skip to content
Hot eventDeveloping

Shard open-source engine pools LAN GPUs for LLM inference

1 reports1 reporting sourcesUpdated 2 h ago

Start here

AI digest

SHARD released Shard (shard-engine), an open-source distributed local LLM inference engine that pools GPU VRAM and system RAM across machines on the same LAN. A master node discovers workers over UDP port 8001, splits model layers across nodes, and serves OpenAI-compatible (/v1/chat/completions) and Anthropic-compatible (/v1/messages) endpoints on port 8000 for tools such as Cursor, Claude Code and Aider. The project is at v1.8.1 under the MIT license and installs via pip install shard-engine; a shard fit command estimates whether a model fits a single GPU, a hybrid CPU+GPU setup or a pooled LAN cluster before download. LAN sharding requires two or more machines with TCP port 8000 and UDP/TCP port 8001 open, and the project says distributed layer execution across heterogeneous LAN GPUs works best on low-latency wired Gigabit/2.5GbE Ethernet, with Wi-Fi possibly showing higher inter-token latency.

Generated from reports · updated 2 h ago

Report timeline

Follow the reports to see the event from different sides.

November 18
  1. Hacker News · AI
    SHARD launches open-source engine for distributed LLM inference across LAN machines

    SHARD (shard-engine) is an open-source distributed local LLM inference engine that pools GPU VRAM and system RAM across LAN machines, using PyTorch, FastAPI, llama.cpp RPC and native C++20/MASM AVX2 kernels. A master node discovers workers over UDP port 8001, splits model layers across nodes, and serves OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) compatible endpoints on port 8000 for tools such as Cursor, Claude Code and Aider. The project is at v1.8.1 under the MIT license and installs via pip install shard-engine, with a shard fit command that estimates whether a model fits a single GPU, a hybrid CPU+GPU setup or a pooled LAN cluster before download.

Heat for this event

Not enough continuous observations yet to draw a trend.