Skip to content
Hot eventDeveloping

Werkbank releases 40-task benchmark for self-hosted LLMs

1 reports1 reporting sourcesUpdated 1 h ago

Start here

AI digest

Werkbank, a benchmark for self-hosted language models, has been released with 40 fixed tool-using tasks across seven categories. It records the time and tokens each solved task costs, and runs through any OpenAI-compatible chat-completions endpoint that returns native tool calls and token counts, such as vLLM, llama.cpp, SGLang, Ollama or LM Studio. It requires Linux, or Linux inside WSL2 on Windows, and the model must be served under a name rather than a file path. Trial, Quick, Normal and Full modes need 64K to 256K context, with Quick at 64K, Normal at 128K and Full at 256K. Results describe one setup of model, quantization, server and hardware, and there is no leaderboard. The code is licensed under Apache 2.0 and the documentation under CC BY 4.0.

Generated from reports · updated 42 min ago

Report timeline

Follow the reports to see the event from different sides.

November 18
  1. Hacker News · AI
    Werkbank: 40 tool-using tasks to evaluate self-hosted LLMs

    Werkbank is a benchmark that gives self-hosted language models 40 fixed tool-using tasks across seven categories and records the time and tokens each solved task costs. It runs through any OpenAI-compatible chat-completions endpoint, including vLLM, llama.cpp, SGLang, Ollama and LM Studio, and offers Trial, Quick, Normal and Full modes needing 64K to 256K context. Results describe one setup of model, quantization, server and hardware, and there is no leaderboard.

Heat for this event

Not enough continuous observations yet to draw a trend.