Werkbank releases 40-task benchmark for self-hosted LLMs
Start here
Werkbank, a benchmark for self-hosted language models, has been released with 40 fixed tool-using tasks across seven categories. It records the time and tokens each solved task costs, and runs through any OpenAI-compatible chat-completions endpoint that returns native tool calls and token counts, such as vLLM, llama.cpp, SGLang, Ollama or LM Studio. It requires Linux, or Linux inside WSL2 on Windows, and the model must be served under a name rather than a file path. Trial, Quick, Normal and Full modes need 64K to 256K context, with Quick at 64K, Normal at 128K and Full at 256K. Results describe one setup of model, quantization, server and hardware, and there is no leaderboard. The code is licensed under Apache 2.0 and the documentation under CC BY 4.0.
Generated from reports · updated 42 min ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIWerkbank: 40 tool-using tasks to evaluate self-hosted LLMs
Werkbank is a benchmark that gives self-hosted language models 40 fixed tool-using tasks across seven categories and records the time and tokens each solved task costs. It runs through any OpenAI-compatible chat-completions endpoint, including vLLM, llama.cpp, SGLang, Ollama and LM Studio, and offers Trial, Quick, Normal and Full modes needing 64K to 256K context. Results describe one setup of model, quantization, server and hardware, and there is no leaderboard.
Heat for this event
Not enough continuous observations yet to draw a trend.