Skip to content
Hacker News · AI·· 1 h agoAI score47

Werkbank: 40 tool-using tasks to evaluate self-hosted LLMs

Show HN: Werkbank – 40 tool-using tasks to evaluate self-hosted LLMs

AI summary

Werkbank is a benchmark that gives self-hosted language models 40 fixed tool-using tasks across seven categories and records the time and tokens each solved task costs. It runs through any OpenAI-compatible chat-completions endpoint, including vLLM, llama.cpp, SGLang, Ollama and LM Studio, and offers Trial, Quick, Normal and Full modes needing 64K to 256K context. Results describe one setup of model, quantization, server and hardware, and there is no leaderboard.

Source:Hacker News · AI · github.com