Baseten adds deferred tool loading for open models
Start here
Baseten has added deferred tool loading to the open models it hosts on Baseten Model APIs — DeepSeek, Kimi and GLM — supporting both Chat Completions and Messages API formats. Instead of injecting every tool schema into the system prompt, the harness passes tool names plus a tool_search tool (about 1.3k tokens, versus roughly 141k tokens for three MCP servers) and renders the full schema inline only when needed, which keeps the KV cache prefix valid. Baseten reports that in Claude Code with three MCPs enabled (Slack, Notion, Baseten), coding benchmarks ran 62.5% cheaper and 26.2% faster on average; on agentic benchmarks using preset tools and MCPs, tasks were 51.1% cheaper and 14.9% faster. These figures are Baseten's own measurements. The capability is generally available through the API for teams building their own context-conscious harness.
Generated from reports · updated 1 h ago
Report timeline
Follow the reports to see the event from different sides.
- Baseten BlogPicksBaseten brings deferred tool loading to open models, cutting coding agent cost by 62.5%
Baseten added deferred tool loading to open models (DeepSeek, Kimi, GLM) on Baseten Model APIs, supporting both Chat Completions and Messages API formats. In Claude Code with 3 MCPs enabled, coding benchmarks ran 62.5% cheaper and 26.2% faster on average, while agentic benchmarks were 51.1% cheaper and 14.9% faster. Instead of injecting every tool schema into the system prompt, the harness passes tool names plus a tool_search tool (~1.3k tokens versus ~141k tokens for 3 MCP servers) and renders the full schema inline only when needed, keeping the KV cache prefix valid.
Heat for this event
Not enough continuous observations yet to draw a trend.