Skip to content
Hot eventDeveloping

Baseten adds deferred tool loading for open models

1 reports1 reporting sourcesUpdated 1 h ago

Start here

AI digest

Baseten has added deferred tool loading to the open models it hosts on Baseten Model APIs — DeepSeek, Kimi and GLM — supporting both Chat Completions and Messages API formats. Instead of injecting every tool schema into the system prompt, the harness passes tool names plus a tool_search tool (about 1.3k tokens, versus roughly 141k tokens for three MCP servers) and renders the full schema inline only when needed, which keeps the KV cache prefix valid. Baseten reports that in Claude Code with three MCPs enabled (Slack, Notion, Baseten), coding benchmarks ran 62.5% cheaper and 26.2% faster on average; on agentic benchmarks using preset tools and MCPs, tasks were 51.1% cheaper and 14.9% faster. These figures are Baseten's own measurements. The capability is generally available through the API for teams building their own context-conscious harness.

Generated from reports · updated 1 h ago

Report timeline

Follow the reports to see the event from different sides.

November 20
  1. Baseten BlogPicks
    Baseten brings deferred tool loading to open models, cutting coding agent cost by 62.5%

    Baseten added deferred tool loading to open models (DeepSeek, Kimi, GLM) on Baseten Model APIs, supporting both Chat Completions and Messages API formats. In Claude Code with 3 MCPs enabled, coding benchmarks ran 62.5% cheaper and 26.2% faster on average, while agentic benchmarks were 51.1% cheaper and 14.9% faster. Instead of injecting every tool schema into the system prompt, the harness passes tool names plus a tool_search tool (~1.3k tokens versus ~141k tokens for 3 MCP servers) and renders the full schema inline only when needed, keeping the KV cache prefix valid.

Heat for this event

Not enough continuous observations yet to draw a trend.