Skip to content
Hacker News · AI·· 4 h agoAI score38

Switching LLMs mid-conversation is a ski-rental problem

Switching LLMs mid-conversation is a ski-rental problem

AI summary

Edgee's Agent Gateway treats LLM routing in coding sessions as two coupled problems: a difficulty tier map that assigns a model to each turn, and a switching policy that decides when a running session actually changes model. Because providers cache conversation history, every model switch discards the warm cache and pays to rebuild it, so the gateway frames switching as a ski-rental online problem. Self-Learning Routing replays the past 30 days of an organization's sessions weekly to pick the cheapest tier map and switching settings that keep every turn on a capable model.

Source:Hacker News · AI · edgee.ai