Hacker News · AI·· 4 h agoAI score38
Switching LLMs mid-conversation is a ski-rental problem
Switching LLMs mid-conversation is a ski-rental problem
AI summary
Edgee's Agent Gateway treats LLM routing in coding sessions as two coupled problems: a difficulty tier map that assigns a model to each turn, and a switching policy that decides when a running session actually changes model. Because providers cache conversation history, every model switch discards the warm cache and pays to rebuild it, so the gateway frames switching as a ski-rental online problem. Self-Learning Routing replays the past 30 days of an organization's sessions weekly to pick the cheapest tier map and switching settings that keep every turn on a capable model.
Source:Hacker News · AI · edgee.ai