AWS Machine Learning Blog· Dmitry Soldatkin·· 3 h agoAI score58
Amazon SageMaker AI real-time endpoints add KV cache-aware routing
Amazon SageMaker AI real-time endpoints now support KV cache aware routing
AI summary
Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints, which uses cache state reported by the model container to send each request to an instance holding the longest cached match while weighing how busy that instance is.
Source:AWS Machine Learning Blog · aws.amazon.com