Skip to content
AWS Machine Learning Blog· Dmitry Soldatkin·· 3 h agoAI score58

Amazon SageMaker AI real-time endpoints add KV cache-aware routing

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

AI summary

Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints, which uses cache state reported by the model container to send each request to an instance holding the longest cached match while weighing how busy that instance is.

Source:AWS Machine Learning Blog · aws.amazon.com