SageMaker AI adds KV cache-aware routing for real-time endpoints
Start here
Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints. The routing strategy uses cache state reported by the model container to send each request to the instance holding the longest cached match, while also weighing how busy that instance is. It is available at no additional charge in all commercial AWS Regions that support SageMaker AI real-time inference, for endpoints using a supported container. SageMaker AI runs and manages the cache index, and customers pay only for the instances their endpoint uses. Because the feature needs a choice of destinations, it is active only when an endpoint has at least two instances, or at least two copies of an inference component.
Generated from reports · updated 3 h ago
Report timeline
Follow the reports to see the event from different sides.
- AWS Machine Learning BlogAmazon SageMaker AI real-time endpoints add KV cache-aware routing
Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints, which uses cache state reported by the model container to send each request to an instance holding the longest cached match while weighing how busy that instance is.
Heat for this event
- Current, comparable range
- 9
- Peak, comparable range
- 10November 18 06:00
- Change in 24 hours
- –
The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.