Skip to content
Hot eventDeveloping

SageMaker AI adds KV cache-aware routing for real-time endpoints

1 reports1 reporting sourcesUpdated 3 h ago

Start here

AI digest

Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints. The routing strategy uses cache state reported by the model container to send each request to the instance holding the longest cached match, while also weighing how busy that instance is. It is available at no additional charge in all commercial AWS Regions that support SageMaker AI real-time inference, for endpoints using a supported container. SageMaker AI runs and manages the cache index, and customers pay only for the instances their endpoint uses. Because the feature needs a choice of destinations, it is active only when an endpoint has at least two instances, or at least two copies of an inference component.

Generated from reports · updated 3 h ago

Report timeline

Follow the reports to see the event from different sides.

November 18
  1. AWS Machine Learning Blog
    Amazon SageMaker AI real-time endpoints add KV cache-aware routing

    Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints, which uses cache state reported by the model container to send each request to an instance holding the longest cached match while weighing how busy that instance is.

Heat for this event

Current, comparable range
9
Peak, comparable range
10November 18 06:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.