Skip to content
Hot eventDeveloping

SageMaker AI adds default drift monitoring for inference endpoints

1 reports1 reporting sourcesUpdated 2 h ago

Start here

AI digest

Amazon SageMaker AI now monitors generative AI inference endpoints by default, flagging latency drift, underutilized GPUs, missing speculative decoding, outdated serving containers and untuned GPU kernels. The monitoring applies to endpoints that use vLLM serving containers on GPU instances and requires no setup; users can opt out per endpoint by setting the OPT_OUT_TRACKING environment variable on the model's container. Findings arrive as AWS Health notifications. From a notification, users can open an agentic experience in SageMaker Studio that benchmarks alternatives on separate compute against response time, throughput and cost per million tokens. According to AWS, monitoring itself incurs no GPU cost, while benchmarking is billed at standard SageMaker AI rates or drawn from an existing ML reservation.

Generated from reports · updated 2 h ago

Report timeline

Follow the reports to see the event from different sides.

November 20
  1. AWS Machine Learning Blog
    Amazon SageMaker AI adds default monitoring for latency drift, idle GPUs and missed optimizations on inference endpoints

    Amazon SageMaker AI now monitors generative AI inference endpoints by default for endpoints using vLLM serving containers and GPU instances, flagging latency drift, underutilized GPUs, missing speculative decoding, outdated serving containers and untuned GPU kernels. Findings arrive as AWS Health notifications, and one selection opens the agentic experience in SageMaker Studio to benchmark alternatives on separate compute against response time, throughput and cost per million tokens. Monitoring reads only latency, GPU utilization and serving configuration, carries no GPU cost, and can be opted out per endpoint via the OPT_OUT_TRACKING environment variable.

Heat for this event

Not enough continuous observations yet to draw a trend.