Self-service LLM gateway built on Amazon Bedrock and LiteLLM
Start here
A new reference architecture puts Amazon Bedrock foundation models behind an OpenAI-compatible API using LiteLLM on AWS Fargate, so developers get one base URL and a per-user key instead of managing model access individually. The setup adds per-user spend controls, per-model quota isolation, and CloudWatch RPM/TPM monitoring, with Langfuse handling prompt-level telemetry. One limit matters for adoption: self-service key issuance requires LiteLLM Enterprise, because the free tier caps usage at five users. The gateway is built on Bedrock Application Inference Profiles.
Generated from reports · updated 17 h ago
Report timeline
Follow the reports to see the event from different sides.
- AWS Machine Learning BlogBuilding a self-service LLM gateway with Amazon Bedrock and LiteLLM
A self-service LLM gateway built on Amazon Bedrock Application Inference Profiles and LiteLLM running on AWS Fargate exposes Bedrock foundation models behind an OpenAI-compatible API, giving developers a single base URL and per-user key. The setup adds per-user spend controls, per-model quota isolation, and CloudWatch RPM/TPM observability, with Langfuse for prompt-level telemetry. Self-service key issuance requires LiteLLM Enterprise, since the free tier is limited to five users.
Heat for this event
Not enough continuous observations yet to draw a trend.