Skip to content
Hot eventDeveloping

Langfuse publishes design and benchmarks for its Rust AI Gateway

1 reports1 reporting sourcesUpdated 1 h ago

Start here

AI digest

Langfuse published a design and benchmark write-up of its AI Gateway, a Rust-built proxy that sits in front of all model calls. Platform teams can issue cross-provider keys, track cost per key, user and team, and record every call as a Langfuse trace. Langfuse reports 2 ms median time to first token at 100 concurrent streams, against 734 ms and 729 ms for the alternatives it compared. The gateway is in public beta on Langfuse Cloud and can be self-hosted alongside a user's own Langfuse, but it always requires a Langfuse deployment, Cloud or self-hosted, as its control plane. Organizations choose between usage-only traces (model, tokens, cost) and full capture of inputs and outputs; on self-hosted Langfuse, admins can turn recording off so the gateway only routes calls. Langfuse says the gateway passes provider APIs through without translating them. It also notes that Claude Code must be logged out of a claude.ai subscription to use gateway credentials, and that subscription-backed connectors are then unavailable in Claude Desktop.

Generated from reports · updated 42 min ago

Report timeline

Follow the reports to see the event from different sides.

November 18
  1. Hacker News · AI
    Langfuse explains how it designed its AI gateway in Rust

    Langfuse published a design write-up of its AI Gateway, a proxy that sits in front of all model calls so platform teams can issue cross-provider keys, track cost per key, user and team, and record every call as a Langfuse trace. 2 ms to median time to first token at 100 concurrent streams versus 734 ms and 729 ms.

Heat for this event

Not enough continuous observations yet to draw a trend.