Skip to content
Hacker News · AI·· 1 h agoAI score58

Langfuse explains how it designed its AI gateway in Rust

Designing the Langfuse AI Gateway

AI summary

Langfuse published a design write-up of its AI Gateway, a proxy that sits in front of all model calls so platform teams can issue cross-provider keys, track cost per key, user and team, and record every call as a Langfuse trace. 2 ms to median time to first token at 100 concurrent streams versus 734 ms and 729 ms.

Source:Hacker News · AI · langfuse.com