Hacker News · AI·· 1 h agoAI score58
Langfuse explains how it designed its AI gateway in Rust
Designing the Langfuse AI Gateway
AI summary
Langfuse published a design write-up of its AI Gateway, a proxy that sits in front of all model calls so platform teams can issue cross-provider keys, track cost per key, user and team, and record every call as a Langfuse trace. 2 ms to median time to first token at 100 concurrent streams versus 734 ms and 729 ms.
Source:Hacker News · AI · langfuse.com