Skip to content
Hot eventDeveloping

The Pragmatic Engineer Publishes Guide to Inference Engineering

1 reports1 reporting sourcesUpdated 8 h ago

Start here

AI digest

The Pragmatic Engineer published a guide on inference engineering for software engineers moving from frontier-lab APIs to open models, based on interviews with Modal, Fireworks AI and Baseten. It covers when switching makes sense for cost, latency control and fine-tuning, how to choose a model and provider using evals, and the path from replacing a multi-tenant API to running dedicated GPUs. The guide also details practical pitfalls, including long-tail time-to-first-token latency, KV cache and prompt cache behavior, and the caching costs that come with stateless APIs.

Generated from reports · updated 7 h ago

Report timeline

Follow the reports to see the event from different sides.

November 18
  1. The Pragmatic Engineer
    Inference engineering for software engineers: moving from frontier APIs to open models

    The Pragmatic Engineer interviewed Modal, Fireworks AI and Baseten to explain inference engineering for software engineers moving off frontier-lab APIs onto open models. The guide covers when to switch for cost, latency control and fine-tuning, how to pick a model and provider with evals, and the graduation path from multi-tenant API replacement to dedicated GPUs. It also details practical pitfalls such as long-tail TTFT latency, KV cache and prompt cache behavior, and stateless API caching costs.

Heat for this event

Current, comparable range
8
Peak, comparable range
10November 18 01:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.