Cursor analyzes Llama-2-70B inference cost and latency against GPT-3.5
Cursor measured Llama-2-70B inference on 2 80-GB A100 GPUs and found it over 3x cheaper than GPT-3.5 for prompt tokens but far more expensive for generated tokens at small batch sizes. Prompt processing costs about $0.00042 per 1K tokens versus GPT-3.5's $0.0015, while generation reaches $0.066 per 1K tokens at batch size 1 and only approaches GPT-3.5 pricing at large batches that push time to first token to several seconds. Cursor recommends open-source models for prompt-heavy tasks like classification and reranking, and closed-source APIs for generation-heavy workloads.
Why it matters: Cursor's measured cost and latency data shows when self-hosting Llama-2-70B beats GPT-3.5 and when it does not, offering a reusable framework for open-source model serving decisions.