Tới nội dung

Lĩnh vực

Kỹ thuật triển khai Mới nhất

Chạy mô hình nhanh hơn và rẻ hơn: tăng tốc suy luận, bộ nhớ và chi phí, phục vụ mô hình và lựa chọn hạ tầng.

62 tin chọn34 trong 30 ngày quaĐã thu thập 472

Cập nhật 13 tháng 11 08:32

Kỹ thuật triển khai · Tuyển chọn · trang 4

5 tháng 8, 2024Thứ HaiMục 61–62 trong 62
  1. Modal Blog62

    Modal replicates Large Language Monkeys: LLaMA 3.1 8B matches GPT-4o on HumanEval via search

    Modal's Charles Frye replicated the core claim of the Large Language Monkeys paper on a new dataset, showing that repeated sampling lifts LLaMA 3.1 8B from 66.4% pass@1 to 90.5% pass@100 and 95.1% pass@1000 on HumanEval, versus GPT-4o's reported 90.2% pass@1. The runs used vLLM on Modal serverless GPUs and Modal Sandboxes for parallel test execution, cost well under $50, and showed smooth, predictable scaling across three orders of magnitude.

    Vì sao đáng chú ý: A hands-on replication shows how repeated sampling with a small open model can approach GPT-4o on HumanEval, offering a concrete, low-cost recipe for trading compute for accuracy.

21 tháng 7, 2023Thứ Sáu
  1. Cursor Changelog62

    Cursor analyzes Llama-2-70B inference cost and latency against GPT-3.5

    Cursor measured Llama-2-70B inference on 2 80-GB A100 GPUs and found it over 3x cheaper than GPT-3.5 for prompt tokens but far more expensive for generated tokens at small batch sizes. Prompt processing costs about $0.00042 per 1K tokens versus GPT-3.5's $0.0015, while generation reaches $0.066 per 1K tokens at batch size 1 and only approaches GPT-3.5 pricing at large batches that push time to first token to several seconds. Cursor recommends open-source models for prompt-heavy tasks like classification and reranking, and closed-source APIs for generation-heavy workloads.

    Vì sao đáng chú ý: Cursor's measured cost and latency data shows when self-hosting Llama-2-70B beats GPT-3.5 and when it does not, offering a reusable framework for open-source model serving decisions.