Eclipse Aerospace uses PySpark custom data sources for flight-data ingestion

Eclipse Aerospace is using Databricks PySpark custom data sources to ingest nearly 20 years of proprietary flight data — over 3TB spanning more than a million flight hours — without always-on dedicated compute, according to a Databricks case study. The Rust-backed Python package delivered twice the benchmarked performance of a similar PyArrow-based pure-Python data source and nearly 20x that of a native Python implementation without Arrow, Databricks says. The API, introduced in Apache Spark 4.0, replaces Eclipse's prior Lambda-to-Avro pipeline. Eclipse and Databricks say the new path reduces latency, duplication and operational overhead while keeping the flexibility of Databricks Serverless. PySpark custom data sources are available in Databricks Runtime 15.4 LTS and above, or in serverless environment version 2.

สร้างจากรายงาน · อัปเดต 14 ชม. ที่แล้ว

ยังมีการเฝ้าดูต่อเนื่องไม่พอจะวาดแนวโน้ม