Eclipse Aerospace uses PySpark custom data sources for flight-data ingestion
Start here
Eclipse Aerospace is using Databricks PySpark custom data sources to ingest nearly 20 years of proprietary flight data — over 3TB spanning more than a million flight hours — without always-on dedicated compute, according to a Databricks case study. The Rust-backed Python package delivered twice the benchmarked performance of a similar PyArrow-based pure-Python data source and nearly 20x that of a native Python implementation without Arrow, Databricks says. The API, introduced in Apache Spark 4.0, replaces Eclipse's prior Lambda-to-Avro pipeline. Eclipse and Databricks say the new path reduces latency, duplication and operational overhead while keeping the flexibility of Databricks Serverless. PySpark custom data sources are available in Databricks Runtime 15.4 LTS and above, or in serverless environment version 2.
Generated from reports · updated 13 h ago
Report timeline
Follow the reports to see the event from different sides.
- Databricks BlogEclipse Aerospace uses Databricks PySpark custom data sources for flight-data ingestion
Eclipse Aerospace is using Databricks PySpark custom data sources to ingest nearly 20 years of proprietary flight data, over 3TB spanning more than a million flight hours, without always-on dedicated compute. The Rust-backed Python package delivered twice the benchmarked performance of a similar PyArrow-based pure-Python data source and nearly 20x that of a native Python implementation without Arrow. The API, introduced in Apache Spark 4.0 and available in Databricks Runtime 15.4 LTS, replaces the prior Lambda-to-Avro pipeline.
Heat for this event
Not enough continuous observations yet to draw a trend.