PLATFORMS Signal 475
AWS Glue 6.0 goes GA on Spark 4.1 with 30% lower price and Iceberg v3
↗
AWS Glue 6.0 is generally available, built on Spark 4.1 with Python 3.12 and Scala 2.13, offering 30% lower pricing and full Apache Iceberg v3 support.
For engineers running ETL on AWS Glue, this release cuts costs by 30% and brings a modernized runtime with faster performance. The full Iceberg v3 support enables handling semi-structured data without flattening schemas, and the real-time streaming mode offers single-digit millisecond latency for stateless workloads. Migration is straightforward since no API changes are required.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
AWS Glue 6.0 is generally available with 30% lower pricing and full Apache Iceberg v3 support.
It runs on Spark 4.1, Python 3.12, and Scala 2.13, adding VARIANT shredding, geometry types, and nanosecond timestamps.
No API changes are needed; select Glue 6.0 via the existing --glue-version parameter or use the auto-upgrade feature.
THE READ
What the cluster adds up to.
AWS Glue 6.0 is now generally available, marking a major update to the managed ETL service. The new version is built on a fully modernized runtime with Apache Spark 4.1, Python 3.12, and Scala 2.13, and it delivers 30% lower pricing than previous versions. This price reduction is a direct incentive for existing users to migrate, while the runtime upgrade promises faster performance for data processing jobs.
The headline feature is full support for Apache Iceberg v3, built on Iceberg 1.11.0. This includes the VARIANT data type with shredding, which allows storing and querying semi-structured data like JSON and logs without flattening schemas. Additional Iceberg v3 capabilities include geometry and geography data types for spatial processing, nanosecond-precision timestamps for high-frequency workloads, and unknown type handling to tolerate evolving schemas. These features eliminate the need for custom parsing code and reduce pipeline breakage when schemas change.
Spark 4.1 brings several improvements to Glue 6.0. Declarative pipelines let engineers specify desired data transformations while the engine handles execution order and optimization, reducing manual orchestration. Arrow-native Python UDFs and UDTFs remove serialization overhead between Python and the JVM, improving PySpark performance. A new real-time streaming mode achieves single-digit millisecond latency for stateless streaming use cases, enabling low-latency event processing and data routing.
Adopting Glue 6.0 requires no API changes; users simply select the new version via the --glue-version parameter in existing APIs or through the console. Existing jobs can be migrated using the Spark upgrade agent or the auto-upgrade feature. However, the real-time streaming mode is limited to stateless workloads, so stateful streaming applications will not benefit from this specific capability. Additionally, the new Iceberg v3 features are only available on Glue 6.0, meaning teams must upgrade to access them, and older job versions will not receive these enhancements.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER