Microsoft Fabric Blog announces the general availability of Azure Synapse Runtime for Apache Spark 3.5, helping customers leverage updated Spark features before migrating to Microsoft Fabric Spark.

Azure Synapse Runtime for Apache Spark 3.5 Now Generally Available

Microsoft has officially released Azure Synapse Runtime for Apache Spark 3.5, making it available for Azure Synapse Spark customers to use for production workloads. This update provides feature enhancements from Apache Spark 3.5 and Delta Lake 3.2, enabling organizations to benefit from the latest updates in Spark performance, reliability, and compatibility.

Key Announcements

  • General Availability: Customers can now create and run workloads using Azure Synapse Runtime for Apache Spark 3.5.
  • Core Upgrades: The release incorporates major improvements from Apache Spark 3.5 and Delta Lake 3.2.
  • Production Ready: Suitable for mission-critical and production data engineering, ETL, and analytics workloads.
  • Release Notes: For a comprehensive list of changes and features, refer to the official release notes for Apache Spark 3.5.

Migration Information

Customers are encouraged to review migration guidelines between Spark 3.4 and 3.5 to identify necessary modifications to existing jobs, applications, and notebooks.

Looking Forward: Transition to Microsoft Fabric Spark

While continuing support for Azure Synapse Spark, Microsoft recommends organizations plan for migration to Microsoft Fabric Spark to take advantage of:

  • Native Execution Engine (NEE): Enhanced query performance at no extra cost (Learn more).
  • Starter Pools: Faster Spark session creation (Details).
  • Unified Lakehouse Security: Manage Row-Level Security (RLS) and Column-Level Security (CLS) easily.
  • Materialized Views: Efficient and flexible data options for analytics (Overview).

To learn more about this runtime, see Azure Synapse Runtime for Apache Spark 3.5 documentation.

Summary

The general availability of Azure Synapse Runtime for Apache Spark 3.5 marks a significant update for data engineers and analytics teams using Azure. Current users should plan for eventual migration to Microsoft Fabric Spark to maximize access to the latest performance and security advancements.

This post appeared first on “Microsoft Fabric Blog”. Read the entire article here