SynapseML is an open-source library developed by Microsoft that simplifies the creation of massively scalable machine learning pipelines built on Apache Spark. Previously known as MMLSpark, the project provides simple, composable, and distributed APIs for diverse machine learning tasks including text analytics, vision, anomaly detection, and deep learning. The library shares the same API as SparkML/MLLib, enabling seamless integration into existing Apache Spark workflows while supporting multiple programming languages including Python, R, Scala, Java, and .NET.
The repository is written primarily in Scala and requires Scala 2.12, Spark 3.4 or higher, and Python 3.8 or higher. SynapseML abstracts over various databases, file systems, and cloud data stores to simplify experiments regardless of data location. The library supports training and evaluation on single-node, multi-node, and elastically resizable clusters, allowing users to scale work without wasting resources. It is usable across Microsoft Fabric, Synapse Analytics, Databricks, and standalone Python environments.
The project encompasses a comprehensive set of features organized around specific machine learning domains. Vowpal Wabbit on Spark provides fast, sparse, and effective text analytics capabilities. The Cognitive Services for Big Data feature enables leveraging Microsoft Cognitive Services at unprecedented scales within existing SparkML pipelines. LightGBM on Spark allows training of gradient boosted machines, while Spark Serving enables serving any Spark computation as a web service with sub-millisecond latency. HTTP on Spark integrates Spark with the HTTP protocol for distributed microservice orchestration, and ONNX on Spark provides distributed and hardware-accelerated model inference. The library also includes Responsible AI tools for understanding opaque-box models and measuring dataset biases, along with Spark Binding Autogeneration for automatically generating PySpark and SparklyR bindings.
Additional specialized features include Isolation Forest on Spark for distributed nonlinear outlier detection, CyberML for machine learning tools focused on cyber security, and Conditional KNN for scalable KNN models with conditional queries. The current version is 1.1.3, with comprehensive API documentation available for both Scala and PySpark.
