SynapseML is an open-source library developed by Microsoft that simplifies the creation of massively scalable machine learning pipelines built on Apache Spark. Previously known as MMLSpark, the project provides simple, composable, and distributed APIs for diverse machine learning tasks including text analytics, vision, anomaly detection, and deep learning. The library shares the same API as SparkML/MLLib, enabling seamless integration into existing Apache Spark workflows while supporting multiple programming languages including Python, R, Scala, Java, and .NET.
The repository is written primarily in Scala and requires Scala 2.12, Spark 3.4 or higher, and Python 3.8 or higher. SynapseML abstracts over various databases, file systems, and cloud data stores to simplify experiments regardless of data location. The library supports training and evaluation on single-node, multi-node, and elastically resizable clusters, allowing users to scale work without wasting resources. It is usable across Microsoft Fabric, Synapse Analytics, Databricks, and standalone Python environments.
The project encompasses a comprehensive set of features organized around specific machine learning domains. Vowpal Wabbit on Spark provides fast, sparse, and effective text analytics capabilities. The Cognitive Services for Big Data feature enables leveraging Microsoft Cognitive Services at unprecedented scales within existing SparkML pipelines. LightGBM on Spark allows training of gradient boosted machines, while Spark Serving enables serving any Spark computation as a web service with sub-millisecond latency. HTTP on Spark integrates Spark with the HTTP protocol for distributed microservice orchestration, and ONNX on Spark provides distributed and hardware-accelerated model inference. The library also includes Responsible AI tools for understanding opaque-box models and measuring dataset biases, along with Spark Binding Autogeneration for automatically generating PySpark and SparklyR bindings.
Additional specialized features include Isolation Forest on Spark for distributed nonlinear outlier detection, CyberML for machine learning tools focused on cyber security, and Conditional KNN for scalable KNN models with conditional queries. The current version is 1.1.3, with comprehensive API documentation available for both Scala and PySpark.
GitGenius activity data reveals the repository maintains active development with a median issue and pull request response latency of 0.0 hours across 85 tracked items, though the mean response time is 6403.0 hours, indicating some longer-term issues. The most active issue labels are triage with 62 occurrences, bug with 52 occurrences, and area/lightgbm with 12 occurrences. Key contributors tracked by GitGenius include imatiach-msft with 32 events, stupidoge with 22 events, and memoryz with 16 events. The repository shares overlapping contributors with related projects including deepspeedai/deepspeed, lightgbm-org/lightgbm, and vllm-project/vllm, indicating integration within a broader ecosystem of machine learning and distributed computing projects.
