Deequ is a library built on Apache Spark for defining unit tests for data that measure data quality in large datasets.
Deequ addresses the problem of validating data quality at scale by providing a framework to express data quality checks as assertions. It works by allowing developers to define constraints and checks on tabular data—such as CSV files, database tables, or Spark dataframes—and then execute these checks against large distributed datasets. The library is designed to catch data errors early before they propagate to downstream systems or machine learning algorithms.
Teams working with large-scale data pipelines should consider Deequ if they need systematic validation of data quality across billions of rows. It suits projects where data lives in distributed filesystems or data warehouses and where early error detection is critical. The tool works with any tabular data that can be represented as a Spark dataframe. Python users have access to PyDeequ, a separate Python interface for the library. Deequ requires Java 11 for recent releases and is built against specific Apache Spark versions, so you must select the artifact suffix matching your Spark version—the README provides a compatibility table covering Spark 3.1 through 3.5.
The project maintains active development with regular releases aligned to new Spark versions. The codebase shows consistent attention to compatibility across multiple Spark releases, with legacy support maintained in a separate branch for older versions. The project accepts contributions and maintains documentation including executable examples demonstrating basic usage patterns.
