ekzhu/datasketch

Datasketch is a Python library that implements probabilistic data structures designed to process and search very large datasets efficiently with minimal accuracy loss. The library provides implementations of MinHash, Weighted MinHash, HyperLogLog, and HyperLogLog++ for estimating Jaccard similarity, weighted Jaccard similarity, and cardinality across massive datasets. These sketching techniques allow users to compress large amounts of data into small probabilistic summaries that can be queried and compared rapidly.

Beyond the core sketching algorithms, datasketch includes several indexing structures that enable sub-linear query times on sketched data. The library offers MinHash LSH for Jaccard threshold queries, LSHBloom for similar threshold-based queries, MinHash LSH Forest for top-K similarity searches, MinHash LSH Ensemble for containment threshold queries, and HNSW for custom metric top-K searches. The MinHash LSH and LSH Ensemble indexes can scale to production environments by leveraging Redis or Cassandra as storage backends, making the library suitable for big data applications.

Enhancement requests and help-wanted issues represent the most common issue types, indicating active development and community interest.

Datasketch requires Python 3.9 or above, NumPy 1.11 or above, and Scipy as core dependencies. The library offers optional dependencies for Redis, Cassandra, and Bloom filter support, allowing users to install only the components they need. Installation is straightforward via pip, with specialized installation commands available for different feature sets.

The development workflow uses uv for Python package management, providing a modern alternative to traditional tools. Contributors are guided through a clear setup process that includes creating virtual environments, installing dependencies, and running tests with pytest. Code quality is maintained through ruff for linting and formatting, with the project following PEP 8 style guidelines.