Datasketch is a Python library that implements probabilistic data structures designed to process and search very large datasets efficiently with minimal accuracy loss. The library provides implementations of MinHash, Weighted MinHash, HyperLogLog, and HyperLogLog++ for estimating Jaccard similarity, weighted Jaccard similarity, and cardinality across massive datasets. These sketching techniques allow users to compress large amounts of data into small probabilistic summaries that can be queried and compared rapidly.
Beyond the core sketching algorithms, datasketch includes several indexing structures that enable sub-linear query times on sketched data. The library offers MinHash LSH for Jaccard threshold queries, LSHBloom for similar threshold-based queries, MinHash LSH Forest for top-K similarity searches, MinHash LSH Ensemble for containment threshold queries, and HNSW for custom metric top-K searches. The MinHash LSH and LSH Ensemble indexes can scale to production environments by leveraging Redis or Cassandra as storage backends, making the library suitable for big data applications.
The repository is maintained primarily by ekzhu, who has logged 33 events in issue and pull request activity tracked by GitGenius. The project shows moderate engagement with contributors like 123epsilon and chris-ha458 each contributing 6 events. Enhancement requests and help-wanted issues represent the most common issue types, indicating active development and community interest. The median response latency for issues and pull requests is 711.5 hours, though the mean of 7146.9 hours suggests occasional longer-term discussions on complex topics.
Datasketch requires Python 3.9 or above, NumPy 1.11 or above, and Scipy as core dependencies. The library offers optional dependencies for Redis, Cassandra, and Bloom filter support, allowing users to install only the components they need. Installation is straightforward via pip, with specialized installation commands available for different feature sets.
The development workflow uses uv for Python package management, providing a modern alternative to traditional tools. Contributors are guided through a clear setup process that includes creating virtual environments, installing dependencies, and running tests with pytest. Code quality is maintained through ruff for linting and formatting, with the project following PEP 8 style guidelines. The repository is linked to other major projects including microsoft/autogen, microsoft/agent-framework, and tensorflow/tensorflow through overlapping contributors, suggesting the library serves as a foundational component in broader data processing ecosystems.
