Lance is an open lakehouse format specifically designed for multimodal AI workloads, providing a complete file format, table format, and catalog specification that enables building lakehouses on object storage. Written primarily in Rust with Python bindings via PyO3 and Java bindings via JNI, Lance addresses fundamental performance gaps in existing data formats when applied to machine learning and AI tasks.
The format delivers several core capabilities tailored to AI workflows. It enables expressive hybrid search by combining vector similarity search, full-text search using BM25, and SQL analytics on the same dataset with accelerated secondary indices. For random access patterns common in ML training and data exploration, Lance achieves 100x faster performance than Parquet or Iceberg without sacrificing scan performance. The format natively supports multimodal data including images, videos, audio, text, and embeddings in a single unified format with efficient blob encoding and lazy loading. Data evolution is handled efficiently, allowing columns to be added with backfilled values without requiring full table rewrites, which is particularly valuable for ML feature engineering pipelines. Lance also provides zero-copy versioning with automatic ACID transactions, time travel capabilities, tags, and branches without requiring additional infrastructure.
The ecosystem integration is extensive, with compatibility across Apache Arrow, Pandas, Polars, DuckDB, Apache Spark, Ray, Trino, Apache Flink, and open catalog systems including Apache Polaris, Unity Catalog, and Apache Gravitino. The repository includes a comprehensive directory structure with core Rust implementation, Python bindings, Java bindings, and documentation source.
Performance benchmarks demonstrate Lance's strengths in AI-specific operations. Vector search on the SIFT dataset with 1M vectors of 128 dimensions achieves sub-millisecond average response times on consumer hardware. When compared to Parquet using the Oxford Pet dataset, Lance delivers 50-100x better performance for analytics queries and 100x improvement for batched random access operations. The format explicitly addresses the gap between traditional lakehouse formats designed for SQL analytics and the requirements of modern ML development cycles, which demand vector search for semantic retrieval, fast random access for sampling and exploration, multimodal data storage capabilities, efficient data evolution, and hybrid search combining multiple query types.
GitGenius tracking shows the repository has maintained steady activity with 6754 stars as of the most recent check. The project demonstrates strong community engagement with a median issue and pull request response latency of 0.8 hours across 1817 tracked items, though mean latency is significantly higher at 2557.6 hours, indicating some older items in the backlog. The most active issue labels are enhancement with 228 items, bug with 145 items, and rust with 140 items. Primary contributors wjones127, Xuanwo, and westonpace have driven 1272, 1180, and 737 events respectively. The repository shares contributors with github/gh-aw, solo-io/gloo, and longhorn/longhorn, indicating cross-project collaboration within the data infrastructure ecosystem.
