Lance is an open lakehouse format specifically designed for multimodal AI workloads, providing a complete file format, table format, and catalog specification that enables building lakehouses on object storage. Written primarily in Rust with Python bindings via PyO3 and Java bindings via JNI, Lance addresses fundamental performance gaps in existing data formats when applied to machine learning and AI tasks.
The format delivers several core capabilities tailored to AI workflows. It enables expressive hybrid search by combining vector similarity search, full-text search using BM25, and SQL analytics on the same dataset with accelerated secondary indices. For random access patterns common in ML training and data exploration, Lance achieves 100x faster performance than Parquet or Iceberg without sacrificing scan performance. The format natively supports multimodal data including images, videos, audio, text, and embeddings in a single unified format with efficient blob encoding and lazy loading. Data evolution is handled efficiently, allowing columns to be added with backfilled values without requiring full table rewrites, which is particularly valuable for ML feature engineering pipelines. Lance also provides zero-copy versioning with automatic ACID transactions, time travel capabilities, tags, and branches without requiring additional infrastructure.
The ecosystem integration is extensive, with compatibility across Apache Arrow, Pandas, Polars, DuckDB, Apache Spark, Ray, Trino, Apache Flink, and open catalog systems including Apache Polaris, Unity Catalog, and Apache Gravitino. The repository includes a comprehensive directory structure with core Rust implementation, Python bindings, Java bindings, and documentation source.
Performance benchmarks demonstrate Lance's strengths in AI-specific operations. Vector search on the SIFT dataset with 1M vectors of 128 dimensions achieves sub-millisecond average response times on consumer hardware. When compared to Parquet using the Oxford Pet dataset, Lance delivers 50-100x better performance for analytics queries and 100x improvement for batched random access operations. The format explicitly addresses the gap between traditional lakehouse formats designed for SQL analytics and the requirements of modern ML development cycles, which demand vector search for semantic retrieval, fast random access for sampling and exploration, multimodal data storage capabilities, efficient data evolution, and hybrid search combining multiple query types.
The repository shares contributors with github/gh-aw, solo-io/gloo, and longhorn/longhorn, indicating cross-project collaboration within the data infrastructure ecosystem.
