lakesoul-io/LakeSoul

LakeSoul is a cloud-native lakehouse framework that provides end-to-end data ingestion, concurrent updates, incremental analytics, and vector search capabilities for BI and AI workloads.

LakeSoul addresses the complexity of assembling production lakehouse infrastructure by delivering a batteries-included platform rather than just a table format. It solves the problem of fragmented implementations across different languages and compute engines by implementing its metadata management and file format IO entirely in Rust, then providing idiomatic bindings for Java, Python, and C++. This single high-performance core ensures that whether data is queried via Spark, streamed through Flink, or used for model training with PyTorch or Ray, all engines share identical ACID guarantees, upsert semantics, and read performance. The tool includes automated multi-level compaction, fine-grained role-based access control with S3 proxy support, high-performance OLAP queries, vector retrieval, full-text retrieval, and native multimodal data processing powered by Ray and Daft.

Teams should adopt LakeSoul when they need a production-ready lakehouse without assembling separate catalog, compaction, and authentication services. It suits organizations running mixed workloads across Spark, Flink, and Python-based ML frameworks who want consistency guarantees across all engines. The project supports multiple storage systems including HDFS and S3, and offers three physical file format modes: vortex-compact as the default, parquet, and vortex for multimodal data and vector embeddings. Unlike solutions that require maintaining per-language or per-engine reimplementations of table format logic, this approach eliminates behavioral divergences between bindings.

The project maintains active test coverage across multiple compute frameworks with dedicated test workflows for Maven, Flink CDC, native builds, and Python. Development activity spans the full stack from Rust core implementation through language bindings and integrations with major data processing ecosystems.