milvus-io/talon

Talon is a distributed object store cache written in Rust that eliminates per-file metadata rows by deriving namespace, file sizes, modification times, and directory structure directly from the object store's key listing.

Most caching filesystems maintain a metadata database with one row per file, which becomes a scaling bottleneck and single point of failure. Talon inverts this approach: any fact that can be rebuilt by listing the object store is not stored anywhere. File namespace, sizes, and directory structure come from the object store itself. Only features that genuinely cannot derive their state—such as hard links, POSIX locking, and write-back caching—use an optional, deliberately sparse metadata store. This design means a singly-linked, unlocked file costs zero metadata records, keeping the metadata store bounded by the features actually in use rather than by data volume. The tool reads without copying, using sendfile from NVMe to socket and splice from socket to NVMe on fill, driven by io_uring. A plain S3 client can read anything Talon writes, eliminating lock-in and migration friction.

Talon suits workloads where scaling and operational simplicity matter more than advanced POSIX semantics. It works well for read-heavy access patterns over large object stores where metadata overhead is a concern. The tool is appropriate for teams comfortable with optional metadata for advanced features and willing to trade some POSIX guarantees for a stateless, horizontally scalable cache. Coordinators run active-active with no recovery burden; losing a worker means a cache miss, not an outage. Access is available through FUSE mounts or language-specific SDKs in Python, Java, C, and Rust, with batch prewarming operations to load files across responsible workers.

Development activity shows sustained focus on performance measurement and real-world validation. The project publishes both loopback and cross-network throughput figures together, treating a change that improves one but not the other as unmeasured. Architectural decisions are documented through formal decision records, including the optional metadata store design. The codebase maintains multiple client implementations across different languages, indicating ongoing investment in usability across diverse environments.