SeaweedFS is a distributed storage system written in Go designed to efficiently store and serve billions of files at scale. The project implements a blob store architecture based on Facebook's Haystack design paper, with the core innovation of spreading file metadata across volume servers rather than centralizing it in a master node. This approach achieves O(1) disk seek performance for file access, typically requiring just a single disk read operation, while maintaining only 40 bytes of metadata overhead per file.
The system consists of two main components. The blob store layer handles efficient storage of small files with support for multiple replication levels, rack and data center awareness, automatic compression based on MIME type, and automatic compaction to reclaim space after deletion. The optional Filer component adds directory support and POSIX attributes on top of the blob store, functioning as a stateless, linearly-scalable server with pluggable metadata backends including MySQL, PostgreSQL, Redis, Cassandra, HBase, MongoDB, Elasticsearch, and various other databases.
SeaweedFS implements erasure coding inspired by Facebook's f4 Warm BLOB Storage System, enabling rack-aware 10.4 erasure coding to reduce storage costs while maintaining availability. The system supports transparent cloud integration through tiered storage, allowing hot data to remain on local clusters while warm data resides in cloud storage with maintained O(1) access time. This hybrid approach provides both fast local access and elastic cloud capacity while minimizing cloud API costs.
The Filer layer provides extensive functionality including S3-compatible API access, Hadoop filesystem compatibility for integration with Spark and Flink, POSIX FUSE mounting for direct filesystem access, WebDAV support for mapped drives on Mac and Windows, AES256-GCM encryption for secure storage, and support for super large files in the tens of terabyte range. Additional features include active-active cross-cluster replication, asynchronous backup to cloud providers like Amazon S3 and Google Cloud Storage, and cloud drive mounting with local caching.
Kubernetes integration is provided through a Container Storage Interface driver and a dedicated SeaweedFS Operator. The system supports automatic master server failover eliminating single points of failure, flexible capacity expansion without mandatory data rebalancing, and customizable storage tiers to balance performance and cost.
The project is Apache-licensed open source with development supported through Patreon sponsorships. An enterprise version is available at seaweedfs.com. Quick start options include a single binary deployment with weed mini for development and testing, Docker containers for S3 API access, and production-grade weed server deployments. The system scales horizontally by simply adding more volume servers, making it suitable for petabyte-scale data lake deployments.
