Gleam is a distributed map/reduce system written in pure Go that executes directed acyclic graphs of computations across multiple machines or locally.
Gleam addresses the challenge of running large-scale data processing workloads efficiently by implementing a distributed execution engine where computations flow through memory by default, with optional disk persistence. The system defines data processing as a DAG where datasets are vertices and computation steps are edges. User-defined operations can be written in Go, Unix pipe tools, or any streaming programs. The architecture separates concerns across a Driver (the user program defining the flow), a Master (coordinating resource allocation), Agents (running on compute machines), and Executors (the actual processes performing work). Data moves through memory and the network rather than touching disk unless explicitly configured, enabling natural backpressure and stream computation support.
Gleam suits teams building data processing pipelines in Go environments who want to avoid the garbage collection overhead and memory tuning complexity of JVM-based systems. The tool works both as a standalone local processor and as a distributed system, making it appropriate for development workflows that scale from single-machine testing to multi-node deployments without code changes. The memory efficiency comes from running each executor as a separate OS process rather than managing memory within a single runtime, and the master and agent servers themselves consume minimal resources. Go's simpler syntax compared to Scala or Java makes the codebase more accessible to maintain and customize.
The project shows consistent development activity with regular commits addressing bug fixes and feature improvements. The codebase maintains test coverage across the distributed execution pipeline. Documentation is maintained through both inline code documentation and a project wiki. The implementation demonstrates attention to performance optimization, particularly in merging multiple map-reduce steps and managing executor concurrency.
