Redpanda Connect is a stream processor that moves data between a wide range of sources and sinks with support for hydration, enrichment, transformation, and filtering.
The tool solves the problem of building and operating data pipelines across heterogeneous systems. It uses a declarative YAML-based approach where a complete stream topology fits in a single configuration file. Pipelines can perform change-data-capture from databases like Postgres, MySQL, MongoDB, Oracle, and MSSQL, allowing database changes to flow as first-class events. Data transformation happens through Bloblang, a mapping language designed specifically for stream data. The tool guarantees at-least-once delivery by default using an in-process transaction model with no disk-persisted state, which means it can maintain delivery guarantees even through crashes or disk corruption without requiring operational complexity around state management.
Redpanda Connect suits teams building ETL and event-streaming infrastructure who need to connect cloud services, message brokers, databases, and HTTP endpoints. The connector catalog includes AWS services like DynamoDB, Kinesis, S3, and SQS; Azure Blob and Queue; GCP Pub/Sub and BigQuery; Kafka; NATS; RabbitMQ; Redis; Cassandra; Elasticsearch; and SQL databases. The tool runs as a single static binary or container image and is designed to be stateless and horizontally scalable, making it cloud-friendly. It exposes health check endpoints for orchestration, emits metrics to Statsd and Prometheus, and natively emits OpenTelemetry traces for end-to-end pipeline visibility.
The project maintains active development with regular updates to its connector catalog and core functionality. Documentation is comprehensive, covering general usage, the Bloblang language, and public Go APIs for building custom plugins. The codebase supports building components with external C library dependencies through optional build tags. Configuration tooling is included for discovery, debugging, and organization of pipelines.
