Spark Dependencies is a Spark job that analyzes distributed traces to extract and store service dependency links for the Jaeger distributed tracing platform.
The tool solves the problem of understanding service topology in microservice architectures by processing trace data collected by Jaeger. It parses all spans from a given day, identifies relationships between services based on span data, and stores the resulting dependency graph for visualization in the Jaeger UI. The job operates on UTC-based daily batches, allowing operators to reprocess specific days as needed. It is required for production Jaeger deployments, though the all-in-one distribution includes built-in dependency analysis and does not require this separate job.
Operators running Jaeger in production should adopt this tool. It suits any deployment using Cassandra, Elasticsearch, or OpenSearch as the trace storage backend. The project provides Docker images with storage-specific variants that automatically configure the appropriate backend, eliminating manual storage selection. It can also be run as a standalone Java executable for environments where containerization is not preferred. Configuration is handled through environment variables, supporting customization of the Spark master, processing date, peer service identification, JVM memory settings, and logging levels.
The project maintains active development with regular updates to support multiple Elasticsearch versions and OpenSearch, reflecting ongoing compatibility work with evolving storage backends. Docker images are published with variant-specific tags corresponding to different storage systems, indicating sustained attention to deployment patterns. The codebase includes configuration flexibility through environment variables and Java options, suggesting responsiveness to operational requirements. Documentation covers both container and executable deployment modes with concrete examples for each supported storage backend.
