Elasticsearch Hadoop is a Java-based integration library that enables real-time search and analytics capabilities from Elasticsearch to work natively within Hadoop ecosystems. The project provides native integration across multiple big data processing frameworks including Apache Hadoop MapReduce, Apache Hive, and Apache Spark, allowing users to read from and write to Elasticsearch clusters directly from their data processing jobs.
The library is designed with a focus on minimal dependencies and ease of deployment. It ships as a small, self-contained jar file of approximately 300 kilobytes with no external dependencies, requiring only network access to an Elasticsearch cluster via REST API. Users can add the jar to their job classpath through various methods including bundling, DistributedCache, or manual cluster provisioning. The project supports Hadoop 2.x and 3.x on YARN, Spark versions 3.0 through 3.4 with Scala 2.12 and 2.13 support depending on the Spark version, and maintains backward compatibility with older Elasticsearch versions, though matching version numbers between ES-Hadoop and Elasticsearch is recommended.
For MapReduce environments, the library provides dedicated InputFormat and OutputFormat classes named EsInputFormat and EsOutputFormat that handle reading and writing operations at the low level. For Apache Hive users, ES-Hadoop offers a storage handler that allows definition of external tables backed by Elasticsearch indices, with field mapping to JSON for communication with Elasticsearch. Hive users can read from and write to Elasticsearch using standard SQL-like syntax with configuration through TBLPROPERTIES.
Apache Spark integration is particularly comprehensive, providing native Java and Scala support through dedicated RDD classes for reading and methods for writing on any RDD. The library also supports Spark SQL, allowing users to work with Elasticsearch data through SQL queries. Configuration across all frameworks uses properties prefixed with "es", with a reserved "es.internal" namespace for library use.
Documentation labels, serialization issues, and bug reports represent the most active issue categories. The codebase is built using Gradle and requires JVM 8 or higher for compilation. The project is released under Apache License 2.0 and maintains connections with other Elastic projects including the main Elasticsearch repository.
