The apache/parquet-format repository contains the formal specification and Thrift definitions for Apache Parquet, an open-source columnar data file format designed for efficient storage and retrieval in the Hadoop ecosystem. The repository is written primarily in Thrift and serves as the authoritative reference for how Parquet files should be structured, encoded, and interpreted across different programming languages and analytics tools.
Apache Parquet was created to provide compressed, efficient columnar data representation to any project in the Hadoop ecosystem. The format is built from the ground up to handle complex nested data structures using the record shredding and assembly algorithm described in the Dremel paper, which the project considers superior to simple flattening of nested namespaces. The format supports very efficient compression and encoding schemes that can be specified on a per-column level, allowing flexibility as new compression techniques are invented and implemented.
The repository defines the hierarchical structure of Parquet files, which consist of one or more row groups, with each row group containing exactly one column chunk per column, and each column chunk containing one or more pages. Files begin with a four-byte magic number "PAR1", followed by column chunks organized by row group, with file metadata written at the end to enable single-pass writing. The format supports eight primitive types: BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BYTE_ARRAY, and FIXED_LEN_BYTE_ARRAY. Logical types extend these primitives to support additional data representations like strings, while maintaining minimal complexity in reader and writer implementations.
The specification covers critical technical details including nested encoding using definition and repetition levels based on the Dremel encoding scheme, null value handling through definition levels with run-length encoding, and data page structure with encoded values, definition levels, and repetition levels. The repository documents supported encodings and compression codecs in separate specification files.
Activity tracking shows this is an actively maintained project with significant community engagement.
The parquet-format project specifically focuses on format specifications and metadata definitions, while related projects like parquet-java provide implementation details and parquet-testing offers cross-language verification files. The repository can be built using Maven for Java resources and Make for C++ Thrift resources, with code generation available for any Thrift-supported language. This design ensures that Parquet implementations across different languages can reliably read and write each other's files while maintaining compatibility with the formal specification.
