Skip to content

CoreFeatures.md

Latest commit

 

History

History
178 lines (146 loc) · 7.06 KB

CoreFeatures.md

File metadata and controls

178 lines (146 loc) · 7.06 KB
Raw

Parquet Core Features

This document lists the core features for each parquet-format release. This list is a subset of the features which parquet-format makes available.

Purpose

The list of core features for a certain release makes a compliance level for implementations. If a writer implementation claims that it is at a certain compliance level then it must use only features from the core feature list of that parquet-format release. If a reader implementation claims the same if must implement all of the listed features. This way it is easier to ensure compatibility between the different parquet implementations.

We cannot and don't want to stop our clients to use any features that are not on this list but it shall be highlighted that using these features might make the written parquet files unreadable by other implementations. We can say that the features available in a parquet-format release (and one of the implementations of it) and not on the core feature list are experimental.

Versioning

This document is versioned by the parquet-format releases which follows the scheme of semantic versioning. It means that no feature will be deleted from this document under the same major version. (We might deprecate some, though.) Because of the semantic versioning if one implementation supports the core features of the parquet-format release a.b.x it must be able to read any parquet files written by implementations supporting the release a.d.y where b >= d.

If a parquet file is written according to a released version of this document it might be a good idea to write this version into the field compliance_level in the thrift object FileMetaData.

Adding new features

The idea is to only include features which are specified correctly and proven to be useful for everyone. Because of that we require to have at least two different implementations that are released and widely tested. We also require to implement interoperability tests for that feature to prove one implementation can read the data written by the other one and vice versa.

Core feature list

This list is based on the parquet thrift file where all the data structures we might use in a parquet file are defined.

File structure

All of the required fields in the structure (and sub-structures) of FileMetaData must be set according to the specification. The following page types are supported:

  • Data page V1 (see DataPageHeader)
  • Dictionary page (see DictionaryPageHeader)

TODO: list optional fields that must be filled properly.

Types

Primitive types

The following primitive types are supported

  • BOOLEAN
  • INT32
  • INT64
  • FLOAT
  • DOUBLE
  • BYTE\_ARRAY
  • FIXED\_LEN\_BYTE\_ARRAY

NOTE: The primitive type INT96 is deprecated so it is intentionally not listed here.

Logical types

The logical types are practically annotations helping to understand the related primitive type (or structure). Originally we have had the ConvertedType enum in the thrift file representing all the possible logical types. After a while we realized it is hard to extend and so introduced the LogicalType union. For backward compatibility reasons we allow to use the old ConvertedType values according to the specified rules but we expect that the logical types in the file schema are defined with LogicalType objects.

The following LogicalTypes are supported:

  • STRING
  • MAP
  • LIST
  • ENUM
  • DECIMAL (for which primitives?)
  • DATE
  • TIME: (Which unit, utc?)
  • TIMESTAMP: (Which unit, utc?)
  • INTEGER: (all bitwidth 8, 16, 32, 64) (unsigned?)
  • UNKNOWN (?)
  • JSON (?)
  • BSON (?)
  • UUID (?)

NOTE: The old ConvertedType INTERVAL has no representation in LogicalTypes. This is becasue INTERVAL is deprecated so we do not include it in this list.

Encodings

The following encodings are supported:

  • PLAIN
    parquet-mr: Basically all value types are written in this encoding in case of V1 pages
  • PLAIN_DICTIONARY (?)
    parquet-mr: As per the spec this encoding is deprecated while we still use it for V1 page dictionaries.
  • RLE
    parquet-mr: Used for both V1 and V2 pages to encode RL and DL and for BOOLEAN values in case of V2 pages
  • DELTA_BINARY_PACKED (?)
    parquet-mr: Used for V2 pages to encode INT32 and INT64 values.
  • DELTA_LENGTH_BYTE_ARRAY (?)
    parquet-mr: Not used directly
  • DELTA_BYTE_ARRAY (?)
    parquet-mr: Used for V2 pages to encode BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY values
  • RLE_DICTIONARY (?)
    parquet-mr: Used for V2 page dictionaries
  • BYTE_STREAM_SPLIT (?)
    parquet-mr: Not used by default; can be used only via explicit configuration

NOTE: BIT_PACKED is deprecated and not used directly (boolean values are encoded with this under PLAIN) so not included in this list.

Compression

The following compression algorithms are supported (including UNCOMPRESSED).

  • SNAPPY
  • GZIP
  • BROTLI (?)
  • LZ4 (?)
  • ZSTD (?)

Statistics

Statistics are not required for reading data but incorrect or under specified statistics implementation can cause data loss. still list these features as wrongly specified/implemented statistics can still cause losing data unnoticed. The following features related to statistics are supported.

  • The row group level min/max values: The fields min\_value and max\_value shall be used in the Statistics object according to the specification
  • Column Index

NOTE: Writing page level statistics to the data page headers is not required.

The list of column\_orders in FileMetaData must be set according to the notes. See the special handlings required for floating point numbers at ColumnOrder.