Skip to content

[SUPPORT] Configure Zstd compression level separately for native Parquet logs #19780

Description

@cshuo

Feature Description

What the feature achieves:

Allow native Parquet log files to use a Zstandard compression level that is independent of the global Hadoop parquet.compression.codec.zstd.level setting used by Parquet base files.

Why this feature is needed:

Native log files favor write latency and do not generally need the same compression ratio as base files. Today, changing the Hadoop Zstd level affects both file types, so users cannot select a faster level for native logs while retaining a higher compression level for base files.

User Experience

How users will use this feature:

  • Configure hoodie.logfile.parquet.compression.codec.zstd.level for native Parquet logs.
  • The setting defaults to 1.
  • Base files continue to use the global parquet.compression.codec.zstd.level setting.
  • The behavior is available through Spark, Flink, and Java writer paths.

Hudi RFC Requirements

RFC PR link: Not applicable.

Why an RFC isn't needed:

  • Does this change public interfaces/APIs? Yes, it adds an optional advanced configuration.
  • Does this change storage format? No.
  • Justification: This is a narrowly scoped writer configuration that changes compression tuning only and does not alter the Parquet or Hudi on-disk format.

Metadata

Metadata

Assignees

Labels

type:featureNew features and enhancements

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions