Feature Description
What the feature achieves:
Allow native Parquet log files to use a Zstandard compression level that is independent of the global Hadoop parquet.compression.codec.zstd.level setting used by Parquet base files.
Why this feature is needed:
Native log files favor write latency and do not generally need the same compression ratio as base files. Today, changing the Hadoop Zstd level affects both file types, so users cannot select a faster level for native logs while retaining a higher compression level for base files.
User Experience
How users will use this feature:
- Configure
hoodie.logfile.parquet.compression.codec.zstd.level for native Parquet logs.
- The setting defaults to
1.
- Base files continue to use the global
parquet.compression.codec.zstd.level setting.
- The behavior is available through Spark, Flink, and Java writer paths.
Hudi RFC Requirements
RFC PR link: Not applicable.
Why an RFC isn't needed:
- Does this change public interfaces/APIs? Yes, it adds an optional advanced configuration.
- Does this change storage format? No.
- Justification: This is a narrowly scoped writer configuration that changes compression tuning only and does not alter the Parquet or Hudi on-disk format.
Feature Description
What the feature achieves:
Allow native Parquet log files to use a Zstandard compression level that is independent of the global Hadoop
parquet.compression.codec.zstd.levelsetting used by Parquet base files.Why this feature is needed:
Native log files favor write latency and do not generally need the same compression ratio as base files. Today, changing the Hadoop Zstd level affects both file types, so users cannot select a faster level for native logs while retaining a higher compression level for base files.
User Experience
How users will use this feature:
hoodie.logfile.parquet.compression.codec.zstd.levelfor native Parquet logs.1.parquet.compression.codec.zstd.levelsetting.Hudi RFC Requirements
RFC PR link: Not applicable.
Why an RFC isn't needed: