Skip to content

perf(parquet): minor tweaks for iceberg write improvement - #595

Merged
zeroshade merged 2 commits into
apache:mainfrom
hamilton-earthscope:perf
Dec 9, 2025
Merged

perf(parquet): minor tweaks for iceberg write improvement#595
zeroshade merged 2 commits into
apache:mainfrom
hamilton-earthscope:perf

Conversation

@hamilton-earthscope

Copy link
Copy Markdown
Contributor

Overview

In apache/iceberg-go#643 a benchmark test suite is introduced to measure write performance to iceberg tables. This PR fixes a few hot spots in the parquet writer that improve write performance to iceberg tables across a variety of scenarios. These improvements should be broadly beneficial to anyone using arrow-go to write parquet.

Optimizations

1. Zstd Encoder Pooling

Implemented sync.Pool for zstd compression encoders in arrow-go/parquet/compress/zstd.go. Previously, every parquet file write allocated a new 96MB encoder, causing a lot of memory bloat and GC churn. Now encoders are pooled and reused across writes.

2. Column Path Caching

Pre-compute column path strings (e.g., "data.field.subfield") in arrow-go/parquet/schema/column.go during Column creation instead of rebuilding via strings.Join() on every access. Eliminates CPU overhead, especially effective for nested schemas.

Performance Results

These results are measured using the test suite linked above. No new performance benchmarks are added to arrow-go.

The metrics in the table below indicate 1. the write speed measured in records per second and 2. the volume of memory allocated during the test (not the number of allocations which these changes don't really affect)

Benchmark Before After Improvement
Simple 100K 734K rec/sec, 3.9 GB 2.5M rec/sec, 155 MB 3.5x faster, 96% less memory
Simple 2.5M 3.9M rec/sec, 9.3 GB 4.3M rec/sec, 1.9 GB 1.1x faster, 79% less memory
ListPrimitive 100K 496K rec/sec, 4.6 GB 1.3M rec/sec, 317 MB 2.7x faster, 93% less memory
ListPrimitive 2.5M 1.5M rec/sec, 14.2 GB 2.0M rec/sec, 3.3 GB 1.3x faster, 76% less memory
ListStruct 100K 377K rec/sec, 5.7 GB 927K rec/sec, 565 MB 2.4x faster, 90% less memory
ListStruct 2.5M 772K rec/sec, 19.7 GB 1.1M rec/sec, 5.0 GB 1.4x faster, 74% less memory
MapPrimitive 100K 446K rec/sec, 5.6 GB 995K rec/sec, 511 MB 2.2x faster, 91% less memory
MapPrimitive 2.5M 1.1M rec/sec, 18.0 GB 1.3M rec/sec, 4.7 GB 1.2x faster, 74% less memory
1000 partitions 35K rec/sec, 85.9 GB 341K rec/sec, 576 MB 9.6x faster, 99% less memory

Key improvements:

  • Small batches (100K): 2.2-3.5x throughput, 90-96% memory reduction
  • Large batches (2.5M): 1.1-1.4x throughput, 74-79% memory reduction
  • High partition counts see dramatic gains (9.6x faster for 1000 partitions)

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this! Looks great with massive benefits. Will merge once the CI finishes (assuming no issues)

@zeroshade
zeroshade merged commit 77348c9 into apache:main Dec 9, 2025
19 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants