feat(metadata): add standalone metadata table services tool - #19731
Open
cshuo wants to merge 1 commit into
Open
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19731 +/- ##
============================================
- Coverage 78.29% 78.07% -0.23%
+ Complexity 33861 33811 -50
============================================
Files 2541 2544 +3
Lines 141676 141981 +305
Branches 17172 17227 +55
============================================
- Hits 110928 110854 -74
- Misses 23039 23328 +289
- Partials 7709 7799 +90
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
cshuo
force-pushed
the
mdt_table_service_tool
branch
2 times, most recently
from
August 28, 2026 03:05
abdf2b4 to
ddf5cf8
Compare
cshuo
marked this pull request as ready for review
August 28, 2026 03:05
cshuo
force-pushed
the
mdt_table_service_tool
branch
from
September 1, 2026 01:46
ddf5cf8 to
62ad906
Compare
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Closes #19025.
Metadata table services currently run through
HoodieBackedTableMetadataWriter#performTableServices, which requires a data-table write even when the only goal is to maintain the metadata table. This can lead users to generate empty commits to trigger MDT compaction, log compaction, cleaning, and archival.This change provides a standalone Spark-submit tool and reusable metadata-writer APIs so an external service can schedule and execute MDT table services without writing to the data table. Scheduling and timeline publication are serialized with data-table writers through the shared data-table lock, while expensive compaction execution uses the MDT OCC path added by #18295.
Summary and Changelog
HoodieMetadataTableServicesToolunderhudi-utilitieswith schedule, execute, and schedule-and-execute modes for MDT compaction, log compaction, cleaning, and archival.scheduleTableServicesandexecuteTableServicesAPIs, backed byMetadataTableServiceRequest, toHoodieTableMetadataWriter.hoodie.metadata.table.service.manager.schedule.actionswhile retaining the existing execution delegation behavior.Impact
This adds public metadata-writer APIs, a new Spark-submit utility, and a new advanced scheduling-delegation configuration. Existing inline table-service behavior remains the default. Users can opt into external MDT maintenance without producing empty data-table commits.
Scheduling and short timeline publication/finalization sections contend on the existing data-table lock. Compaction and log-compaction execution do not hold that lock for the full execution duration.
Risk Level
Medium. The change affects MDT table-service orchestration and concurrent-writer locking. The implementation preserves the existing inline order, keeps table-version-six fallback behavior, reuses existing MDT write-client APIs, and includes focused coverage for request modes, delegation, lock derivation, failure propagation, and table-version-specific scheduling.
Documentation Update
Document the new
HoodieMetadataTableServicesToolinvocation and scheduling-delegation configuration before the feature is finalized.Contributor's checklist