[spark] Let MSCK see the null partition of a value-only format table - #9521
Open
sundapeng wants to merge 1 commit into
Open
[spark] Let MSCK see the null partition of a value-only format table#9521sundapeng wants to merge 1 commit into
sundapeng wants to merge 1 commit into
Conversation
`__DEFAULT_PARTITION__` is `partition.default-name`, the value Paimon substitutes when a dynamic partition column is null or empty. It is the same value whatever the path layout. What the layout decides is the directory name: with `key=value` the null partition lives in `dt=__DEFAULT_PARTITION__`, whose first character is `d`; with a value-only layout the directory name is the value itself, so it begins with `_` and the generic hidden-directory rule swallows it. `PartitionPathUtils.isHiddenFile` already knows this and spares the directory whose name equals the table's default partition name, but only when it is told what that name is. `FormatTablePartitionRepair` called the five-argument `searchPartSpecAndPaths`, which passes `defaultPartValue` as null, so the rescue never applied and MSCK never saw the directory. Two ways to lose data followed: - `MSCK REPAIR TABLE ... ADD PARTITIONS` never registered the null partition; - `MSCK REPAIR TABLE ... SYNC PARTITIONS` read it as "registered but the directory is gone" and unregistered a partition that still holds data. Later queries then skip it without a word. `listStatusRecursively` applies the same rule while descending, so a null value on a non-leaf level hid the whole subtree rather than one directory. The scan path already gets this right: `FileSystemSplitEnumerator` passes `table.defaultPartName()` to the eight-argument overload. Repair now does the same. `partitionFilter` and `partitionType` stay null on purpose — repair diffs raw directory names, and handing it a partition type would reintroduce the cast that rewrites `month=01` to `1` and no longer round-trips to the real directory. The registered spec keeps the literal `__DEFAULT_PARTITION__`, which is what `FormatTableCommit` writes through the same `extractPartitionSpecFromPathOnlyValue` helper, so both sides of the diff agree and a second SYNC stays a no-op. Tests cover the missed registration, the wrongful unregistration, a subtree under a null non-leaf level, a table that overrides `partition.default-name` (hardcoding the literal passes every other case), and a key=value layout whose value starts with `_`, which is unaffected because the underscore sits on the value rather than on the first character of the directory name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fix
MSCK REPAIR TABLEfor catalog-managed Format Tables that use value-only partition paths.The default partition directory (normally
__DEFAULT_PARTITION__) starts with_. Repair did not pass the configured default partition name to filesystem discovery, so that directory was treated as hidden.ADD PARTITIONScould not discover it, whileSYNCorDROP PARTITIONScould unregister a live partition and make its rows invisible even though the data files remained.Pass the table's default partition name to the existing discovery overload. Keep raw path discovery unchanged so values such as
month=01are not cast and rewritten.Tests
mvn -pl paimon-spark/paimon-spark-common -DskipITs -Dtest=FormatTablePartitionRepairTest -DfailIfNoTests=false testmvn -pl paimon-spark/paimon-spark-common spotless:checkThe new regression cases cover ADD, SYNC, a custom default partition name, a default partition on a non-leaf value-only level, and the unaffected key-value layout.