Skip to content

[common][spark] Fix z-order boolean FALSE colliding with the null sentinel - #9527

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/zorder-boolean-null-collision
Open

[common][spark] Fix z-order boolean FALSE colliding with the null sentinel#9527
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/zorder-boolean-null-collision

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #9526

Z-order encodes each order column into eight bytes and then interleaves the bits, and ZOrderByteUtils.NULL_BYTES is eight zero bytes standing for null. The boolean encoder writes only the first byte of its per-column buffer, and the remaining seven are never written, so they stay zero for the life of the indexer. FALSE therefore encoded to eight zero bytes, byte for byte identical to NULL_BYTES: a FALSE row and a NULL row got the same z-order key and were clustered as if the column held the same value, with nothing reporting a problem. FALSE now writes 0x01, which keeps the unsigned order NULL, FALSE, TRUE and leaves TRUE at 0x81.

SparkZOrderUDF.booleanToOrderedBytesUDF carries its own copy of the encoding, so both are changed together. They have to agree: a column clustered by Spark and later compacted by Flink must land in the same order.

Tests

  • TestZOrderByteUtil.testBooleanDistinctFromNullSentinel builds a two-boolean-column ZIndexer and asserts the three states are pairwise distinct and that the unsigned order really is NULL, then FALSE, then TRUE, using the UnsignedBytes comparator the file already uses. Interleaving two identical inputs is monotone in the input, so comparing the interleaved output compares the encodings. Pinning the order matters as much as the distinctness: 0x82 for FALSE would also be distinct while sorting FALSE above TRUE.
  • SparkZOrderUDFTest.testBooleanColumnKeepsFalseOffTheNullSentinel (new file) runs a local SparkSession over a nullable BOOLEAN column with true, false and null rows through sortedLexicographically(col, BooleanType) and asserts the three encodings are 8100000000000000, 0100000000000000 and 0000000000000000. It compares hex rather than the raw arrays on purpose: the UDF returns a per-column buffer it reuses for every row, so three collected byte[] values all alias one another and carry the last row's contents. The conversion happens inside the same projection, before the buffer is overwritten. Production is unaffected, because ZorderSorter feeds the result to interleaveBits within the same row's evaluation, which copies the bits into its own output buffer.

Both fail against the pre-fix code: the indexer test on the FALSE-versus-NULL comparison, and the Spark test with FALSE coming back as 0000000000000000.

Note on running them: TestZOrderByteUtil is named with a Test prefix, and the root pom's test.unit.pattern is **/*Test.*, so that class runs in the integration-test execution rather than in mvn test. CI runs mvn clean install and reaches it; running it directly needs -Dtest=TestZOrderByteUtil, which gives 14 tests, 0 failures.

mvn -pl paimon-common test on JDK 8: 12465 tests, 0 failures, 0 errors. mvn -pl paimon-spark/paimon-spark-common -Dtest='SparkZOrderUDFTest,SortedIndexTopoBuilderTest' test: 5 tests, 0 failures. checkstyle, spotless, enforcer and rat run clean on both modules.

…tinel

The boolean encoder writes only the first byte of its eight-byte buffer, and the
rest stay zero for the life of the indexer, so FALSE encoded to eight zero bytes,
byte for byte identical to ZOrderByteUtils.NULL_BYTES. A FALSE row and a NULL row
got the same z-order key and were clustered as if the column held the same value.
FALSE now writes 0x01, which keeps the unsigned order NULL < FALSE < TRUE and
leaves TRUE at 0x81.

Both engines carry their own copy of the encoding, so ZIndexer and
SparkZOrderUDF are changed together.

Assisted-by: GLM-5.3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant