Skip to content

[common] Reject non-struct inner fields in variant shredding schema - #9518

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/variant-shredding-schema-validation
Open

[common] Reject non-struct inner fields in variant shredding schema#9518
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/variant-shredding-schema-validation

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #9517

In a variant shredding schema, every field of an object's typed_value is itself a group of value / typed_value. buildVariantSchema cast each inner field to RowType without checking it, so a physical schema whose inner field is a scalar died on a bare ClassCastException naming two class names, instead of the Invalid variant shredding schema: <rowType> error the method raises for every other malformed shape. That path now checks the type and raises the same error as the rest.

Two checks in the same branch are removed because they cannot fire:

  • if (!(dataType instanceof RowType)) sits inside case ROW: of a switch on field.type().getTypeRoot(), and RowType is the only type in paimon-api with DataTypeRoot.ROW.
  • if (fields.isEmpty() || fields.stream().distinct().count() != fields.size()) tests the outer field list, not the inner struct it appears to guard. Emptiness is already rejected at the top of the method; duplicates cannot exist because RowType's constructor calls validateFields, which throws on duplicate field names. Duplicate typed_value / value / metadata are caught by the typedIdx != -1 style checks, and any other outer name falls into the throwing default: of the name switch.

An empty inner struct stays legal, exactly as before: the removed check never guarded the inner list, so nothing changes there. It yields a zero-length objectSchema, which is what ShreddingUtils and VariantShreddingWriter already consume by looping on .length.

Tests

Both cases live in PaimonShreddingUtilsTest.

  • testBuildVariantSchemaRejectsNonStructInnerField builds a schema that is valid except for a scalar inner field and asserts the message is the invalid-schema one. Verified red before the change, where it fails with ClassCastException: org.apache.paimon.types.IntType cannot be cast to org.apache.paimon.types.RowType. The message is what separates the two outcomes, since ClassCastException is itself a RuntimeException.
  • testBuildVariantSchemaAcceptsEmptyInnerStruct pins that an empty typed_value struct produces a zero-length objectSchema rather than null, which is the shape the two consumers above depend on. This one passes before the change as well; it is there to hold that behavior still.

mvn -pl paimon-common test on JDK 8: 12467 tests, 0 failures, 0 errors. checkstyle, spotless, enforcer and rat run clean.

buildVariantSchema's ROW branch cast every field of an object's typed_value to
RowType without checking it, so a physical schema whose inner field is a scalar
died on a bare ClassCastException instead of the invalid-schema error the method
raises for every other malformed shape. Check the type and raise that error.

The two checks removed alongside it were dead. `!(dataType instanceof RowType)`
sits inside `case ROW:` of a switch on getTypeRoot(), and RowType is the only
type with that root. The other check tested the outer field list: its emptiness
half is already rejected at the top of the method, and its duplicate half cannot
hold because RowType's constructor calls validateFields, which rejects duplicate
field names outright. An empty inner struct stays legal, as before, and produces
a zero-length objectSchema.

Assisted-by: GLM-5.3
@LuciferYang

Copy link
Copy Markdown
Contributor Author

The UTCase and ITCase Others on JDK 11 job was killed at the 6 hour default timeout, and it is not this change.

I pulled the job log: of the 826 test classes it started, exactly one never reported a result, org.apache.paimon.operation.FileStoreCommitTest in paimon-core, which began at 08:14 and was still running when the job was cancelled at 14:01. That class has no relation to variant shredding (it never touches PaimonShreddingUtils or a variant column), and it has a structural way to hang: it starts several TestCommitThreads and joins them with a bare thread.join(), while those threads retry inside while (true) loops with FailingFileIO failures injected, so a retry that cannot make progress spins with nothing to preempt it. The class is also randomized in thread count and data volume, and this job runs with a random timezone.

For the part that is mine, I re-ran every variant and shredding test locally on JDK 11 (Zulu 11.0.30), paimon-common plus paimon-format: 51 tests, 0 failures. mvn -pl paimon-common clean test on JDK 8 is also green at 12467 tests.

A re-run should clear the job.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant