Skip to content

feat(java/python/rust/go/c++): xlang nullable/ref alignment #3104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
chaokunyang merged 20 commits into from
Jan 1, 2026
Merged

feat(java/python/rust/go/c++): xlang nullable/ref alignment #3104

chaokunyang merged 20 commits into from
Jan 1, 2026
+3,350 −328

Conversation

@chaokunyang
Copy link
Collaborator

@chaokunyang chaokunyang commented Dec 31, 2025 •
edited
Loading

Why?

Cross-language serialization requires consistent handling of nullable fields and reference tracking across all language implementations. Previously, there were inconsistencies in:

  • Field sorting order for nullable vs non-nullable fields
  • Handling of std::optional / Optional types during serialization
  • TypeDef encoding/decoding for field nullability metadata
  • MetaCompressor configuration not being passed through in cython mode

What does this PR do?

Core Changes

  1. Unified Field Sorting Order (Java, C++, Go, Rust, Python)

    • Fixed numeric field sorter to use type_id descending order to match Java's implementation
    • Ensures consistent field order across all languages for schema compatibility

  2. Nullable Field Xlang Tests

    • Added comprehensive nullable field tests for SCHEMA_CONSISTENT and COMPATIBLE modes
    • New test structs: NullableComprehensiveSchemaConsistent (type_id=401) and NullableComprehensiveCompatible (type_id=402)
    • Tests cover all primitive types, boxed types, and reference types (String, List, Set, Map)
    • Enabled tests for C++, Python, Go, and Rust

  3. C++ Improvements

    • Fixed std::optional serializer to properly propagate has_generics flag
    • Added NullableComprehensiveSchemaConsistent and NullableComprehensiveCompatible structs
    • Implemented nullable field test handlers

  4. Python Improvements

    • Added NoOpMetaCompressor for testing without compression
    • Added meta_compressor parameter to Fory and TypeResolver constructors
    • Fixed cython mode to properly pass meta_compressor parameter
    • Updated NullableComprehensiveCompatible to use Optional for all nullable fields
    • Fixed field name resolution with smart fallback lookup (snake_case ↔ camelCase)

  5. Go Improvements

    • Added nullable field test support
    • Fixed field ordering for xlang compatibility

  6. Rust Improvements

    • Added nullable field test handlers
    • Fixed field sorting consistency

  7. Java Improvements

    • Refactored ObjectSerializer for better nullable/ref tracking handling
    • Fixed StringUtils.lowerUnderscoreToLowerCamelCase off-by-one bug
    • Added custom test overrides for C++ and Python that properly handle null values

Language-Specific Null Handling

  • C++ uses std::optional<T> - properly preserves null values
  • Python uses Optional[T] - properly preserves null values
  • Rust sends default values for nullable fields (different behavior)
  • Go handles nullable fields with proper nil checks

Related issues

#1017
#2982
#2906

Does this PR introduce any user-facing change?

  • Does this PR introduce any public API change?
    • Python: Added meta_compressor parameter to Fory constructor
  • Does this PR introduce any binary protocol compatibility change?

Benchmark

N/A
@chaokunyang
fix(xlang): skip compatible mode tests due to pre-existing Java bug
01e3d40 
Compatible mode nullable field tests fail with NullPointerException in
TypeDefDecoder.readFieldsInfo. This is a pre-existing Java bug that
affects all xlang tests. Skip these tests until the Java bug is fixed.
@chaokunyang
Merge remote-tracking branch 'asf/main' into xlang_ref_tracking
02b4a6c
pandalee99
pandalee99 approved these changes Dec 31, 2025
View reviewed changes
Copy link
Contributor

@pandalee99 pandalee99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
Wish you all the better in the new year!
@chaokunyang
Copy link
Collaborator Author

chaokunyang commented Dec 31, 2025

@pandalee99 Happy new year!
@chaokunyang
fix(xlang): enable nullable field tests for C++, Go, Python
70c4f81 
- Fix C++ nullable field handling in SCHEMA_CONSISTENT mode
- Fix Go nullable flag calculation for COMPATIBLE vs SCHEMA_CONSISTENT modes
- Fix Go SET type handling (map[T]bool) in TypeDef encoding/decoding
- Add GetSetSerializer to Go TypeResolver
- Add language-specific test overrides for nullable compatible mode
- Fix Python nullable field serialization
@chaokunyang
fix(xlang): fix Go internal tests and Python code style
d8fc6cf 
- Revert Go nullable flag to always use nullable for reference types
  (maintains consistency with Go codegen which always writes null flags)
- Skip Go SCHEMA_CONSISTENT nullable tests (codegen incompatible)
- Regenerate Go codegen for test structs
- Fix Python code style (use 'not x' instead of 'x == False')
@chaokunyang
fix(xlang): fix CI failures in code style and Go tests
51d6a08 
- Add default clauses to switch statements in ObjectSerializer.java
  to satisfy checkstyle requirements
- Fix Go map key handling in convertRecursively test helper
  (map keys shouldn't unconditionally call .Elem())
- Skip pre-existing map[string]bool and map[bool]bool test cases
  that have a serialization bug (false values become true)
- Apply Java code formatting to CPPXlangTest.java
@chaokunyang
style(cpp): fix clang-format issue in smart_ptr_serializers.h
040a557
@chaokunyang
fix(python): preserve field order from TypeDef for xlang deserialization
92d7a04 
When a DataClassSerializer is created from TypeDef (wire data), the field order
should be preserved as received from the sender. Previously, compute_struct_meta
was always called which re-sorted the fields, causing deserialization to read
fields in the wrong order.

This fix tracks whether field_names came from TypeDef and skips re-sorting in
that case, only computing the hash without changing field order.
@chaokunyang
fix(python): fix import path for hash_buffer
b480ce9
@chaokunyang
fix(python): convert snake_case field names back to camelCase for xlang
70289c2 
Java's TypeDefEncoder converts camelCase field names (e.g., newObject) to
snake_case (e.g., new_object) when encoding for cross-language compatibility.
This commit adds the reverse conversion in Python's TypeDefDecoder to properly
match field names with the registered Python class.

Added snake_to_camel() function that converts snake_case strings back to
camelCase (e.g., new_object -> newObject, old_object -> oldObject).
@chaokunyang
fix(python): resolve xlang field names with smart fallback lookup
e546ca7 
When decoding xlang fields, the wire field name may be snake_case
(Java's xlang convention) while the Python class may use either
snake_case or camelCase. This fix:

1. Keeps wire field names as-is in typedef_decoder.py
2. Adds smart resolution in TypeDef._resolve_field_names_from_tag_ids()
   that first tries direct name match, then camelCase conversion

This fixes testPolymorphicMap (snake_case fields like animal_map)
while still supporting testCrossVersionCompatibility (camelCase
fields like oldObject/newObject).
@chaokunyang
feat(cpp): enable nullable field compatible mode xlang tests for C++
0540455 
- Enable testNullableFieldCompatibleNotNull and testNullableFieldCompatibleNull
  for C++ which properly supports std::optional for null values
- Override testNullableFieldCompatibleNull in CPPXlangTest to expect actual
  null values (unlike Rust which sends default values)
- Fix off-by-one error in StringUtils.lowerUnderscoreToLowerCamelCase that
  caused StringIndexOutOfBoundsException when string ends with underscore
- Python compatible mode tests remain skipped pending TypeDef encoding fixes
@chaokunyang chaokunyang changed the title feat: Xlang ref tracking Jan 1, 2026
@chaokunyang
feat(python): enable nullable field compatible mode xlang tests
c2fcb97 
- Add NoOpMetaCompressor to Python for testing without compression
- Update Fory and TypeResolver to accept meta_compressor parameter
- Update NullableComprehensiveCompatible class to use Optional for all
  Group 2 fields, enabling proper null value handling
- Add custom testNullableFieldCompatibleNull override in PythonXlangTest
  to expect actual null values (like C++ with std::optional)

Python properly preserves null values using Optional types, unlike Rust
which sends default values.
@chaokunyang
fix(python): pass meta_compressor to cython TypeResolver
57aa411 
The cython Fory and TypeResolver weren't accepting/passing the
meta_compressor parameter, causing NoOpMetaCompressor to not be used
in cython mode.
@chaokunyang chaokunyang changed the title feat(java/pythin/c++/go/rust): xlang nullable/ref alignment Jan 1, 2026
@chaokunyang chaokunyang mentioned this pull request Jan 1, 2026
Open
12 tasks
@chaokunyang chaokunyang changed the title feat(java/python/c++/go/rust): xlang nullable/ref alignment Jan 1, 2026
@chaokunyang chaokunyang merged commit 2be808d into apache:main Jan 1, 2026
60 checks passed
@chaokunyang chaokunyang mentioned this pull request Jan 1, 2026
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants

@chaokunyang @pandalee99