docs(rfc): RFC-109 native vector search support in Apache Hudi

[chrevanthreddy]

Describe the issue this Pull Request addresses

This PR adds the full RFC document for RFC-109: Native Vector Search Support in Apache Hudi, following the RFC number claim merged in #19303. It links the RFC into the registry (rfc/README.md) and adds the design document plus architecture/read/write diagrams.

Umbrella tracking issue: #19094

Summary and Changelog

• Add rfc/rfc-109/rfc-109.md — the RFC-109 design document covering: goals/non-goals, architecture, the IVF + RaBitQ index algorithm, the metadata-table posting-block storage model, bootstrap/write path, read path, and maintenance/rebalancing/cleaner semantics.
• Add rfc/rfc-109/diagrams/ — architecture overview, read path, and write/maintenance SVG diagrams referenced by the doc.
• Update rfc/README.md — link the existing RFC-109 registry row (UNDER REVIEW) to the new document.

This is a docs-only change. No production code, tests, or build files are touched. The implementation lands separately as the focused PR series tracked under #19094 (child issues #19096–#19105).

Impact

None on runtime, build, or existing functionality — documentation only. Establishes the reviewable design of record for the vector index feature so the community can evaluate architecture separately from implementation.

Risk Level

none

Docs-only; no code paths, dependencies, or CI behavior affected.

Documentation Update

This PR is the documentation update: it introduces the RFC-109 design document and diagrams and links them from the RFC index. No user-facing site docs change until the feature is implemented and merged.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact described above
• Necessary doc changes done or have another open PR (this PR is the doc)
• For large changes, please consider breaking it into sub-tasks under an umbrella issue (tracked in RFC-109: MDT-backed vector index umbrella tracking issue #19094)

[hudi-agent]

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the RFC! This proposes native IVF + RaBitQ approximate nearest-neighbor vector search stored in the Metadata Table via a column-wise "posting block" layout, with exact re-ranking always done from the base table. The design is thorough and the storage model is well thought through, but a few areas could use more detail before implementation — multi-writer concurrency semantics, the hard dependency on the Record Level Index, recall behavior under deletes/updates between compactions, backward compatibility of the new MDT partition and the VECTOR(D) type, and a system-level "Alternatives Considered" section (see inline comments). Once those are addressed, a Hudi committer or PMC member can take it from here for a deeper design review.

[hudi-agent]

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the thorough revision — this addresses all of the prior feedback. Every point from the last pass is now covered: RLI is an explicit hard prerequisite (§6.3), __centroids__ is chunked into size-bounded T| records (§4.2/§4.5), the centroid epoch is gone in favor of generation as the sole version axis (§4.5/§7.3), multi-writer rebuild is replay-only with a marker-frontier freshness gate (§5.5/§9.1), shard-count changes force a routing-versioned rewrite (§4.3), an Alternatives Considered section was added (§1.3), engine support for unaware writers is spelled out (§5.4), and the recall/nprobe numbers are now backed by an artifacted benchmark table with the default raised to 32 (§11.1). The freshness/marker model in §9.1 is a nice addition. One forward-looking design question on how the marker frontier interacts with timeline archival is noted inline — worth a sentence before implementation. Please take a look at the inline comment, and this should be ready for a Hudi committer or PMC member to take it from here.

[hudi-agent]

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the RFC! This is a well-organized proposal for an MDT-native IVF + RaBitQ vector index, and it has clearly absorbed a lot of prior-round feedback (generation model, marker frontier, RLI arbitration, multi-writer rebuild). One remaining correctness question stands out around how the exact re-rank obtains a validated row position after RLI re-resolution — flagged inline. Once that's addressed, a Hudi committer or PMC member can take it from here for a deeper design review.

[vinothchandar]

Did a pass comparing this against my comments on #19107.

Level setting on what carried over:

Addressed:

• RFC number/handles/registry link — sorted.
• Posting block replacing per-vector records — this is the right reframe; MDT record count was never the concern, amplification was.
• IVF vs HNSW rationale, generation model, active generation pointer, generation-aware cleaning — all now spelled out.
• Stale entry handling is locked down to delta supersession + RLI arbitration instead of a menu of options. Good.
• Rebuild is replay-only with catch-up + short lock at cutover, so background build under live writes is covered.
• nprobe/refineFactor now explicit config with defaults; benchmark section added.

Still open from my side:

• LIMITATIONS section — We'd still want VECTOR schema evolution (dimension change → rebuild?) written down so we throw the right errors in the first cut.
• Concurrency across OCC/NBCC still needs a table to clearly spell out the scenarios; today it's scattered across §5.5/§9.1 (?)
• "Generation" is used in §2 before it's defined in §4.5 — introduce it in a line up front.
• How do we detect a vector is unchanged? §5.3 assumes "non-vector update". we need to flesh this out
• Re-rank cost varies by base file format — worth a line, and a recommendation.
• Partition pushdown to bound the search is deferred as non-goal; fine for v1, but let's note it as a known gap since the partition dictionary is already in the block.
• §7 still fully specs LIRE split/merge even though MVP is create/drop/rebuild — let's mark clearly what's in v1 vs later. is LIRE in the v1?
• Minor: define "posting" for the average reader, and clarify "shard" here vs how we use it in RLI.

[vinothchandar]

High level comments.. if addressed, we can land this and keep adding more.

I am also happy to push some changes to get this landed. lmk @chrevanthreddy if you are okay with that.

[chrevanthreddy]

RFC review closeout update:

• Replied to and resolved all 23 review threads against the corresponding landed sections (limitations, terminology, generation/shard invariants, bootstrap recovery, engine scope, stale-position recovery, OCC/NBCC matrix, LIRE scope, and Appendix A evidence).
• Draft implementation PR: feat(index): add IVF RaBitQ vector indexing #19802.
• Independent core Spark task-retry/RLI correctness fix: fix(metadata): use committed Spark write statuses after retries #19801.
• Corrected BIGANN 1B evidence is recorded in Appendix A; the retired-key “approximate” artifact remains explicitly excluded.
• Final implementation acceptance is still running: corrected 10M COW/MOR lifecycle + physical 512-file-group proof, followed by any required corrected 1B rebuild and bounded 1B mutation test.

I have kept the RFC’s claims conservative: the current 1B query numbers describe the measured 16,384-physical-group layout, not the intended 512-group layout, and the independent 1B RLI audit remains open because both attempted Hudi-reader validators bottlenecked on ten giant RLI HFile input partitions.

[hudi-bot]

CI report:

• 9a8b9ed Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build