feat(index): add IVF RaBitQ vector indexing - #19802
Draft
chrevanthreddy wants to merge 72 commits into
Draft
Conversation
Add the Metadata Table storage foundation for native vector search: - HoodieMetadata.avsc: vector index record types (centroids, quantizer, manifest, cluster stats, posting block, posting delta, tombstone) - HoodieMetadataPayload: VectorIndexMetadata field, entry-type constants, serde - MetadataPartitionType.VECTOR_INDEX partition type + record-type wiring - HoodieTableMetadataUtil: vector posting key -> file-group mapping - posting block codec (PostingBlockBuilder/View, structure-of-arrays) - vector index options/type/metric enums - MDT raw keys (VectorIndexMetadataKey, posting prefix, cluster, manifest) Tests: TestVectorDistanceMetric, TestVectorIndexMetadataKey, TestVectorIndexMetadataPayload (17 cases). Existing HoodieMetadataRecord all-args callers updated for the appended field. Compiles + tests green standalone on current master (hudi-common, JDK17). Part of apache#19094; resolves apache#19097.
Add the RaBitQ quantization core for the RFC-104 vector index (apache#19094): - RaBitQEncoder: residual + random-rotation encoding to multibit codes, with the unbiased distance estimator and per-vector error bound used for safe pruning (nested RaBitQQueryState for query-side scoring) - VectorQuantizer: quantizer contract over the encoder - MetricQueryState: per-query rotation/normalization state - RaBitQFactorConfig / RaBitQNeutralFactors: factor packing and neutral defaults Tests (47 cases): encoder roundtrip, estimator identity, neutral factors, query-state rotation, residual hypothesis, and residual recall. Builds on the MDT schema foundation (apache#19097). Compiles + tests green on hudi-common (JDK17). Resolves apache#19098.
…d exact rerank Add the engine-agnostic vector search core for the RFC-104 vector index (apache#19094), all in hudi-common with no Spark/engine dependencies: - search/: candidate generation and top-K accumulation over MDT postings, fetch planning, execution-mode selection (approx vs exact) with deadline and budget control, record-index candidate arbitration, and continuation - DefaultExactVectorScorer / DefaultVectorExactReranker: exact rerank from authoritative base-table vectors - VectorIndexPruner: RaBitQ error-bounded two-pass pruning - VectorIndexArbiter: approximate vs exact decisioning Pure-CPU algorithm library over the MDT schema (apache#19097) and RaBitQ encoder (apache#19098); IO is injected via interfaces (candidate sources, fetch tasks). Tests: 34 cases across executor, scorer, reranker, fetch planner, arbiter, pruner, execution-mode selector, and continuation. Compiles + tests green on hudi-common (JDK17). Part of apache#19101, apache#19102, apache#19103. Note: VectorIndexMetadataCache (generation-visibility, apache#19100) is deferred to the reader-consistency PR where its shard-count fallback semantics belong.
…pache#19098) Implements MDT vector-index initialization the idiomatic way against master's HUDI-9358 Indexer SPI (instead of the removed switch-case writer): - VectorIndexer (hudi-client-common): partition sizing (IVF clusters -> file groups) + IndexInitializationPlan assembly; update/clean deferred to the lifecycle PR. - EngineIndexerSupport.generateVectorIndexRecords + SparkIndexerSupport impl (reads source VECTOR column via HoodieFileGroupReader, runs IVF+RaBitQ bootstrap); UnsupportedEngineIndexerSupport no-op. - IndexerFactory: register VECTOR_INDEX. - SparkVectorIndexBootstrap + TwoLevelKMeansBootstrap (Spark ML KMeans), spark-mllib (provided) dep, VectorIndexBootstrapUtils + tests. - HoodieMetadataConfig: vector index file-group-count option. - HoodieIndexVersion: VECTOR_INDEX current version. TestSparkVectorIndexBootstrap: end-to-end functional test on Spark 3.5 green (centroids/quantizer/cluster-stats/posting + generation-1 manifest records produced through the SPI).
…egration) Index-path wiring so DDL reaches the new VectorIndexer on current master: - HoodieIndexUtils.getVectorIndexDefinition + validateEligibilityForVectorIndex - HoodieSparkIndexClient.createVectorIndex + create() dispatch branch (lean, no RaBitQ materialization: mdt_lookup storage path) - ScheduleIndexActionExecutor: extracted resolvePartitionName() (DRY) with VECTOR_INDEX dynamic partition-name resolution Writer/IndexerFactory/RunIndexActionExecutor already generic via HUDI-9358 SPI (VectorIndexer plugs in via MetadataPartitionType.isMetadataPartitionAvailable). Query path (hudi_vector_search TVF) already present from rfc-104-search. NOTE: integration branch only (rfc-104-e2e-integration) for live 10M validation; NOT part of the clean PR-4 slice.
… values CREATE INDEX ... USING VECTOR routes through HoodieSparkIndexClient -> getWriteClient -> HoodieIndexingConfig.fromIndexDefinition, which sets INDEX_TYPE to the definition type (vector_index). Current master enforces ConfigProperty.checkValues (case-insensitive), which rejected vector_index. Add VECTOR_INDEX.name() to the valid values. Found via live 10M run on useast4-dev-gke-st-spark00: IllegalArgumentException: The value of hoodie.expression.index.type should be one of COLUMN_STATS,RECORD_INDEX,BLOOM_FILTERS,SECONDARY_INDEX, but was vector_index.
CreateIndexCommand.run had no branch for indexType=VECTOR, so USING VECTOR fell through to 'throw HoodieIndexException(%s is not supported)'. Add a vector branch that routes to HoodieSparkIndexClient.create with PARTITION_NAME_VECTOR_INDEX. Also teach ShowIndexesCommand to recognize the vector partition prefix. Found via live 10M run: HoodieIndexException: VECTOR is not supported (after the config valid-values fix let it past getWriteClient).
Ports the approximate vector search read path from the feature branch onto current master: - hudi-common: VectorIndexMdtSearchUtils (IVF probe + MDT posting read + RaBitQ scoring), VectorIndexMetadataCache, RaBitQByteLutScorer, RaBitQPlaneKernel, VectorQueryPlanes. - hudi-spark-common: HoodieVectorSearchTableValuedFunction (4-7 arg TVF with kv runtime options + ivf_rabitq_mdt algorithm), HoodieVectorSearchPlanBuilder (IvfRaBitQMdtSearchAlgorithm + VectorSearchTable), PositionalParquetFetcher (exact re-rank fetch), VectorDistanceUtils. Surgically restored the vector-search analysis glue in HoodieSparkBaseAnalysis (VectorSearchTable + runtimeOptions) while preserving the branch's MergeIntoTable fix. - hudi-io: added physicalReadSnapshot()/globalCacheStatsString() + thread-local physical-read counters to CachingHFileReaderImpl (incremented only in the cache-miss supplier). Replaces the rfc-104-search brute-force-only TVF. Compiles under spark4.0/scala-2.13.
…_INDEX Ports the missing read-side file-slice targeting from the feature branch into HoodieBackedTableMetadata.getRecordsByKeyPrefixes. Without it, vector posting prefix lookups fanned out to ALL metadata file groups (4096 at 10M/4096 clusters) instead of the ~nprobes routed groups, making search pathologically slow and returning 0 matches on current master's prefix path. Adds getTargetFileSlicesWithPrefixesForKeyPrefixLookup + isRouteableVector* helpers: for the VECTOR_INDEX partition (when all prefixes are routeable vector metadata keys), each prefix is mapped to its target file group via HoodieTableMetadataUtil.mapVectorPostingKeyToFileGroupIndex (mirroring the write path's MetadataPartitionType.VECTOR_INDEX file-group mapping) and only those slices are scanned. All other partitions keep the full-scan behavior.
Revert the bounded range prefetch implementation after the controlled BigANN 1B A/B showed warm p95 regression and 2.61x physical read amplification. Restore the proven batched exact-fetch path while retaining the later candidate-retention and query-planning improvements.
…d exact rerank Add the engine-agnostic vector search core for the RFC-104 vector index (apache#19094), all in hudi-common with no Spark/engine dependencies: - search/: candidate generation and top-K accumulation over MDT postings, fetch planning, execution-mode selection (approx vs exact) with deadline and budget control, record-index candidate arbitration, and continuation - DefaultExactVectorScorer / DefaultVectorExactReranker: exact rerank from authoritative base-table vectors - VectorIndexPruner: RaBitQ error-bounded two-pass pruning - VectorIndexArbiter: approximate vs exact decisioning Pure-CPU algorithm library over the MDT schema (apache#19097) and RaBitQ encoder (apache#19098); IO is injected via interfaces (candidate sources, fetch tasks). Tests: 34 cases across executor, scorer, reranker, fetch planner, arbiter, pruner, execution-mode selector, and continuation. Compiles + tests green on hudi-common (JDK17). Part of apache#19101, apache#19102, apache#19103. Note: VectorIndexMetadataCache (generation-visibility, apache#19100) is deferred to the reader-consistency PR where its shard-count fallback semantics belong.
…pache#19098) Implements MDT vector-index initialization the idiomatic way against master's HUDI-9358 Indexer SPI (instead of the removed switch-case writer): - VectorIndexer (hudi-client-common): partition sizing (IVF clusters -> file groups) + IndexInitializationPlan assembly; update/clean deferred to the lifecycle PR. - EngineIndexerSupport.generateVectorIndexRecords + SparkIndexerSupport impl (reads source VECTOR column via HoodieFileGroupReader, runs IVF+RaBitQ bootstrap); UnsupportedEngineIndexerSupport no-op. - IndexerFactory: register VECTOR_INDEX. - SparkVectorIndexBootstrap + TwoLevelKMeansBootstrap (Spark ML KMeans), spark-mllib (provided) dep, VectorIndexBootstrapUtils + tests. - HoodieMetadataConfig: vector index file-group-count option. - HoodieIndexVersion: VECTOR_INDEX current version. TestSparkVectorIndexBootstrap: end-to-end functional test on Spark 3.5 green (centroids/quantizer/cluster-stats/posting + generation-1 manifest records produced through the SPI).
added 26 commits
August 7, 2026 11:51
…vector-engine-neutral-executor) The index.vector.search package (engine-neutral executor + candidate source, RLI arbiter, fetch planner, exact reranker, continuation, deadline policy) is complete and unit-tested but not wired into the Spark TVF, which uses its own path in HoodieVectorSearchPlanBuilder. Wiring requires a production Parquet positional HoodieVectorBatchReadHandle that does not exist yet. Removing it here keeps this PR's reviewable surface equal to its claim (maintain vector index across source writes). The code is preserved verbatim on branch vector-engine-neutral-executor and returns as the opening, pre-certified commit of the engine-neutral search slice.
…single-slice RLI arbitrateFinalistsForPartition used HoodiePairData.leftOuterJoin, which requires both operands to share a backing flavor. readRecordIndexLocationsWithKeys returns list-backed pair data for a single-slice RLI (the common 1-file-group case) and RDD-backed for multi-slice; joining an RDD-backed finalist set against list-backed locations throws ClassCastException (HoodieListPairData cannot be cast to HoodieJavaPairRDD) during the Spark TVF approximate plan. Resolve RLI locations for the bounded finalist key set into a driver-side map and attach via finalists.map(...), preserving the finalists' backing flavor and working for single- and multi-slice RLI alike. Mirrors the exact path (arbitrateMaterializedFinalists), which already used a driver map, not a join.
…erank Exact rerank previously skipped MOR log-resident candidates (a fresh, uncompacted update whose raw embedding lives in a log block, not the base Parquet file), leaving a recall shortfall until compaction. The fixture gated the exact-vs-brute-force assertion to COW because of this. This materializes those candidates by reusing Hudi's existing HoodieFileGroupReader + HoodieAvroReaderContext seam to read the merged (base + log) file slice, scoring only the bounded log-resident key set with the SAME distance math as the positional path: - VectorExactScorer: extracted byte->distance scorer, shared by the positional and log-resident fetchers (single source of truth so exact == brute-force holds identically). - LogResidentVectorFetcher: driver-side merged read (the driver already holds the meta client and resolved file slices; the set is bounded by refineK) producing output-schema InternalRows. - HoodieVectorSearchPlanBuilder: pre-split candidates into base-resident (fast positional RDD path) and log-resident (merged read), then union. - isLogResidentCandidate: position < 0 in a slice with logs is the authoritative signal. The prior instant != baseInstant clause was wrong: a log delta can share the slice's base instant time, which misrouted the moved record to the base key-lookup and returned its stale pre-update row. TestHoodieVectorIndexSearch now asserts exact == brute-force for MERGE_ON_READ too (including the log-resident moved record), not just COPY_ON_WRITE.
4 tasks
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19802 +/- ##
=============================================
- Coverage 67.91% 25.77% -42.14%
+ Complexity 31356 11859 -19497
=============================================
Files 2695 2540 -155
Lines 152729 144831 -7898
Branches 19052 17535 -1517
=============================================
- Hits 103721 37336 -66385
- Misses 40412 104463 +64051
+ Partials 8596 3032 -5564
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Implements the vector-index design proposed in RFC-109 (#19309), providing native approximate nearest-neighbor search over Hudi vector columns while retaining the data table as the authoritative source for exact re-ranking.
The implementation uses the Metadata Table for IVF routing, residual RaBitQ postings, freshness arbitration, and incremental maintenance. It also adds positional/key-based exact fetch across COW and MOR file slices.
Summary and Changelog
No code was copied.
Impact
Adds an opt-in vector-index feature. Existing tables and non-vector readers/writers are unchanged unless a vector index is explicitly created. The patch adds MDT vector partitions and Spark SQL planning/execution paths.
Risk Level
high — this is a substantial new indexing and query path. The PR is intentionally opened as a draft while the final acceptance battery completes.
Completed evidence:
TestVectorIndexOptions: 10 passed, including mandatory key verification.Still running or required before ready-for-review:
Documentation Update
RFC-109 is updated in #19309 with architecture, correctness contracts, limitations, test plan, and BIGANN evidence.
Contributor's checklist