enhance: cherry pick patch of new DDL framework and CDC 2 #45241

chyezh · 2025-11-03T08:53:27Z

issue: #43897, #44123
pr: #45224
also pick pr: #45216,#45154,#45033,#45145,#45092,#45058,#45029

enhance: Close channel replicator more gracefully (#45029)

issue: #44123

enhance: Show create time for import job (#45058)

issue: #45056

fix: wal state may be unconsistent after recovering from crash (#45092)

issue: #45088, #45086

Message on control channel should trigger the checkpoint update.
LastConfrimedMessageID should be recovered from the minimum of
checkpoint or the LastConfirmedMessageID of uncommitted txn.
Add more log info for wal debugging.

fix: make ack of broadcaster cannot canceled by client (#45145)

issue: #45141

make ack of broadcaster cannot canceled by rpc.
make clone for assignment snapshot of wal balancer.
add server id for GetReplicateCheckpoint to avoid failure.

enhance: support collection and index with WAL-based DDL framework (#45033)

issue: #43897

Part of collection/index related DDL is implemented by WAL-based DDL
framework now.
Support following message type in wal, CreateCollection,
DropCollection, CreatePartition, DropPartition, CreateIndex, AlterIndex,
DropIndex.
Part of collection/index related DDL can be synced by new CDC now.
Refactor some UT for collection/index DDL.
Add Tombstone scheduler to manage the tombstone GC for collection or
partition meta.
Move the vchannel allocation into streaming pchannel manager.

enhance: support load/release collection/partition with WAL-based DDL framework (#45154)

issue: #43897

Load/Release collection/partition is implemented by WAL-based DDL
framework now.
Support AlterLoadConfig/DropLoadConfig in wal now.
Load/Release operation can be synced by new CDC now.
Refactor some UT for load/release DDL.

enhance: Don't start cdc by default (#45216)

issue: #44123

fix: unrecoverable when replicate from old (#45224)

issue: #44962

issue: milvus-io#44123 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Signed-off-by: chyezh <chyezh@outlook.com>

issue: milvus-io#45056 Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Signed-off-by: chyezh <chyezh@outlook.com>

…s-io#45092) issue: milvus-io#45088, milvus-io#45086 - Message on control channel should trigger the checkpoint update. - LastConfrimedMessageID should be recovered from the minimum of checkpoint or the LastConfirmedMessageID of uncommitted txn. - Add more log info for wal debugging. --------- Signed-off-by: chyezh <chyezh@outlook.com>

issue: milvus-io#45141 - make ack of broadcaster cannot canceled by rpc. - make clone for assignment snapshot of wal balancer. - add server id for GetReplicateCheckpoint to avoid failure. Signed-off-by: chyezh <chyezh@outlook.com>

…ilvus-io#45033) issue: milvus-io#43897 - Part of collection/index related DDL is implemented by WAL-based DDL framework now. - Support following message type in wal, CreateCollection, DropCollection, CreatePartition, DropPartition, CreateIndex, AlterIndex, DropIndex. - Part of collection/index related DDL can be synced by new CDC now. - Refactor some UT for collection/index DDL. - Add Tombstone scheduler to manage the tombstone GC for collection or partition meta. - Move the vchannel allocation into streaming pchannel manager. --------- Signed-off-by: chyezh <chyezh@outlook.com>

… framework (milvus-io#45154) issue: milvus-io#43897 - Load/Release collection/partition is implemented by WAL-based DDL framework now. - Support AlterLoadConfig/DropLoadConfig in wal now. - Load/Release operation can be synced by new CDC now. - Refactor some UT for load/release DDL. --------- Signed-off-by: chyezh <chyezh@outlook.com>

issue: milvus-io#44123 Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Signed-off-by: chyezh <chyezh@outlook.com>

issue: milvus-io#44962 Signed-off-by: chyezh <chyezh@outlook.com>

sre-ci-robot · 2025-11-03T08:54:26Z

[INFO] PR Label Summary by Default
[SUCCESS] PR #45224 merged to master

Title: fix: unrecoverable when replicate from old
Link: fix: unrecoverable when replicate from old #45224

[WARNING] Milestone not set

PR: enhance: cherry pick patch of new DDL framework and CDC 2 #45241
Title: enhance: cherry pick patch of new DDL framework and CDC 2
Please set a milestone for better release tracking

You can set milestone by commenting:
/set-milestone
Example:
/set-milestone 2.5.0

Use /refresh-label to update related check and label manually

chyezh · 2025-11-03T09:05:06Z

/refresh-label

sre-ci-robot · 2025-11-03T09:05:38Z

[INFO] PR Label Summary by Refresh-Label

Title: enhance: cherry pick patch of new DDL framework and CDC 2
Target: 2.6
Labels: kind/enhancement, area/dependency, sig/testing, size/XXL, area/test, dco-passed, do-not-merge/need-milestone

[SUCCESS] PR #45224 merged to master

Title: fix: unrecoverable when replicate from old
Link: fix: unrecoverable when replicate from old #45224

Use /refresh-label to update related check and label manually

bigsheeper · 2025-11-03T09:07:20Z

/lgtm

sre-ci-robot · 2025-11-03T09:07:54Z

[INFO] PR Label Summary by Default
[SUCCESS] PR #45224 merged to master

Title: fix: unrecoverable when replicate from old
Link: fix: unrecoverable when replicate from old #45224

Use /refresh-label to update related check and label manually

mergify · 2025-11-03T10:04:08Z

@chyezh cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2025-11-03T10:17:57Z

/run-cpu-e2e

sre-ci-robot · 2025-11-03T10:23:39Z

[ci-v2-notice]
Notice: We are gradually rolling out the new ci-v2 system.

Legacy CI jobs remain unaffected, you can just ignore ci-v2 if you don't want to run it.
Additional "ci-v2/*" checkers will run for this PR to ensure the new ci-v2 system is working as expected.
For tests that exist in both v1 and v2, passing in either system is considered PASS.

To rerun ci-v2 checks, comment with:

/ci-rerun-code-check // for ci-v2/code-check
/ci-rerun-build // for ci-v2/build
/ci-rerun-ut-integration // for ci-v2/ut-integration
/ci-rerun-ut-go // for ci-v2/ut-go
/ci-rerun-ut-cpp // for ci-v2/ut-cpp
/ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp
/ci-rerun-e2e-arm // for ci-v2/e2e-arm

If you have any questions or requests, please contact @zhikunyao.

codecov · 2025-11-03T12:11:10Z

Codecov Report

❌ Patch coverage is 81.06945% with 308 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.80%. Comparing base (cb08db0) to head (b5ba2d4).
⚠️ Report is 179 commits behind head on 2.6.

Files with missing lines	Patch %	Lines
internal/datacoord/index_service.go	73.73%	19 Missing and 7 partials ⚠️
internal/querycoordv2/job/load_config.go	83.66%	20 Missing and 5 partials ⚠️
...nternal/rootcoord/ddl_callbacks_drop_collection.go	77.31%	16 Missing and 6 partials ⚠️
...ernal/rootcoord/ddl_callbacks_create_collection.go	85.81%	14 Missing and 7 partials ⚠️
internal/rootcoord/ddl_callbacks_drop_partition.go	76.47%	15 Missing and 5 partials ⚠️
...ternal/rootcoord/ddl_callbacks_create_partition.go	73.61%	13 Missing and 6 partials ⚠️
internal/querycoordv2/services.go	77.92%	17 Missing ⚠️
internal/rootcoord/meta_table.go	59.52%	12 Missing and 5 partials ⚠️
...2/ddl_callbacks_alter_load_info_load_collection.go	81.70%	9 Missing and 6 partials ⚠️
internal/querycoordv2/meta/replica_manager.go	68.88%	10 Missing and 4 partials ⚠️
... and 25 more

❌ Your project check has failed because the head coverage (76.80%) is below the target coverage (77.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##              2.6   #45241      +/-   ##
==========================================
- Coverage   76.99%   76.80%   -0.20%     
==========================================
  Files        1700     1855     +155     
  Lines      262533   286258   +23725     
==========================================
+ Hits       202142   219862   +17720     
- Misses      53550    59121    +5571     
- Partials     6841     7275     +434

Components	Coverage Δ
Client	`78.03% <65.76%> (-0.11%)`	⬇️
Core	`84.09% <77.61%> (+1.87%)`	⬆️
Go	`74.74% <52.50%> (-0.65%)`	⬇️

Files with missing lines	Coverage Δ
...replication/replicatemanager/channel_replicator.go	`76.85% <100.00%> (+12.32%)`	⬆️
internal/datacoord/ddl_callbacks_alter_index.go	`100.00% <100.00%> (ø)`
internal/datacoord/ddl_callbacks_drop_index.go	`100.00% <100.00%> (ø)`
internal/datacoord/index_meta.go	`91.86% <100.00%> (-0.12%)`	⬇️
internal/datacoord/server.go	`67.76% <100.00%> (-0.88%)`	⬇️
internal/datacoord/services.go	`85.46% <100.00%> (+4.10%)`	⬆️
internal/metastore/catalog.go	`100.00% <ø> (ø)`
internal/metastore/kv/rootcoord/kv_catalog.go	`76.35% <100.00%> (+1.19%)`	⬆️
internal/metastore/model/partition.go	`100.00% <ø> (ø)`
internal/querycoordv2/job/job_release.go	`80.43% <100.00%> (-1.08%)`	⬇️
... and 57 more

... and 480 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

czs007

/lgtm

sre-ci-robot · 2025-11-03T13:18:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chyezh, czs007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot · 2025-11-03T13:19:01Z

[INFO] PR Label Summary by Default
[SUCCESS] PR #45224 merged to master

Title: fix: unrecoverable when replicate from old
Link: fix: unrecoverable when replicate from old #45224

Use /refresh-label to update related check and label manually

chyezh · 2025-11-03T14:51:55Z

/run-cpu-e2e

bigsheeper and others added 8 commits November 3, 2025 16:45

enhance: Close channel replicator more gracefully (milvus-io#45029)

6af3891

issue: milvus-io#44123 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Signed-off-by: chyezh <chyezh@outlook.com>

enhance: Show create time for import job (milvus-io#45058)

94e8dd2

issue: milvus-io#45056 Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Signed-off-by: chyezh <chyezh@outlook.com>

enhance: Don't start cdc by default (milvus-io#45216)

5c5a24d

issue: milvus-io#44123 Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Signed-off-by: chyezh <chyezh@outlook.com>

fix: unrecoverable when replicate from old (milvus-io#45224)

b5ba2d4

issue: milvus-io#44962 Signed-off-by: chyezh <chyezh@outlook.com>

sre-ci-robot requested review from bigsheeper and congqixia November 3, 2025 08:53

sre-ci-robot added area/dependency Pull requests that update a dependency file area/test sig/testing size/XXL Denotes a PR that changes 1000+ lines. labels Nov 3, 2025

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Nov 3, 2025

sre-ci-robot added the do-not-merge/need-milestone generate by v2-label-manager label Nov 3, 2025

chyezh added this to the 2.6.5 milestone Nov 3, 2025

sre-ci-robot removed the do-not-merge/need-milestone generate by v2-label-manager label Nov 3, 2025

sre-ci-robot assigned bigsheeper Nov 3, 2025

sre-ci-robot added the lgtm label Nov 3, 2025

czs007 approved these changes Nov 3, 2025

View reviewed changes

sre-ci-robot assigned czs007 Nov 3, 2025

sre-ci-robot added the approved label Nov 3, 2025

mergify bot added the ci-passed label Nov 3, 2025

sre-ci-robot merged commit 02e2170 into milvus-io:2.6 Nov 3, 2025
14 of 19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

enhance: cherry pick patch of new DDL framework and CDC 2 #45241

enhance: cherry pick patch of new DDL framework and CDC 2 #45241

chyezh commented Nov 3, 2025

sre-ci-robot commented Nov 3, 2025

chyezh commented Nov 3, 2025

sre-ci-robot commented Nov 3, 2025

bigsheeper commented Nov 3, 2025

sre-ci-robot commented Nov 3, 2025

mergify bot commented Nov 3, 2025

chyezh commented Nov 3, 2025

sre-ci-robot commented Nov 3, 2025

codecov bot commented Nov 3, 2025 •

edited

Loading

czs007 left a comment

sre-ci-robot commented Nov 3, 2025

sre-ci-robot commented Nov 3, 2025

chyezh commented Nov 3, 2025 •

edited

Loading

Uh oh!

Labels

4 participants

enhance: cherry pick patch of new DDL framework and CDC 2 #45241

enhance: cherry pick patch of new DDL framework and CDC 2 #45241

Conversation

chyezh commented Nov 3, 2025

sre-ci-robot commented Nov 3, 2025

chyezh commented Nov 3, 2025

sre-ci-robot commented Nov 3, 2025

bigsheeper commented Nov 3, 2025

sre-ci-robot commented Nov 3, 2025

mergify bot commented Nov 3, 2025

chyezh commented Nov 3, 2025

sre-ci-robot commented Nov 3, 2025

codecov bot commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

czs007 left a comment

Choose a reason for hiding this comment

sre-ci-robot commented Nov 3, 2025

sre-ci-robot commented Nov 3, 2025

chyezh commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Labels

4 participants

codecov bot commented Nov 3, 2025 •

edited

Loading

chyezh commented Nov 3, 2025 •

edited

Loading