Batch libraries test Helix work items by compatibility metadata

[mmitche]

Summary

Reduces the number of libraries-test Helix work items by batching multiple test assemblies into a single work item, cutting per-work-item overhead (queueing, machine provisioning, payload/runtime download) while keeping total test execution time roughly flat.

Today each libraries test assembly is sent to Helix as its own work item, so the fixed per-work-item overhead is paid thousands of times. This change groups compatible desktop libraries work items together and runs them through a structured batch runner.

How it works

• GroupWorkItems MSBuild task (src/tasks/HelixTestTasks/GroupWorkItems.cs): greedy bin-packs assemblies into batches by file size. Items above a size threshold get their own solo batch. Assemblies are first partitioned by a compatibility-key tuple so only co-runnable assemblies share a work item:
  RunnerType;PlatformFamily;RuntimeFlavor;TargetOS;TargetArchitecture;Scenario;TimeoutClass;PayloadShape
• DesktopBatchRunner.sh / .cmd (eng/testing/): inside a single work item, iterates the batched suites, gives each its own upload root so results don't collide, runs each suite's existing RunTests script, and aggregates per-suite exit codes/durations.
• sendtohelixhelp.proj: composes the desktop batch command from structured runner parts and emits batched work items. Non-batch-compatible (e.g. stress Scenario) items are emitted directly so they keep their original runner and timeout instead of flowing through ComputeBatchTimeout.
• Unit tests added in HelixTestTasks.Tests/GroupWorkItemsTests.cs.

Batching can be disabled via BatchLibraryHelixWorkItems=false.

Validation

A manual pipeline run of this branch showed:

• ~90% fewer libraries Helix work items vs a comparable main run, with total test-execution time roughly flat.
• Work-item efficiency (test-execution time / wall-clock time) improved substantially.
• No new failures observed inside the batched libraries work items.

Opening as draft to get full CI coverage and review on the approach.

Note

This pull request description was drafted with the assistance of GitHub Copilot (AI-generated).

[dotnet-policy-service]

Tagging subscribers to this area: @akoeplinger, @matouskozak, @simonrozsival
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR changes the libraries Helix submission flow to batch multiple compatible test payload ZIPs into fewer Helix work items, and adds a desktop batch runner that executes each suite inside a single work item while keeping per-suite results separated.

Changes:

• Extend the GroupWorkItems MSBuild task to partition candidates by a compatibility-metadata tuple (and optionally force “solo” batches).
• Update sendtohelixhelp.proj to emit either classic per-assembly work items or new batched work items (with per-batch payload creation and timeouts).
• Add DesktopBatchRunner scripts and a new HelixTestTasks.Tests project with unit tests for the grouping logic.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/tasks/HelixTestTasks/GroupWorkItems.cs	Adds compatibility partitioning + “solo” batching logic and assigns globally-unique batch IDs across partitions.
src/tasks/HelixTestTasks.Tests/HelixTestTasks.Tests.csproj	Introduces a new xUnit test project for HelixTestTasks.
src/tasks/HelixTestTasks.Tests/GroupWorkItemsTests.cs	Adds unit tests validating partitioning and solo-batch behaviors.
src/libraries/sendtohelixhelp.proj	Adds batching opt-in default + emits batched work items, payload staging, and batch-runner command composition.
src/libraries/pretest.proj	Ensures HelixTestTasks is built for desktop library Helix batching scenarios (not just browser).
eng/testing/DesktopBatchRunner.sh	Adds Unix batch runner that unpacks suite ZIPs, sets per-suite upload roots, runs each suite, and summarizes outcomes.
eng/testing/DesktopBatchRunner.cmd	Adds Windows batch runner with similar behavior to the Unix runner.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

[github-actions]

Workflow state for the Holistic Review Orchestrator.

{
  "version": 5,
  "last_dispatched_commit": "3a38379a4d2810c8132ed00c4a88f2cbf2d55312",
  "last_dispatched_base_ref": "main",
  "last_dispatched_base_sha": "81df0579e5cd5560cbc3bfc2505a790e0e8d6511",
  "last_reviewed_commit": "3a38379a4d2810c8132ed00c4a88f2cbf2d55312",
  "last_reviewed_base_ref": "main",
  "last_reviewed_base_sha": "81df0579e5cd5560cbc3bfc2505a790e0e8d6511",
  "last_recorded_worker_run_id": "29677691788",
  "review_attempt_commit": "",
  "review_attempt_base_ref": "",
  "review_attempt_count": 0,
  "max_review_attempts": 5,
  "review_history_format": "holistic-review-disclosure-v1",
  "review_history": [
    {
      "commit": "3a38379a4d2810c8132ed00c4a88f2cbf2d55312",
      "review_id": 4730523095
    }
  ]
}

[github-actions]

Holistic Review

Motivation: Reducing the thousands of per-assembly libraries-test Helix work items to cut fixed per-work-item overhead (queueing, provisioning, payload download) is a well-justified, real infrastructure cost problem, and the ~90% work-item reduction reported from a manual pipeline run is compelling evidence.

Approach: Partitioning assemblies by a compatibility-key tuple before greedy bin-packing, then running batched suites through a structured batch runner with isolated upload roots, is a reasonable and appropriately conservative design that keeps non-batch-compatible (e.g. stress) items on their original path. The opt-out flag (BatchLibraryHelixWorkItems=false) and unit tests are good hygiene.

Summary: ⚠️ Needs Human Review. The code is internally consistent and the design is sound, but a naming/semantics mismatch around MaxTestsPerWorkItem (it actually bounds the number of batches per partition, not tests-per-work-item) could surprise operators tuning it and interacts with timeout computation. This is test-infrastructure only (no product/API change) and gated behind a flag, so risk is contained; a human familiar with the Helix libraries pipeline should confirm the intended batching contract and that per-suite result upload/reporting still surfaces correctly under batching.

---

Detailed Findings

⚠️ Correctness / Naming — MaxTestsPerWorkItem bounds batch count, not tests per work item

See the inline comment on src/libraries/sendtohelixhelp.proj line 372. BatchSize is consumed as the number of bins per compatibility partition (numBatches = Math.Min(BatchSize, smallItems.Count)), so the property name overstates the guarantee — large partitions produce work items with more than 20 suites. This also feeds ComputeBatchTimeout's count * 20 min calculation. Recommend renaming or reworking to a true per-item cap.

✅ Compatibility partitioning and solo-batch handling

The compatibility-key tuple (RunnerType;PlatformFamily;RuntimeFlavor;TargetOS;TargetArchitecture;Scenario;TimeoutClass;PayloadShape) is applied before packing, and items missing required metadata or marked BatchCompatible!=true are correctly kept in solo batches. Non-batch items are emitted through the original HelixCommand/timeout path, preserving stress-scenario behavior. Negative solo batch IDs for large items are preserved. Behavior is well covered by the added unit tests.

✅ Batch runner isolation

DesktopBatchRunner.sh/.cmd give each suite its own HELIX_WORKITEM_UPLOAD_ROOT subdirectory to avoid result collisions, restore the original upload root afterward, aggregate per-suite exit codes, and fail the work item if any suite fails. Exit-code aggregation and the no-zip guard look correct.

💡 Minor observations

• DesktopBatchRunner.sh does not set -euo pipefail; it relies on ${VAR:-} guards and explicit exit-code checks, which is acceptable but slightly more error-prone than an explicit strict mode. Not blocking.
• ComputeBatchTimeout's doc comment references WASM/Cryptography rationale ("~17 min", "WASM startup overhead") while now being used for desktop libraries batching; consider updating the comment so the 20-min/30-min-minimum rationale reflects the desktop path.
• The chmod +x DesktopBatchRunner.sh in the batch runner entry point plus chmod +x RunTests.sh inside the runner is reasonable for Helix payloads; worth a human confirming permission bits survive the zip round-trip on all queues.

Note

This review was generated by this repository's Holistic Review agentic workflow to complement the built-in Copilot review.

Generated by Holistic Review · 133.7 AIC · ⌖ 11 AIC · ⊞ 10K

[github-actions]

The property is named MaxTestsPerWorkItem (defaulting to 20), but it is passed as GroupWorkItems.BatchSize, which the task interprets as the number of bins per compatibility partition (numBatches = Math.Min(BatchSize, smallItems.Count)), not a per-work-item cap on the number of test suites. As a partition grows, each of the (up to) 20 bins receives count/20 suites, so a partition with, say, 400 compatible assemblies yields 20 work items of ~20 suites each, and a larger partition would produce work items with more than 20 suites. In other words this value bounds the number of work items, and there is no actual cap on tests-per-work-item. Consider renaming the property (e.g. MaxLibraryHelixBatches/LibraryHelixBatchCount) to match the semantics, or reworking the task to bin by a true per-item test/size cap if a hard per-work-item limit is intended. This also interacts with ComputeBatchTimeout, which computes timeout as count * 20 min per batch — an unexpectedly large count would inflate timeouts. Worth confirming the intended contract.

[mmitche]

I think we're likely to take a different approach here.

[mmitche]

Closing — not proceeding with this approach (batching libraries test Helix work items by compatibility metadata).