[9.2.0] Recover lost inputs from deduplicated Merkle subtrees under path mapping - #30193
Merged
Merged
Conversation
…ing (bazelbuild#30085) Merkle subtrees for tree artifacts are deduplicated across actions by content. When such a shared subtree is uploaded for remote execution and one of its blobs is missing from the CAS, the resulting `CacheNotFoundException` is annotated with the exec path of whichever action happened to compute the shared subtree. With path mapping, other actions that join the deduplicated computation can then no longer map the lost input back to their own inputs, so the `BulkTransferException` surfaces as a fatal `EXEC_IO_EXCEPTION` instead of a `LostInputsExecException` and action rewinding never runs. This change adds the unmapped exec path to the cache key of a tree artifact to avoid a shared upload in this case. Other artifact types are not affected and the fast path of a remote cache check without uploads is unaffected. Fixes bazelbuild#30065. No - [x] I have added tests for the new use cases (if any). - [ ] I have updated the documentation (if applicable). RELNOTES: Fixed a spurious remote-execution failure with `--experimental_output_paths=strip` where a lost input in a tree artifact shared between actions could not be recovered by action rewinding. Closes bazelbuild#30085. PiperOrigin-RevId: 944321975 Change-Id: If36408a56ab8137b915970827ebbd7a91d9ad9c1 (cherry picked from commit 412de86)
github-actions
Bot
added
team-Remote-Exec
Wyverald
approved these changes
Jul 8, 2026
github-actions
Bot
removed
the
awaiting-review
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/datadog-agent
that referenced
this pull request
Jul 14, 2026
### What does this PR do? Bump the pinned Bazel version from 9.1.1 to 9.2.0 and refresh `MODULE.bazel.lock` accordingly. ### Motivation Bazel 9.2.0 ships fixes this repo has been working around over the past month. #### Windows issues Our contributions: - bazelbuild/bazel#29984, which should remove the need for temporary countermeasures we had to come up with: - DataDog/datadog-agent-buildimages#1212 (alas ineffective), - DataDog/datadog-agent-buildimages#1215 (effective), - #52500, - bazelbuild/bazel#30182, which should remove the need for: - #52061, - #53591. #### Caching issues - bazelbuild/bazel#29791, - bazelbuild/bazel#29868, - bazelbuild/bazel#29885, - bazelbuild/bazel#30193, a caveat mentioned in: - #53356. #### Others Also worth noting, though not tied to a specific incident here: - bazelbuild/bazel#29807, - bazelbuild/bazel#29898, - bazelbuild/bazel#30194. ### Describe how you validated your changes `bazel version` reports 9.2.0 after `bazelisk` re-bootstraps. `bazel build //pkg/config/schema/...` completes successfully. `bazel shutdown` followed by `bazel mod deps --lockfile_mode=refresh` leaves `MODULE.bazel.lock` unchanged. ### Additional Notes None of the .bazelrc UAC/8dot3 workarounds (DataDog/datadog-agent-buildimages#1215, #52500, #52061, #53591) are removed here. Each needs its own re-verification on Windows before being reverted as follow-ups. Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
to DataDog/datadog-agent
that referenced
this pull request
Aug 21, 2026
…xes (#53619) (#55228) Bump the pinned Bazel version from 9.1.1 to 9.2.0 and refresh `MODULE.bazel.lock` accordingly. Bazel 9.2.0 ships fixes this repo has been working around over the past month. Our contributions: - bazelbuild/bazel#29984, which should remove the need for temporary countermeasures we had to come up with: - DataDog/datadog-agent-buildimages#1212 (alas ineffective), - DataDog/datadog-agent-buildimages#1215 (effective), - #52500, - bazelbuild/bazel#30182, which should remove the need for: - #52061, - #53591. - bazelbuild/bazel#29791, - bazelbuild/bazel#29868, - bazelbuild/bazel#29885, - bazelbuild/bazel#30193, a caveat mentioned in: - #53356. Also worth noting, though not tied to a specific incident here: - bazelbuild/bazel#29807, - bazelbuild/bazel#29898, - bazelbuild/bazel#30194. `bazel version` reports 9.2.0 after `bazelisk` re-bootstraps. `bazel build //pkg/config/schema/...` completes successfully. `bazel shutdown` followed by `bazel mod deps --lockfile_mode=refresh` leaves `MODULE.bazel.lock` unchanged. None of the .bazelrc UAC/8dot3 workarounds (DataDog/datadog-agent-buildimages#1215, #52500, #52061, #53591) are removed here. Each needs its own re-verification on Windows before being reverted as follow-ups. (cherry picked from commit 695123c) <!--Please give us some feedback on your experience writing this PR ! https://app.datadoghq.com/forms/43db4c02-6837-400c-8083-692e141b1b88 !--> ### What does this PR do? ### Motivation ### Describe how you validated your changes ### Additional Notes Co-authored-by: rdesgroppes <rdesgroppes@gmail.com> Co-authored-by: ali.benabdallah <ali.benabdallah@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merkle subtrees for tree artifacts are deduplicated across actions by content. When such a shared subtree is uploaded for remote execution and one of its blobs is missing from the CAS, the resulting
CacheNotFoundExceptionis annotated with the exec path of whichever action happened to compute the shared subtree. With path mapping, other actions that join the deduplicated computation can then no longer map the lost input back to their own inputs, so theBulkTransferExceptionsurfaces as a fatalEXEC_IO_EXCEPTIONinstead of aLostInputsExecExceptionand action rewinding never runs.This change adds the unmapped exec path to the cache key of a tree artifact to avoid a shared upload in this case. Other artifact types are not affected and the fast path of a remote cache check without uploads is unaffected.
Fixes #30065.
RELNOTES: Fixed a spurious remote-execution failure with
--experimental_output_paths=stripwhere a lost input in a tree artifact shared between actions could not be recovered by action rewinding.Closes #30085.
PiperOrigin-RevId: 944321975
Change-Id: If36408a56ab8137b915970827ebbd7a91d9ad9c1
(cherry picked from commit 412de86)
Closes #30103