Skip to content

[9.1.1] Fix JVM crash from VerifyException in GrpcCacheClient.onNext() (https://github.com/bazelbuild/bazel/pull/29316) - #29583

Merged
iancha1992 merged 1 commit into
bazelbuild:release-9.1.1from
bazel-io:cp29316-9.1.1-210401
May 27, 2026
Merged

[9.1.1] Fix JVM crash from VerifyException in GrpcCacheClient.onNext() (https://github.com/bazelbuild/bazel/pull/29316)#29583
iancha1992 merged 1 commit into
bazelbuild:release-9.1.1from
bazel-io:cp29316-9.1.1-210401

Conversation

@bazel-io

Copy link
Copy Markdown
Member

Description

When a remote cache blob download is cancelled (e.g. dynamic execution choosing the local branch), the output stream is closed via a directExecutor() listener on the download future (CombinedCache#downloadFile). A pending onNext() callback can still be drained afterwards via DelayedClientCall's pending-callback path, which, unlike the normal StreamObserver path, does not convert RuntimeExceptions into onError(). The onNext() handler writes to the closed output stream, then catches the IOException, and throws VerifyException, which escapes to the gRPC executor's worker thread and hits Bazel's default uncaught exception handler.

Replace the throw with graceful error handling: cancel the gRPC stream via the stored requestStream reference and propagate the error through the SettableFuture. In the common case (future already cancelled by dynamic execution), setException is a no-op and the build continues using the local branch.

Motivation

Fixes #22930

Build API Changes

No

Checklist

  • I have added tests for the new use cases (if any).
  • I have updated the documentation (if applicable).

Release Notes

RELNOTES: None

Closes #29316.

PiperOrigin-RevId: 901186846
Change-Id: Id9d9ca1bd824c0d4c62d69f05b6c9dbf606ab3ec

Commit 588172a

@bazel-io
bazel-io requested a review from a team as a code owner May 18, 2026 21:04
@bazel-io bazel-io added team-Remote-Exec Issues and PRs for the Execution (Remote) team awaiting-review PR is awaiting review from an assigned reviewer labels May 18, 2026
@bazel-io
bazel-io requested review from coeuvre and fmeum May 18, 2026 21:04
@iancha1992
iancha1992 enabled auto-merge (squash) May 18, 2026 21:24
@iancha1992
iancha1992 removed the request for review from fmeum May 18, 2026 21:50
@iancha1992
iancha1992 force-pushed the cp29316-9.1.1-210401 branch from 9617204 to e003233 Compare May 19, 2026 17:49
…build#29316)

### Description
When a remote cache blob download is cancelled (e.g. dynamic execution choosing the local branch), the output stream is closed via a directExecutor() listener on the download future (CombinedCache#downloadFile). A pending onNext() callback can still be drained afterwards via DelayedClientCall's pending-callback path, which, unlike the normal StreamObserver path, does not convert RuntimeExceptions into onError(). The onNext() handler writes to the closed output stream, then catches the IOException, and throws VerifyException, which escapes to the gRPC executor's worker thread and hits Bazel's default uncaught exception handler.

Replace the throw with graceful error handling: cancel the gRPC stream via the stored requestStream reference and propagate the error through the SettableFuture. In the common case (future already cancelled by dynamic execution), setException is a no-op and the build continues using the local branch.

### Motivation
Fixes bazelbuild#22930

### Build API Changes

No

### Checklist

- [ ] I have added tests for the new use cases (if any).
- [ ] I have updated the documentation (if applicable).

### Release Notes

RELNOTES: None

Closes bazelbuild#29316.

PiperOrigin-RevId: 901186846
Change-Id: Id9d9ca1bd824c0d4c62d69f05b6c9dbf606ab3ec
@iancha1992
iancha1992 force-pushed the cp29316-9.1.1-210401 branch from e003233 to 235a920 Compare May 20, 2026 20:58
@iancha1992
iancha1992 requested a review from Wyverald May 27, 2026 15:33
@iancha1992
iancha1992 added this pull request to the merge queue May 27, 2026
Merged via the queue into bazelbuild:release-9.1.1 with commit c7a3bb4 May 27, 2026
45 checks passed
@github-actions github-actions Bot removed the awaiting-review PR is awaiting review from an assigned reviewer label May 27, 2026
rdesgroppes added a commit to DataDog/datadog-agent that referenced this pull request Jun 9, 2026
### What does this PR do?
Bump `.bazelversion` from 9.1.0 to 9.1.1.

### Motivation
9.1.1 ships several fixes in the Remote Execution section, some of which
are relevant to remote caching:
- bazelbuild/bazel#29583,
- bazelbuild/bazel#29593,
- bazelbuild/bazel#29596,
- bazelbuild/bazel#29614,
- bazelbuild/bazel#29616
  May affect us: `.adms/bazel/bazel.bazelrc` sets this flag, so the
  9.1.0 regression could have caused two configs to stack silently,
  potentially bypassing the ADMS mirror,
- bazelbuild/bazel#29671,
- bazelbuild/bazel#29677.

### Describe how you validated your changes
CI.

### Additional Notes
See:
https://github.com/bazelbuild/bazel/releases/tag/9.1.1
gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/datadog-agent that referenced this pull request Jun 9, 2026
### What does this PR do?
Bump `.bazelversion` from 9.1.0 to 9.1.1.

### Motivation
9.1.1 ships several fixes in the Remote Execution section, some of which are relevant to remote caching:
- bazelbuild/bazel#29583,
- bazelbuild/bazel#29593,
- bazelbuild/bazel#29614,
- bazelbuild/bazel#29616 - `.adms/bazel/bazel.bazelrc` sets this flag, so the 9.1.0 regression could have caused two configs to stack silently (i.e. potentially bypassing the ADMS mirror),
- bazelbuild/bazel#29677.

### Describe how you validated your changes
CI.

### Additional Notes
See: https://github.com/bazelbuild/bazel/releases/tag/9.1.1

Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
Lewis-E pushed a commit to DataDog/datadog-agent that referenced this pull request Jun 12, 2026
### What does this PR do?
Bump `.bazelversion` from 9.1.0 to 9.1.1.

### Motivation
9.1.1 ships several fixes in the Remote Execution section, some of which are relevant to remote caching:
- bazelbuild/bazel#29583,
- bazelbuild/bazel#29593,
- bazelbuild/bazel#29614,
- bazelbuild/bazel#29616 - `.adms/bazel/bazel.bazelrc` sets this flag, so the 9.1.0 regression could have caused two configs to stack silently (i.e. potentially bypassing the ADMS mirror),
- bazelbuild/bazel#29677.

### Describe how you validated your changes
CI.

### Additional Notes
See: https://github.com/bazelbuild/bazel/releases/tag/9.1.1

Co-authored-by: regis.desgroppes <regis.desgroppes@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team-Remote-Exec Issues and PRs for the Execution (Remote) team

3 participants