Skip to content

--mount=type=cache unexpectedly empty in parallel builds #7088

Open
Open
--mount=type=cache unexpectedly empty in parallel builds#7088

Description

@ap-wtioit
ap-wtioit
opened on Aug 27, 2026

Contributing guidelines and issue reporting guide

Well-formed report checklist

  • I have found a bug that the documentation does not mention anything about my problem
  • I have found a bug that there are no open or closed issues that are related to my problem
  • I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

When using RUN --mount=target=/cache_dir,type=cache in multiple stages in parallel builds the directory filled in stage1 can be empty in stage2 causing unexpected build errors. Initially discovered when trying to improve locking of build scripts with flock instead of sharing=locked but i think this is the minimal reproducer. Our real usecase are run commands that take about 15-20 Minutes that would block the system for too long when using sharing=locked.

Reproduction

`bash.Dockerfile` Click to expand 
FROM python:3.12-slim-bookworm
RUN --mount=target=/cache_dir,type=cache \
    set -x \
    && find /cache_dir \
    && echo "$((RANDOM % 5000))" > "/cache_dir/$((RANDOM % 5000)).txt" \
    && sleep 2 \
    && echo "OK stage 1"

RUN --mount=target=/cache_dir,type=cache \
    set -x \
    && find /cache_dir \
    && ls -al /cache_dir/*.txt \
    && echo "OK stage 2"
`build_parallel.sh` Click to expand 
#!/usr/bin/env bash
set -ex
builds=${1:-2}
started_builds=()
function prefix() {
    local prefix=$1
    shift
    "$@" 2>&1 | awk "{print \"$prefix: \" \$0}"
    return "${PIPESTATUS[0]}"
}
function run_build() {
    local i=$1
    sleep "$i.5"
    prefix "$i" docker build --no-cache -f "bash$i.Dockerfile" .
}
prefix 1 run_build &
started_builds+=("$!")
for i in $(seq 2 "$builds"); do
    cp bash.Dockerfile "bash$i.Dockerfile"
    run_build "$i" &
    started_builds+=("$!")
done

status=0
for started_build in "${started_builds[@]}"; do
    if ! wait "$started_build"; then
        status=$((status + 1))
    fi
done
if [[ $status != 0 ]] ; then
    echo "There were errors!" 1>&2
    exit 1
fi

Running parallel builds with ./build_parallel.sh shows errors like:

2: 0.246 + ls -al /cache_dir/*.txt
2: 0.247 ls: cannot access '/cache_dir/*.txt': No such file or directory
full build log, Click to expand 
+ builds=2
+ started_builds=()
+ started_builds+=("$!")
+ prefix 1 run_build
+ local prefix=1
+ shift
++ seq 2 2
+ run_build
+ awk '{print "1: " $0}'
+ for i in $(seq 2 "$builds")
+ cp bash.Dockerfile bash2.Dockerfile
1: + local i=
1: + sleep .5
+ started_builds+=("$!")
+ status=0
+ for started_build in "${started_builds[@]}"
+ wait 864374
+ run_build 2
+ local i=2
+ sleep 2.5
1: + prefix '' docker build --no-cache -f bash.Dockerfile .
1: + local prefix=
1: + shift
1: + docker build --no-cache -f bash.Dockerfile .
1: + awk '{print ": " $0}'
+ prefix 2 docker build --no-cache -f bash2.Dockerfile .
+ local prefix=2
+ shift
+ docker build --no-cache -f bash2.Dockerfile .
+ awk '{print "2: " $0}'
2: #0 building with "default" instance using docker driver
2: 
2: #1 [internal] load build definition from bash2.Dockerfile
2: #1 transferring dockerfile: 412B done
2: #1 DONE 0.0s
2: 
2: #2 [internal] load metadata for docker.io/library/python:3.12-slim-bookworm
2: #2 DONE 0.2s
2: 
2: #3 [stage-0 1/3] FROM docker.io/library/python:3.12-slim-bookworm@sha256:0f5b26b9518d002b6173fd61daad821fa340635ebfec5bba471013f9ca114579
2: #3 CACHED
2: 
2: #4 [internal] load .dockerignore
2: #4 transferring context: 311B done
2: #4 DONE 0.0s
2: 
2: #5 [stage-0 2/3] RUN --mount=target=/cache_dir,type=cache     set -x     && find /cache_dir     && echo "$((RANDOM % 5000))" > "/cache_dir/$((RANDOM % 5000)).txt"     && sleep 2     && echo "OK stage 1"
2: #5 0.176 + find /cache_dir
2: #5 0.177 /cache_dir
2: #5 0.177 + echo 0
2: #5 0.177 + sleep 2
2: #5 2.178 + echo OK stage 1
2: #5 2.178 OK stage 1
2: #5 DONE 2.2s
2: 
2: #6 [stage-0 3/3] RUN --mount=target=/cache_dir,type=cache     set -x     && find /cache_dir     && ls -al /cache_dir/*.txt     && echo "OK stage 2"
2: #6 0.245 + find /cache_dir
1: : #0 building with "default" instance using docker driver
1: : 
1: : #1 [internal] load build definition from bash.Dockerfile
1: : #1 transferring dockerfile: 411B done
1: : #1 DONE 0.0s
1: : 
1: : #2 [internal] load metadata for docker.io/library/python:3.12-slim-bookworm
1: : #2 DONE 0.2s
1: : 
1: : #3 [internal] load .dockerignore
1: : #3 transferring context: 311B done
1: : #3 DONE 0.0s
1: : 
1: : #4 [stage-0 1/3] FROM docker.io/library/python:3.12-slim-bookworm@sha256:0f5b26b9518d002b6173fd61daad821fa340635ebfec5bba471013f9ca114579
1: : #4 CACHED
1: : 
1: : #5 [stage-0 2/3] RUN --mount=target=/cache_dir,type=cache     set -x     && find /cache_dir     && echo "$((RANDOM % 5000))" > "/cache_dir/$((RANDOM % 5000)).txt"     && sleep 2     && echo "OK stage 1"
1: : #5 0.176 + find /cache_dir
1: : #5 0.177 /cache_dir
1: : #5 0.177 + echo 0
1: : #5 0.177 + sleep 2
1: : #5 2.178 + echo OK stage 1
1: : #5 2.178 OK stage 1
1: : #5 DONE 2.2s
1: : 
1: : #6 [stage-0 3/3] RUN --mount=target=/cache_dir,type=cache     set -x     && find /cache_dir     && ls -al /cache_dir/*.txt     && echo "OK stage 2"
1: : #6 0.245 + find /cache_dir
1: : #6 0.246 /cache_dir
1: : #6 0.246 + ls -al /cache_dir/*.txt
1: : #6 0.247 ls: cannot access '/cache_dir/*.txt': No such file or directory
1: : #6 ERROR: process "/bin/sh -c set -x     && find /cache_dir     && ls -al /cache_dir/*.txt     && echo \"OK stage 2\"" did not complete successfully: exit code: 2
1: : ------
1: :  > [stage-0 3/3] RUN --mount=target=/cache_dir,type=cache     set -x     && find /cache_dir     && ls -al /cache_dir/*.txt     && echo "OK stage 2":
1: : 0.245 + find /cache_dir
1: : 0.246 /cache_dir
1: : 0.246 + ls -al /cache_dir/*.txt
1: : 0.247 ls: cannot access '/cache_dir/*.txt': No such file or directory
1: : ------
1: : bash.Dockerfile:9
1: : --------------------
1: :    8 |     
1: :    9 | >>> RUN --mount=target=/cache_dir,type=cache \
1: :   10 | >>>     set -x \
1: :   11 | >>>     && find /cache_dir \
1: :   12 | >>>     && ls -al /cache_dir/*.txt \
1: :   13 | >>>     && echo "OK stage 2"
1: :   14 |     
1: : --------------------
1: : ERROR: failed to build: failed to solve: process "/bin/sh -c set -x     && find /cache_dir     && ls -al /cache_dir/*.txt     && echo \"OK stage 2\"" did not complete successfully: exit code: 2
1: + return 1
+ return 1
+ status=1
+ for started_build in "${started_builds[@]}"
+ wait 864380
2: #6 0.246 /cache_dir
2: #6 0.246 + ls -al /cache_dir/*.txt
2: #6 0.247 ls: cannot access '/cache_dir/*.txt': No such file or directory
2: #6 ERROR: process "/bin/sh -c set -x     && find /cache_dir     && ls -al /cache_dir/*.txt     && echo \"OK stage 2\"" did not complete successfully: exit code: 2
2: ------
2:  > [stage-0 3/3] RUN --mount=target=/cache_dir,type=cache     set -x     && find /cache_dir     && ls -al /cache_dir/*.txt     && echo "OK stage 2":
2: 0.245 + find /cache_dir
2: 0.246 /cache_dir
2: 0.246 + ls -al /cache_dir/*.txt
2: 0.247 ls: cannot access '/cache_dir/*.txt': No such file or directory
2: ------
2: bash2.Dockerfile:9
2: --------------------
2:    8 |     
2:    9 | >>> RUN --mount=target=/cache_dir,type=cache \
2:   10 | >>>     set -x \
2:   11 | >>>     && find /cache_dir \
2:   12 | >>>     && ls -al /cache_dir/*.txt \
2:   13 | >>>     && echo "OK stage 2"
2:   14 |     
2: --------------------
2: ERROR: failed to build: failed to solve: process "/bin/sh -c set -x     && find /cache_dir     && ls -al /cache_dir/*.txt     && echo \"OK stage 2\"" did not complete successfully: exit code: 2
+ return 1
+ status=2
+ [[ 2 != 0 ]]
+ echo 'There were errors!'
There were errors!
+ exit 1

Running single builds with ./build_parallel.sh 1 always succeeds.

Version information

docker version | awk '{print "#" $0}'
#Client: Docker Engine - Community
# Version:           29.7.2
# API version:       1.55
# Go version:        go1.26.5
# Git commit:        a7dcaa6
# Built:             Wed Aug  5 18:28:53 2026
# OS/Arch:           linux/amd64
# Context:           default
#
#Server: Docker Engine - Community
# Engine:
#  Version:          29.7.2
#  API version:      1.55 (minimum version 1.40)
#  Go version:       go1.26.5
#  Git commit:       6a43e3d
#  Built:            Wed Aug  5 18:28:53 2026
#  OS/Arch:          linux/amd64
#  Experimental:     false
# containerd:
#  Version:          v2.3.3
#  GitCommit:        aad11006b869517fcd3009450b6f82da282e1a9b
# runc:
#  Version:          1.4.3
#  GitCommit:        v1.4.3-0-gbb14dabe
# docker-init:
#  Version:          0.19.0
#  GitCommit:        de40ad0
docker buildx version | awk '{print "#" $0}'
#github.com/docker/buildx v0.36.1 1d8dde89b8aba914e05e45366770736fea1fd690
docker buildx ls | awk '{print "#" $0}'
#NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
#default*      docker                                 
# \_ default    \_ default       running   v0.32.2    linux/amd64 (+4), linux/arm64, linux/arm (+2), linux/ppc64le, (7 more)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions