Background
To ensure Docker Hub displays the correct/most-recent tag by default, we push image tags in a specific sorted order (see #484) rather than in parallel. This guarantees the last layer pushed corresponds to the newest version, keeping Docker Hub's ordering sane for users browsing tags.
The tradeoff is that the sorted/ordered push is effectively serialized, which slows down the overall publish stage compared to pushing all tags in parallel.
Idea
Investigate whether we can get the speed benefits of parallel pushing and keep correct sort order by doing a two-phase push:
- Parallel push phase: Push all image tags to Docker Hub in parallel (order not guaranteed), to get the bulk of the layer data uploaded as fast as possible.
- Sorted re-push phase: After the parallel phase completes, re-push the tags again in the correct sorted order. Since the layers already exist in the registry from phase 1, this second push should be nearly instant (Docker Hub/registry layer dedup means only manifest updates are needed, no blob re-upload) while still producing the final correct tag ordering.
If this works as expected, we get the best of both worlds: the fast wall-clock time of parallel pushing, with the final state matching what a fully serialized/sorted push would produce.
Goals of this spike
Related
Background
To ensure Docker Hub displays the correct/most-recent tag by default, we push image tags in a specific sorted order (see #484) rather than in parallel. This guarantees the last layer pushed corresponds to the newest version, keeping Docker Hub's ordering sane for users browsing tags.
The tradeoff is that the sorted/ordered push is effectively serialized, which slows down the overall publish stage compared to pushing all tags in parallel.
Idea
Investigate whether we can get the speed benefits of parallel pushing and keep correct sort order by doing a two-phase push:
If this works as expected, we get the best of both worlds: the fast wall-clock time of parallel pushing, with the final state matching what a fully serialized/sorted push would produce.
Goals of this spike
bakery(or wherever the current publish/push logic lives) and measure end-to-end publish stage time before/after.Related