Skip to content

[SPIKE] Investigate parallel push + sorted re-push to speed up publish stage #759

Description

@ianpittwood

Background

To ensure Docker Hub displays the correct/most-recent tag by default, we push image tags in a specific sorted order (see #484) rather than in parallel. This guarantees the last layer pushed corresponds to the newest version, keeping Docker Hub's ordering sane for users browsing tags.

The tradeoff is that the sorted/ordered push is effectively serialized, which slows down the overall publish stage compared to pushing all tags in parallel.

Idea

Investigate whether we can get the speed benefits of parallel pushing and keep correct sort order by doing a two-phase push:

  1. Parallel push phase: Push all image tags to Docker Hub in parallel (order not guaranteed), to get the bulk of the layer data uploaded as fast as possible.
  2. Sorted re-push phase: After the parallel phase completes, re-push the tags again in the correct sorted order. Since the layers already exist in the registry from phase 1, this second push should be nearly instant (Docker Hub/registry layer dedup means only manifest updates are needed, no blob re-upload) while still producing the final correct tag ordering.

If this works as expected, we get the best of both worlds: the fast wall-clock time of parallel pushing, with the final state matching what a fully serialized/sorted push would produce.

Goals of this spike

  • Confirm Docker Hub / registry behavior: does re-pushing a tag whose layers already exist in the registry actually skip layer upload and only update the manifest (i.e. is the second push fast)?
  • Confirm that the final tag order on Docker Hub reflects the order of the second (sorted) push, not the first (parallel) push.
  • Prototype this in bakery (or wherever the current publish/push logic lives) and measure end-to-end publish stage time before/after.
  • Identify any edge cases (e.g. partial failures in phase 1, manifest lists/multi-arch images, rate limiting from double-pushing).
  • Write up findings and a recommendation (adopt, adopt with caveats, or not worth it) for follow-up implementation work.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    cicdContinuous integration/deploymentdockerRelated to container images we produceneeds proposalNeeds a proposaltech debtTechnical debt we should address

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions