[CI] Add sparse-checkout plugin to remaining pipelines and pipeline steps

[mrodm]

Proposed commit message

Extend the use of sparse-checkout#v1.6.0 introduced in #19006 to cover
all remaining Buildkite pipelines and individual pipeline steps that do
not need a full repository checkout.

WHAT:

• catalog-info.yaml: added initial_step_plugins with
  sparse-checkout#v1.6.0 to the five pipelines that were missing it:
  schedule-daily, schedule-weekly, serverless, backport, and
  publish. The integrations and integrations-test-stack pipelines
  already had it from bk: run faster builds #19006.

• .buildkite/pipeline.yml: added the plugin at step level to the six
  steps whose scripts only access .buildkite/ and .go-version:

  • Get reference from target branch
  • Python CI scripts unit tests
  • Sources Junit annotate
  • Publish benchmarks
  • Junit annotate
  • Report failed tests

• .buildkite/pipeline.serverless.yml: added the plugin to the
  Junit annotate and Report failed tests steps for the same reason.

WHY:

The integrations repository is large. Steps that only run scripts under
.buildkite/ or download artifacts were still paying the cost of a full
clone. Sparse checkout limits the checkout to .buildkite and
.go-version (required by the pre-command hook) for every step that
doesn't touch package sources.

Author's Checklist

• Remove debugging changes before merging: common.sh nginx filter in list_all_directories, DRY_RUN: "true" and CI_MAX_TESTS_REPORTED: 1 in both pipeline files, and the commented-out if: conditions on the Report failed tests steps.
• Validate that it works for Serverless pipeline
• Validate that it works for PR pipeline.
• Validate that it works for the report failure steps (executed in daily jobs).

How to test this PR locally

Trigger a build in Buildkite and verify that the affected steps show a
sparse checkout in their logs (only .buildkite/ and .go-version are
checked out) and complete successfully.

Related issues

• Relates bk: run faster builds #19006

---

This PR was generated with the assistance of Claude (claude-sonnet-4-6).

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: caebbd7

Failed CI Steps

• :pipeline:⬆️ Upload Pipeline: .buildkite/pipeline.yml

History

• 💔 Build #42886 failed 0ac7291

cc @mrodm

[github-actions]

TL;DR

The failing Buildkite step (:pipeline::arrow_up: Upload Pipeline) is not failing on pipeline upload logic itself; it fails earlier in the repository post-checkout merge because the sparse-checkout step is shallow and the PR commit is treated as unrelated history. Immediate fix: ensure full history before the merge hook runs (or stop forcing unrelated-history merges).

Remediation

• In steps that use sparse-checkout and still rely on .buildkite/hooks/post-checkout merging target branch + PR commit, enable unshallowing (e.g. under sparse-checkout post_checkout: { unshallow: true }) so git merge has a real merge-base.
• Revert --allow-unrelated-histories in .buildkite/hooks/post-checkout and keep normal merge behavior; with full history, this merge should fast-forward/merge cleanly.
• Re-run Buildkite for PR #19014 after the above adjustment.

Investigation details

Root Cause

post-checkout now merges with --allow-unrelated-histories at .buildkite/hooks/post-checkout:47, which masks the real issue (shallow checkout/no shared ancestry available locally) and causes a content merge attempt with conflicts.

Relevant code references:

• .buildkite/hooks/post-checkout:47 (git merge --no-edit --allow-unrelated-histories "${BUILDKITE_COMMIT}")
• .buildkite/pipeline.yml:47-52 (new sparse-checkout#v1.6.0 on the target-branch reference step)

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/42887
• Job/step: :pipeline::arrow_up: Upload Pipeline: .buildkite/pipeline.yml
• Key log excerpt:
  • WARN No meta-data value exists with key "REPOSITORY_TARGET_BRANCH_COMMIT"
  • CONFLICT (add/add): Merge conflict in .buildkite/hooks/post-checkout
  • CONFLICT (add/add): Merge conflict in .buildkite/pipeline.serverless.yml
  • CONFLICT (add/add): Merge conflict in .buildkite/pipeline.yml
  • CONFLICT (add/add): Merge conflict in .buildkite/scripts/common.sh
  • CONFLICT (add/add): Merge conflict in catalog-info.yaml
  • Error: running "repository post-checkout" shell hook

The add/add conflict pattern across files already shared by main and this PR is consistent with missing local merge-base history (shallow checkout), not a normal line-level conflict in current changes.

Verification

• Local merge of main -> caebbd7 with full history fast-forwards cleanly.
• Buildkite failure originates before pipeline upload command execution, during repo post-checkout hook merge.

Follow-up

If needed, apply unshallowing only to the pipeline-upload/reference steps first to keep sparse checkout benefits while restoring merge-hook correctness.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• #19014 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [CI] Add sparse-checkout plugin to remaining pipelines and pipeline steps #19014 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.