Add fleet policy revisions cleanup task

[MichelLosier]

Summary

Resolves: #119963
PR to add new Kibana config to allow list: https://github.com/elastic/cloud/pull/149322

• Adds the background task FleetPolicyRevisionsCleanupTask which removes excess policy revisions from the .fleet-policies index
• Enabled by default, disabled by new feature flag:

 xpack.fleet.experimentalFeatures:
  enableFleetPolicyRevisionsCleanupTask: true

• Configurable with the following:

xpack.fleet.fleetPolicyRevisionsCleanup:
  interval: string // Time interval expression. Default: '1h'
  max_revisions: number // Maximum amount of policy revisions. Default: 10.
  max_policies_per_run: number // Maximum policies to delete revisions from per run. Default 100.

• Also adds an internal endpoint for the same operation:

POST kbn:/internal/fleet/agent_policies/_cleanup_revisions
{
    "maxRevisions": number, // max revisions a policy should have
    "maxPolicies": number, // max policies to process
}

Release notes

• Adds the background task FleetPolicyRevisionsCleanupTask which removes excess policy revisions from the .fleet-policies index

Details

• The task will look for the top max_policies_per_run sorted by document count, filtered down to those exceeding the max_revisions
• It will then lookup the minimum revision index used by agents for each policy.
• Policy revision documents will be deleted for each policy that have an idx at or below the max_revisions cutoff, which is calculated from the minimum revision used by agents for that policy or the latest revision idx
• The delete_by_query for removing policy revisions is configured for a max_docs of 5000, which means at most 1 job run will only delete 5000 at a time.

The idea here is to optimize for low resource environments, and slowly burn down revisions for environments that have had revisions counts run away. The exposed configuration allows for more aggressive use for those that need it.

Manual testing

This PR introduces a --revisions flag for the create_agents script to generate a set amount of revisions per policy.
Example:

node scripts/create_agents --kibana http://0.0.0.0:5601/kibana --count 5 --batches 100 --concurrentBatches 100  --revisions 1000

Will create 100 policies with 5 agents and 1000 revisions each.

Configure kibana.dev.yml to the task parameters, as described above, you want to test with.

Check revisions counts in the kibana dev console:

GET .fleet-policies/_search
{
	"size": 0,
	"aggs": {
		"latest_revisions_by_policy_id": {
			"terms": {
				"field": "policy_id",
				"order": { "_count": "desc" }
			},
			"aggs": {
				"latest_revision": {
					"max": {
						"field": "revision_idx"
					}
				}
			}
		}
	}
}

Additionally the new internal endpoint can also be used to perform the same operation in the dev console:

POST kbn:/internal/fleet/agent_policies/_cleanup_revisions?apiVersion=1
{
    "maxRevisions": 10,
    "maxPolicies": 100
}

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
• Review the backport guidelines and apply applicable backport:* labels.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• See some risk examples
• ...

[MichelLosier]

/ci

[MichelLosier]

Open for early review. Next steps here:

• Add internal API endpoint to run this job on demand
  • Use internal API for FTR tests

[MichelLosier]

/ci

[nchaulet]

a few suggestions, code LGTM 🚀

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[shahzad31]

@MichelLosier would it be possible to backport this to 8.19?

[kibanamachine]

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/20076604803

[kibanamachine]

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/20076604866

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 242612

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 242612

Questions ?

Please refer to the Backport tool documentation

[MichelLosier]

💚 All backports created successfully

Status	Branch	Result
✅	9.2
✅	9.1
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[lucabelluccini]

Thanks for the backports!