[UII] Support searchAfter and PIT (point-in-time) parameters for get agents list API

[jen-huang]

Summary

Resolves #206924.

This PR adds the following query parameters to the agent list API (GET /api/fleet/agents) in order to enable fetching beyond the first 10,000 hits:

    searchAfter?: string;
    openPit?: boolean;
    pitId?: string;
    pitKeepAlive?: string;

The list agent API response can now include the following properties

    // the PIT ID used
    pit?: string;

    // stringified version of the last agent's `sort` field,
    // can be passed as `searchAfter` in the next request
    nextSearchAfter? string;

• searchAfter can be used with or without a pitId. If using searchAfter, page parameter is not accepted.

• searchAfter expects a stringified array. (Reviewers: I couldn't get the Kibana request schema to accept a multi-part query param and convert it to an array... I think this would be better, please let me know if you know how to get that to work 🙏)

• pitKeepAlive duration (i.e. 30s, 1m, etc) must be present when opening a PIT or retrieving results using a PIT ID.

• These can be used with the existing sortField and sortOrder params. They default to enrolled_at and desc respectively.

Example using only searchAfter:

# Retrieve the first 10k hits
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000'

# Grab the `nextSearchAfter` param from the response
# Pass it to the new request to retrieve the next page of 10k hits
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000&searchAfter=<nextSearchAfter>'

Example using searchAfter with point-in-time parameters:

# Retrieve the first 10k hits and open a PIT
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000&openPit=true&pitKeepAlive=5m'

# Grab the `pit` ID from the response
# Grab the `nextSearchAfter` param from the response
# Pass both to the new request to retrieve the next page of 10k hits
curl -X GET 'http://<user>:<pass>@<kibana url>/api/fleet/agents?perPage=10000&searchAfter=<nextSearchAfter>&pitId=<pit id>&pitKeepAlive=5m'

Testing

I recommend using scripts/create_agents to generate bulk agents and testing the above requests. You can generate new agents between PIT requests to test that using a PIT ID retains the original state. (An API functional test was added for this)

Note: you may need to add &showInactive=true to all requests if your fake agents become inactive.

TBD

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Unit or functional tests were updated or added to match the most common scenarios
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jen-huang]

reposting from PR description:

@elastic/fleet I couldn't get the Kibana request schema to accept a multi-part query param and convert it to an array... I think this would be better, please let me know if you know how to get that to work 🙏

changing this to schema.maybe(schema.arrayOf(schema.any())) and passing params like &searchAfter=123&searchAfter=test or even &searchAfter[]=123&searchAfter[]=test didn't work. it only returned error about searchAfter not matching expected schema

[nchaulet]

it seems when using an array of any it do not parse correctly number or string, (it seems expected as there not really way to get that), so I guess it's okay to use a string here I am wondering if it will make the API easier to use if we add a meta to agent list response something like nextSearchAfter: "[stringified...]" so user will not have to know about getting the last sort field of the agents, WDYT?

[jen-huang]

I think this is a good idea, I will take a look

[jen-huang]

@nchaulet added in 0783feb and updated PR description. it helped writing the API tests a little easier too :)

[jen-huang]

/ci test this

[nchaulet]

code LGTM 🚀 should we add some doc for that? having a recipe in the doc on how to query for more than 10k agents could be a good idea

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9c9d29b

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1337	1339	+2

Unknown metric groups

API count

id	before	after	diff
fleet	1465	1467	+2

History

• 💔 Build #281655 failed fddfbf3

cc @jen-huang

[jen-huang]

should we add some doc for that? having a recipe in the doc on how to query for more than 10k agents could be a good idea

I filed elastic/ingest-docs#1726. Thanks for quick re-review!