allows PIT to be cross project

[piergm]

Enables _pit endpoint to be cross project. (allowsCrossProject return true)
Disable cross project chaining (by calling indicesOptionsForCrossProjectFanout that sets CrossProjectModeOptions back to default)
Adds project routing parameter in _pit if crossProjectEnabled (query param only)
Disallows projectRouting for _search if in CPS context and pit id is provided.

Missing: Allow UIAM for close pit endpoint.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[elasticsearchmachine]

Hi @piergm, I've created a changelog YAML for you.

[pawankartik-elastic]

Initial comments.

[quux00]

First pass review complete with suggestions and questions.

[quux00]

For the description comment:

Missing: Allow UIAM for close pit endpoint.

Do we intend to tackle that in this PR or will be a follow-on? (Asking partly because we may not want Wei to do QA tests on this until close PIT is done?)

[luigidellaquila]

Thanks @piergm, I had a first look and I tested it with EQL, see my comment below

[piergm]

@quux00

Do we intend to tackle that in this PR or will be a follow-on? (Asking partly because we may not want Wei to do QA tests on this until close PIT is done?)

Really depends on when the non-Replaceable endpoints will allow UIAM. If it's early next week we can wait, otherwise I'd merge this and do a follow-up for PIT close.

With that said when we merge this PR we can add to the E2E tests PIT open and search with PIT IMO.

[quux00]

LGTM.

I left a few questions and nits only.

[quux00]

(asking to learn) - When is this set vs. taking the default of 3 seconds? I think for CPS we want to always ensure it is 3 seconds, so is search.ccs.force_connect_timeout always defined (and set to 3 seconds) for CPS or should we be using the CrossProjectModeDecider here to determine what timeout setting to use?

[piergm]

It's only being used by CPS because we only call it in executeOpenPitCrossProject, so I don't think we need to have another if statement for CrossProjectModeDecider.
On when/if is set I think we (ES eng) are the only one that can override/set this setting through gitops repo

[pawankartik-elastic]

When is this set

When we manually "inject" the corresponding value from our end for Serverless CPS. I had a brief chat with Matteo about this, and there's scope for improvement/refactoring here: we'll abstract this away and move it elsewhere so that we won't have to:

1. Hardcode the setting in n number of places across the codebase,
2. Repeatedly check for the existence of this setting, and,
3. Repeatedly declare the fallback values.

Currently, it's being used in Search, Field Caps, and PIT (this).

[quux00]

Approved, but I'd like to have Pawan's approval before merging.

[pawankartik-elastic]

LGTM. Any outstanding comments are minor nitpicks and not blockers. Approving.