[Fleet] enable feature flag enableSyncIntegrationsOnRemote by juliaElastic · Pull Request #220215 · elastic/kibana

juliaElastic · 2025-05-06T09:31:12Z

Summary

Closes #217490

Enable feature flag enableSyncIntegrationsOnRemote
Added check to hide sync integrations feature in serverless
Moved creating the follower index from Fleet setup to output create/update API and async task (create if does not exist)
Follower index is not hidden for now, because if hidden, it's not showing up on the CCR UI

elasticmachine · 2025-05-06T09:31:19Z

Pinging @elastic/fleet (Team:Fleet)

nchaulet

🚀

juliaElastic · 2025-05-07T08:01:24Z

I found that there is an error in serverless when trying to query Cluster.remoteInfo.
I think we have to disable the feature altogether in serverless, as CCS and CCR is not supported in serverless.

[00:00:28]         │ debg --- retry.try failed again with the same message...
[00:00:29]         │ proc [kibana] [2025-05-07T07:28:12.465+00:00][ERROR][plugins.fleet] ResponseError: This API is unavailable in the version of Elasticsearch you are using.
[00:00:29]         │ proc [kibana]     at KibanaTransport._request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@elastic/transport/lib/Transport.js:547:27)
[00:00:29]         │ proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:95:5)
[00:00:29]         │ proc [kibana]     at /opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@elastic/transport/lib/Transport.js:645:32
[00:00:29]         │ proc [kibana]     at KibanaTransport.request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@elastic/transport/lib/Transport.js:641:20)
[00:00:29]         │ proc [kibana]     at KibanaTransport.request (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:60:16)
[00:00:29]         │ proc [kibana]     at Cluster.remoteInfo (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@elastic/elasticsearch/lib/api/api/cluster.js:688:16)
[00:00:29]         │ proc [kibana]     at createCCSIndexPatterns (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/fleet-plugin/server/services/setup/fleet_synced_integrations.js:123:22)
[00:00:29]         │ proc [kibana]     at createSetupSideEffects (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/fleet-plugin/server/services/setup.js:192:3)
[00:00:29]         │ proc [kibana]     at awaitIfPending (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/fleet-plugin/server/services/setup_utils.js:35:20)
[00:00:29]         │ proc [kibana]     at setupFleet (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/fleet-plugin/server/services/setup.js:73:14)
[00:00:29]         │ proc [kibana]     at fleetSetupHandler (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/fleet-plugin/server/routes/setup/handlers.js:59:23)
[00:00:29]         │ proc [kibana]     at defaultErrorHandlerWrapper (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/fleet-plugin/server/services/security/fleet_router.js:40:14)
[00:00:29]         │ proc [kibana]     at CoreVersionedRoute.handle (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/core-http-router-server-internal/src/versioned_router/core_versioned_route.js:104:22)
[00:00:29]         │ proc [kibana]     at Router.handle (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/core-http-router-server-internal/src/router.js:127:30)
[00:00:29]         │ proc [kibana]     at handler (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/core-http-router-server-internal/src/router.js:113:52)
[00:00:29]         │ proc [kibana]     at exports.Manager.execute (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
[00:00:29]         │ proc [kibana]     at Object.internals.handler (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@hapi/hapi/lib/handler.js:46:20)
[00:00:29]         │ proc [kibana]     at exports.execute (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@hapi/hapi/lib/handler.js:31:20)
[00:00:29]         │ proc [kibana]     at Request._lifecycle (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@hapi/hapi/lib/request.js:370:32)
[00:00:29]         │ proc [kibana]     at Request._execute (/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746602353840141449/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@hapi/hapi/lib/request.js:280:9) {"service":{"node":{"roles":["background_tasks","ui"]}}}
[00:00:29]         │ warn browser[SEVERE] http://localhost:5620/api/fleet/setup - Failed to load resource: the server responded with a status of 500 (Internal Server Error)

Hiding sync feature from UI and API on serverless

TattdCodeMonkey

The change to set enableSyncIntegrationsOnRemote: true is resulting in the index fleet-synced-integrations to be created which is breaking the search tests.

The broken tests are testing our onboarding flows which expect no non-system indices to exist. Which is why those tests are failing on this PR.

Why is fleet-synced-integrations not hidden or not a system index?

TattdCodeMonkey · 2025-05-07T15:29:41Z

x-pack/test/functional_search/tests/search_overview.ts

  const indexName = 'test-my-index';

-  describe('Search Overview page', function () {
+  describe.skip('Search Overview page', function () {


please do not skip these tests, your changes broke them

TattdCodeMonkey · 2025-05-07T15:30:01Z

x-pack/test/functional_solution_sidenav/tests/search_sidenav.ts

    describe('sidenav & breadcrumbs', () => {
-      it('renders the correct nav and navigate to links', async () => {
+      // https://github.com/elastic/kibana/issues/202974
+      it.skip('renders the correct nav and navigate to links', async () => {


your changes broke these tests

juliaElastic · 2025-05-08T06:44:07Z

The change to set enableSyncIntegrationsOnRemote: true is resulting in the index fleet-synced-integrations to be created which is breaking the search tests.

The broken tests are testing our onboarding flows which expect no non-system indices to exist. Which is why those tests are failing on this PR.

Why is fleet-synced-integrations not hidden or not a system index?

@TattdCodeMonkey Thanks for reviewing, I didn't realize the tests broke because of the new index.
We need a non-system index to be able to use CCR for a feature, more context here: #206237
Should I change the tests to allow this index?

TattdCodeMonkey · 2025-05-08T20:18:17Z

Should I change the tests to allow this index?

No, thats a larger discussion. We would need to update our code to ignore that index when we're checking for indices. Which is a bit of a hack. We'll have to discuss further what we should do in this case.

serenachou · 2025-05-08T20:41:04Z

@juliaElastic I think the more pressing concern is that Elasticsearch Index Management will show this index by default to customers.

As the ES product lead, this is internally relevant, and shouldn't be messed around with by customers. Also, having this not hidden means that our telemetry starts to track this as a potential signal that this ES cluster has ingestion (to signify better paid conversion etc) and that's not true because we have created this for the user on their behalf. By breaking the convention we introduce more problems. Where can we discuss this in more detail?

kpollich · 2025-05-09T13:45:58Z

I think we have two options to fix this, and we need to fix it on the Fleet side before we can enable this feature flag.

First, we should try to make this index hidden. I don't think there's anything preventing us from CCRing hidden indices, and I imagine this would fix the false signal issue for onboarding and user data flagging.

Second, we should fix this more robustly by only creating the index on-demand when a user takes some action. We initially wanted a more purposed onboarding flow for setting up the integration sync feature across clusters, but opted for a "follow these docs to enable CCR and set up the indices on each cluster" manual process. If we had a more purposed onboarding UX here we could defer creation of this index until the user actually opts into syncing integrations, which would be the "most correct" approach here.

We should do both of these things, if possible, where marking the index as hidden is the quick fix and providing a more robust UX that defers index creation until the user opts in is the more involved architectural correction.

@juliaElastic let me know what you think about this.

juliaElastic · 2025-05-27T08:02:32Z

I made the change to make fleet-synced-integrations index hidden, and it seems to work, however after adding the follower index on the hidden index, it is not showing up in the list, even though the follower index is created and the replication works.
It seems the API /api/cross_cluster_replication/follower_indices doesn't return hidden indices, which is a bad UX as users will be confused that the follower index is not showing up on the UI. The code is here.
I'm not sure if we can change this API to return hidden indices too.

Main cluster:

Remote cluster:

GET _all/_ccr/info

{
  "follower_indices": []
}

GET _all/_ccr/stats

{
  "indices": [
    {
      "index": "fleet-synced-integrations-ccr-local",
      "total_global_checkpoint_lag": 0,
      "shards": [
        {
          "remote_cluster": "local",
          "leader_index": "fleet-synced-integrations",
          "follower_index": "fleet-synced-integrations-ccr-local",
          "shard_id": 0,
          "leader_global_checkpoint": 10,
          "leader_max_seq_no": 10,
          "follower_global_checkpoint": 10,
          "follower_max_seq_no": 10,
          "last_requested_seq_no": 10,
          "outstanding_read_requests": 1,
          "outstanding_write_requests": 0,
          "write_buffer_operation_count": 0,
          "write_buffer_size_in_bytes": 0,
          "follower_mapping_version": 1,
          "follower_settings_version": 2,
          "follower_aliases_version": 1,
          "total_read_time_millis": 325843,
          "total_read_remote_exec_time_millis": 325831,
          "successful_read_requests": 7,
          "failed_read_requests": 0,
          "operations_read": 7,
          "bytes_read": 4921,
          "total_write_time_millis": 354,
          "successful_write_requests": 7,
          "failed_write_requests": 0,
          "operations_written": 7,
          "read_exceptions": [],
          "time_since_last_read_millis": 9330
        }
      ]
    }
  ]
}

criamico · 2025-05-27T08:55:51Z

So with the hidden index do we always have an empty response as below?

GET _all/_ccr/info

{
  "follower_indices": []
}

We rely on the ccr info api in getFollowerIndex to get the integrations doc, so I'm wondering if everything will work as expected.

…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --include-path /api/alerting/maintenance_window --update'

juliaElastic · 2025-06-03T09:22:22Z

x-pack/platform/plugins/shared/fleet/server/routes/remote_synced_integrations/index.ts

-              FLEET_API_PRIVILEGES.INTEGRATIONS.READ,
-            ],
-          },
+  router.versioned


removed the canEnableSyncIntegrations() check because it seems licenceService is not available when this code runs in kibana start

juliaElastic · 2025-06-03T10:06:18Z

I made the change to create the follower index only if the sync is enabled on a remote output. This fixes the failed tests in onboarding.
Also reverted the change to make the follower index hidden for now, because it doesn't work well with the CCR UI and it would be confusing to users.
This should probably be an enhancement on the ES API side to support returning hidden indices in the _ccr/info API.

cc @kpollich

criamico · 2025-06-03T12:50:24Z

x-pack/platform/plugins/shared/fleet/server/services/setup/fleet_synced_integrations.ts

 export const FLEET_SYNCED_INTEGRATIONS_INDEX_CONFIG = {
  settings: {
    auto_expand_replicas: '0-1',
+    // hidden: true,


should this be removed?

criamico · 2025-06-03T12:52:02Z

x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md

+  - Equivalent Dev Tools API request
+
+  ```
+  PUT /fleet-synced-integrations-ccr-local/_ccr/follow?wait_for_active_shards=1


I'm confused if wait_for_active_shards is is still needed, I thought it was decided to remove the hidden: true ?

it's an optional parameter, not related to hidden, copied it from the docs: https://www.elastic.co/guide/en/elasticsearch/reference/8.18/ccr-put-follow.html

criamico

LGTM 🚢

jillguyonnet

LGTM 🚀

…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --include-path /api/maintenance_window --update'

elasticmachine · 2025-06-04T09:40:03Z

💚 Build Succeeded

Buildkite Build
Commit: a08b8da

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.7MB	1.7MB	+100.0B

History

💔 Build #304693 failed 5b48a3c
💔 Build #304531 failed 8a55b0c
💛 Build #304247 was flaky b8c118f
💔 Build #304041 failed 6236b00
💔 Build #304022 failed 072b268

cc @juliaElastic

fixed the cause of failing onboarding tests

…20215) ## Summary Closes elastic#217490 - Enable feature flag `enableSyncIntegrationsOnRemote` - Added check to hide sync integrations feature in serverless - Moved creating the follower index from Fleet setup to output create/update API and async task (create if does not exist) - Follower index is not hidden for now, because if hidden, it's not showing up on the CCR UI --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

enable feature flag enableSyncIntegrationsOnRemote

0f1cb66

juliaElastic self-assigned this May 6, 2025

juliaElastic added the backport:skip This PR does not require backporting label May 6, 2025

juliaElastic requested a review from a team as a code owner May 6, 2025 09:31

juliaElastic added the release_note:feature Makes this part of the condensed release notes label May 6, 2025

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label May 6, 2025

juliaElastic added 3 commits May 6, 2025 11:31

Merge branch 'main' into enable-sync-ff

259fdda

Merge branch 'main' into enable-sync-ff

d6a7b55

Merge branch 'main' into enable-sync-ff

0165c77

nchaulet approved these changes May 6, 2025

View reviewed changes

juliaElastic added 2 commits May 6, 2025 16:41

Merge branch 'main' into enable-sync-ff

352ccf9

Merge branch 'main' into enable-sync-ff

101adcb

juliaElastic added 2 commits May 7, 2025 10:24

disable sync feature in serverless

76d7b59

skip failing tests

143dbed

juliaElastic requested a review from a team as a code owner May 7, 2025 11:01

juliaElastic added 2 commits May 7, 2025 14:17

Update search_overview.ts

167d302

Update search_overview.ts

83a0a88

TattdCodeMonkey previously requested changes May 7, 2025

View reviewed changes

juliaElastic added 2 commits May 8, 2025 09:55

revert change search_overview.ts

c10f12e

revert change search_sidenav.ts

f3213ba

juliaElastic and others added 2 commits May 27, 2025 00:01

Merge branch 'main' into enable-sync-ff

230310c

make sync index hidden

a6f2f49

juliaElastic and others added 10 commits June 2, 2025 11:35

fix checks

072b268

fix import

6236b00

fix integration tests

8b2ae57

fix test

1a5c79c

move create sync index creation to the task

b8c118f

move create index function

7c015a9

removed condition when registering route

1cf5213

[CI] Auto-commit changed files from 'make api-docs'

fcd997b

cleanup isServerless check

4f1ecce

juliaElastic commented Jun 3, 2025

View reviewed changes

create sync index on output update

b465259

juliaElastic requested a review from a team June 3, 2025 09:58

update readme

8a55b0c

criamico reviewed Jun 3, 2025

View reviewed changes

juliaElastic and others added 2 commits June 3, 2025 15:13

Merge branch 'main' into enable-sync-ff

ac26e8a

fix tests

2fcd440

criamico approved these changes Jun 3, 2025

View reviewed changes

jillguyonnet approved these changes Jun 3, 2025

View reviewed changes

kibanamachine added 2 commits June 3, 2025 13:37

[CI] Auto-commit changed files from 'make api-docs'

5b48a3c

juliaElastic mentioned this pull request Jun 3, 2025

API _ccr/info doesn't return hidden follower indices elastic/elasticsearch#128836

Open

fix condition

a08b8da

juliaElastic merged commit d65f9c5 into elastic:main Jun 4, 2025
10 checks passed

kibanamachine added the v9.1.0 label Jun 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] enable feature flag enableSyncIntegrationsOnRemote#220215

[Fleet] enable feature flag enableSyncIntegrationsOnRemote#220215
juliaElastic merged 35 commits intoelastic:mainfrom
juliaElastic:enable-sync-ff

juliaElastic commented May 6, 2025 •

edited

Loading

elasticmachine commented May 6, 2025

nchaulet left a comment

juliaElastic commented May 7, 2025 •

edited

Loading

TattdCodeMonkey left a comment

TattdCodeMonkey May 7, 2025

TattdCodeMonkey May 7, 2025

juliaElastic commented May 8, 2025

TattdCodeMonkey commented May 8, 2025 •

edited

Loading

serenachou commented May 8, 2025

kpollich commented May 9, 2025

juliaElastic commented May 27, 2025 •

edited

Loading

criamico commented May 27, 2025 •

edited

Loading

juliaElastic Jun 3, 2025

juliaElastic commented Jun 3, 2025

criamico Jun 3, 2025

criamico Jun 3, 2025

juliaElastic Jun 3, 2025

criamico left a comment

jillguyonnet left a comment

elasticmachine commented Jun 4, 2025

Uh oh!

Labels

9 participants

Conversation

juliaElastic commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

elasticmachine commented May 6, 2025

nchaulet left a comment

Choose a reason for hiding this comment

juliaElastic commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TattdCodeMonkey left a comment

Choose a reason for hiding this comment

TattdCodeMonkey May 7, 2025

Choose a reason for hiding this comment

TattdCodeMonkey May 7, 2025

Choose a reason for hiding this comment

juliaElastic commented May 8, 2025

TattdCodeMonkey commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

serenachou commented May 8, 2025

kpollich commented May 9, 2025

juliaElastic commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

criamico commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

juliaElastic Jun 3, 2025

Choose a reason for hiding this comment

juliaElastic commented Jun 3, 2025

criamico Jun 3, 2025

Choose a reason for hiding this comment

criamico Jun 3, 2025

Choose a reason for hiding this comment

juliaElastic Jun 3, 2025

Choose a reason for hiding this comment

criamico left a comment

Choose a reason for hiding this comment

jillguyonnet left a comment

Choose a reason for hiding this comment

elasticmachine commented Jun 4, 2025

💚 Build Succeeded

Metrics [docs]

Async chunks

History

Uh oh!

Labels

9 participants

juliaElastic commented May 6, 2025 •

edited

Loading

juliaElastic commented May 7, 2025 •

edited

Loading

TattdCodeMonkey commented May 8, 2025 •

edited

Loading

juliaElastic commented May 27, 2025 •

edited

Loading

criamico commented May 27, 2025 •

edited

Loading