okta: allow user configuration of debug_data flattened use

[efd6]

Status: Blocked by elastic/kibana#183496 which will require v8.15.0.

Proposed commit message

See title.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes [okta.system] Utilize 'subobjects: false' for debugContext.debugData #9863

Screenshots

[efd6]

This change does not currently pass.

--- Test results for package: okta - START ---
FAILURE DETAILS:
okta/system test-okta-system-no-flattened-events.log:
[0] field "okta.debug_context.debug_data.behaviors.new_country" is undefined
[1] field "okta.debug_context.debug_data.behaviors.new_device" is undefined
[2] field "okta.debug_context.debug_data.behaviors.new_geo_location" is undefined
[3] field "okta.debug_context.debug_data.behaviors.new_ip" is undefined
[4] field "okta.debug_context.debug_data.behaviors.new_state" is undefined
[5] field "okta.debug_context.debug_data.behaviors.velocity" is undefined
[6] field "okta.debug_context.debug_data.behaviors.velocity_behavior" is undefined
[7] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_city" is undefined
[8] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_country" is undefined
[9] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_device" is undefined
[10] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_geo_location" is undefined
[11] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_ip" is undefined
[12] field "okta.debug_context.debug_data.log_only_security_data.behaviors.new_state" is undefined
[13] field "okta.debug_context.debug_data.log_only_security_data.behaviors.velocity" is undefined
[14] field "okta.debug_context.debug_data.log_only_security_data.risk.level" is undefined
[15] field "okta.debug_context.debug_data.log_only_security_data.risk.reasons" is undefined
[16] field "okta.debug_context.debug_data.original_principal.alternate_id" is undefined
[17] field "okta.debug_context.debug_data.original_principal.display_name" is undefined
[18] field "okta.debug_context.debug_data.original_principal.id" is undefined
[19] field "okta.debug_context.debug_data.original_principal.type" is undefined
[20] field "okta.debug_context.debug_data.prompting_policy_types" is undefined
[21] field "okta.debug_context.debug_data.risk.level" is undefined
[22] field "okta.debug_context.debug_data.risk.reasons" is undefined
[23] field "okta.debug_context.debug_data.risk_object" is undefined



╭─────────┬─────────────┬───────────┬───────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                 │ RESULT                                                                      │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ okta    │ system      │ pipeline  │ test-okta-system-events.log               │ PASS                                                                        │  17.328337ms │
│ okta    │ system      │ pipeline  │ test-okta-system-no-flattened-events.log  │ FAIL: test case failed: one or more problems with fields found in documents │  34.252526ms │
│ okta    │ system      │ pipeline  │ test-okta-system-yes-flattened-events.log │ PASS                                                                        │  28.417599ms │
│ okta    │ system      │ pipeline  │ (ingest pipeline warnings)                │ PASS                                                                        │ 395.535137ms │
╰─────────┴─────────────┴───────────┴───────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────┴──────────────╯

So add one of these to fix this…

diff --git a/packages/okta/data_stream/system/fields/fields.yml b/packages/okta/data_stream/system/fields/fields.yml
index b0f51b1b0..c314f671c 100644
--- a/packages/okta/data_stream/system/fields/fields.yml
+++ b/packages/okta/data_stream/system/fields/fields.yml
@@ -273,3 +273,5 @@
   fields:
     - name: ip_chain
       type: flattened
+- name: okta.debug_context.debug_data.behaviors.new_city
+  type: keyword
\ No newline at end of file
diff --git a/packages/okta/docs/README.md b/packages/okta/docs/README.md
index 00f10e7ea..9762a1b64 100644
--- a/packages/okta/docs/README.md
+++ b/packages/okta/docs/README.md
@@ -304,6 +304,7 @@ An example event for `system` looks as following:
 | okta.client.zone | The zone information of the client. | keyword |
 | okta.debug_context.debug_data |  | object |
 | okta.debug_context.debug_data.authn_request_id | The authorization request ID. | keyword |
+| okta.debug_context.debug_data.behaviors.new_city |  | keyword |
 | okta.debug_context.debug_data.device_fingerprint | The fingerprint of the device. | keyword |
 | okta.debug_context.debug_data.dt_hash | The device token hash | keyword |
 | okta.debug_context.debug_data.factor | The factor used for authentication. | keyword |

This however causes a mapping failure:

Error: error running package system tests: could not complete test run: can't install the package: there was an apply error: installation failed: can't install the package: could not zip-install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"mapper_parsing_exception\n\tCaused by:\n\t\tmapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects\n\tRoot causes:\n\t\tmapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects"}

mapper_parsing_exception
	Caused by:
		mapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects
	Root causes:
		mapper_parsing_exception: Tried to add subobject [behaviors] to object [debug_data] which does not support subobjects

A minimal version of this is here:

- name: grandparent
  type: object
  object_type: keyword
  object_type_mapping_type: "*"
  subobjects: false
- name: grandparent.parent.child
  type: keyword

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[andrewkroh]

I was testing the upgrade process going from 2.9.0 to 2.10.0 (751ce10) where I already had some data indexed, and the upgrade (using 8.13.4) fails with

mapper_exception: the [subobjects] parameter can't be updated for the object mapping [okta.debug_context.debug_data]

@zmoog @ruflin Is this the expected behavior? Can we adopt subobjects: false for an existing field?

The subobjects documentation says

The subobjects setting for existing fields and the top-level mapping definition cannot be updated.

but shouldn't Fleet be doing a rollover?

[andrewkroh]

remove_flattened_debug is declared in two places. I think one of them needs to be removed.

Relates: elastic/package-spec#421

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[zmoog]

Is this the expected behavior? Can we adopt subobjects: false for an existing field?

The subobjects documentation says

The subobjects setting for existing fields and the top-level mapping definition cannot be updated.

but shouldn't Fleet be doing a rollover?

Yes, Fleet should do a rollover.

@flash1293 mentioned this issue during our sync, a couple of days ago. We are looking into this.

[flash1293]

Sorry for this hurdle, the fleet team is already working on it here: elastic/kibana#183496

[andrewkroh]

@flash1293 Now that 8.14 is out, could we avoid the problem of elastic/kibana#183496 if we switched this data stream over to using data subobjects: false at the data stream level instead of doing it at the per-field level? Not sure if that changes anything, but wanted to see if it would unblock this.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[flash1293]

@andrewkroh I don't think this changes anything - in fact the custom integration case is also using subobjects:false on the datastream level.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[andrewkroh]

8.15.0 is out now so this should be unblocked by the earlier bug elastic/kibana#183496.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 9fb6beb

History

• 💚 Build #12738 succeeded c78ad0144e3a2f4bc2786c3afea7bbeddf1af5c6
• 💚 Build #11710 succeeded 4604582c7c24f1683a286b1e081ff013ad4bf081
• 💚 Build #11621 succeeded 751ce10474faffe859dc0e2ec5dadc1a819ed266
• 💔 Build #11620 failed 720cfe86c15ec656bb11f340ba57980a110cdea6
• 💔 Build #11613 failed 7c922b5e2e664a188664638fd481d26742d474d8
• 💚 Build #11454 succeeded 283d7b38c9b73d87d480ade7452c84db38281dea

cc @efd6

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
80.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Package okta - 2.12.0 containing this change is available at https://epr.elastic.co/search?package=okta