[ti_opencti] OpenCTI integration by chrisberkhout · Pull Request #7385 · elastic/integrations

chrisberkhout · 2023-08-14T18:32:52Z

Proposed commit message

[ti_opencti] OpenCTI integration

A new integration for the OpenCTI threat intelligence platform:
https://www.filigran.io/en/solutions/products/opencti/

It queries a GraphQL API for indicator data using the CEL input,
populates ECS threat fields where possible and preserves additional
indicator-related data under an `opencti` prefix.

Background

OpenCTI integration development notes has some some background information about OpenCTI and the approach taken with this integration.

OpenCTI Mappings is a spreadsheet with field-level notes.

Status

What's working

This version is ready for final review and then release.

Since the demo timestamp mapping has been changed. The current mappings of the various timestamps are described in the README. The motivations for the change were to show incoming data at recent @timestamp values for a more intuitive user experience, and to build visualizations on logs-* rather than ad-hoc data views to avoid issues with persistent visualization errors and controls not working.

Potential future enhancements

Set up a transform to provide a view that shows the most recent non-expired data, as done elsewhere.
Support limiting the time range of the initial fetch.
Parse STIX rules for atomic indicators, so alert rules can work when no related observable is present, if requested.
Infosec have said that, in general, observables will be present. For the demo.opencti.io data, only about 0.17% of indicators have a STIX pattern but no observables.

How to test this PR locally

Build and install the integration as usual:

cd packages/ti_opencti
elastic-package stack up -d
elastic-package build -v
elastic-package install -v
elastic-package test pipeline -v
elastic-package test system -v

With a free account on the OpenCTI public demo instance you can get an API key (in the UI under Profile > API access) that will work as a bearer token as long as you're logged in. The OpenCTI demo instance has about 1.2 million indicators. The API token and base URL are the only setting required when adding a policy.

Making dashboard changes

If dashboards need to be updated, it may be necessary to downgrade to version 8.7.1 of the stack and export them from there, to avoid failing with the error "packages with dashboards exported since Kibana 8.8.0 may not be installed till 8.10.0, please export the dashboard/s from a different version". To make the integration work on 8.7.1, some further changes are necessary:

Changes necessary to run on 8.7.1

diff --git a/packages/ti_opencti/data_stream/indicator/agent/stream/cel.yml.hbs b/packages/ti_opencti/data_stream/indicator/agent/stream/cel.yml.hbs
index f4d975ed9..20067bf9a 100644
--- a/packages/ti_opencti/data_stream/indicator/agent/stream/cel.yml.hbs
+++ b/packages/ti_opencti/data_stream/indicator/agent/stream/cel.yml.hbs
@@ -43,15 +43,15 @@ program: |
         {}
     )
   }).with({
-    "Body": {
-      "query": state.query,
+    "Body": '''{
+      "query": ''' + state.query.encode_json() + ''',
       "variables": {
-        "after": has(state.cursor) && has(state.cursor.value) ? state.cursor.value : null,
-        "first": state.page_size,
+        "after": ''' + (has(state.cursor) && has(state.cursor.value) ? state.cursor.value.encode_json() : 'null') + ''',
+        "first": ''' + string(state.page_size) + ''',
         "orderBy": "modified",
-        "orderMode": "asc",
+        "orderMode": "asc"
       }
-    }.encode_json()
+    }'''
   }).do_request().as(resp,
     bytes(resp.Body).decode_json().as(body, state.with({
       "events": body.data.indicators.edges.map(e, e.node.with(
diff --git a/packages/ti_opencti/manifest.yml b/packages/ti_opencti/manifest.yml
index 02d8ca449..881eb2eff 100644
--- a/packages/ti_opencti/manifest.yml
+++ b/packages/ti_opencti/manifest.yml
@@ -11,7 +11,7 @@ categories:
   - threat_intel
 conditions:
   kibana:
-    version: "^8.9.0"
+    version: "^8.7.1"
 screenshots:
   - src: /img/screenshot1.png
     title: "Dashboard: OpenCTI Overview"

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Closes OpenCTI #6186

Screenshots

Please see the screenshots in the integration.

elasticmachine · 2023-08-14T18:38:40Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-12T14:52:14.782+0000
Duration: 115 min 43 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5027
Skipped	6
Total	5033

Steps errors

Expand to view the steps failures

`Test integration: aws`

Took 3 min 33 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

`Google Storage Download`

Took 0 min 0 sec . View more details here

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-08-23T12:17:44Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`464/464`)	💚
Files	96.126% (`794/826`)	👎 -3.874
Classes	96.126% (`794/826`)	👎 -3.874
Methods	92.455% (`7695/8323`)	👎 -7.545
Lines	88.365% (`175204/198274`)	👎 -7.158
Conditionals	100.0% (`0/0`)	💚

ebeahan

Great work! I added some nits as a scanned your progress so far.

For the list of remaining items under To Do, let's talk to @jamiehynds if there are items/features we can descope in this initial version. For example, maybe we don't need to support additional auth methods, likemTLS certificate and OAuth2, up front.

chrisberkhout · 2023-08-31T08:42:27Z

/test

chrisberkhout · 2023-09-29T10:00:17Z

/test

elasticmachine · 2023-10-02T09:15:21Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6 · 2023-10-03T00:00:42Z

@jsoriano I suspect the "run all the things" testing here is because of the change to links_table.yml. This is a similar situation to what I saw when I changed the pr template. Should we be doing a full test of all packages when this file changes? (I can see that we should do a check to make sure the change has not broken any generated docs).

jsoriano · 2023-10-03T15:11:14Z

Should we be doing a full test of all packages when this file changes? (I can see that we should do a check to make sure the change has not broken any generated docs).

Yes, it looks like a check would be enough, or we could try to make the pipeline smarter and detect what packages are using each link. Could you please open an issue to improve the situation? Though this file doesn't change frequently.

jsoriano

Change in CODEOWNERS and links table looks good to me.

efd6 · 2023-10-08T20:11:32Z

The test failure here is unrelated; due to sql_input. This is being tested because of the change in the links_table.yml file. This is #8077.

…ndicator types.

…t conflicts with the 'SEI Integration Dashboard styling guide' recommendation 'Only use filters on visualizations panels, not dashboards'.

This reverts commit 9a90cdaa084c15cf54a81c29e08a5c02954220ea.

…observable processing.

taylor-swanson

Just a couple of typos to fix, but otherwise LGTM

ebeahan · 2023-10-16T15:59:59Z

@chrisberkhout anything else you need here or is this ready to merge and release?

chrisberkhout · 2023-10-16T16:55:32Z

@ebeahan All feedback addressed and tests for this integration are passing. I think this is good to go.

It looks like I can't override and merge without tests passing, but if someone else can that would be good.

elasticmachine · 2023-10-16T20:04:51Z

Package ti_opencti - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=ti_opencti

colin-stubbs · 2023-11-01T08:02:11Z

a pleasant surprise to see this pop up, nice work @chrisberkhout !

@timestamp

* Initial commit of ti_opencti. * Fix cel program, add sys test, show vars. * Note about CEL encode_json() workaround. * Update graphql query. * Add some initial field config. * Older start to ingest pipeline. * Basic ingest pipeline. * A working pipeline. * Basic mappings done, for indicator and observables and each observable type. * Delete obsolete system test data. * Update CODEOWNERS. * Add responses for system test. * Add pipeline tests. * Fixes to make tests run without error. * Rename pipeline test files to have test- prefix. * Rename input.yml.hbs to cel.yml.hbs. * Tidy whitespace. * Move state section with GraphQL query to the end. * Switch to dynamic ECS mappings. * Override expected values for some ECS fields, so validation passes during system tests. * Remove not necessary and not allowed expected_values fields. * Match threat.indicator.confidence value to ECS expectations (Med->Medium). * Remove LICENSE.txt, to fall back to the default at the repo root. * Replace placeholder logo with OpenCTI logo. * Correct ecs.version field value. * Add a link to the GraphQL schema. * Remove commented-out fields from GraphQL query. * Update pipeline tests expected results for correct ecs.version. * For ECS, remove any subnet mask from IP values arriving in CIDR notation. * Add descriptions for opencti fields. * Update package manifest format version. * Fill out draft PR number in changelog. * Expanded GraphQL API info comment. * Update field types for opencti fields. * Move additional hash fields to ECS file since they're nested under the ECS threat fields. * Fix related.hash to be one array, not separated out by hash algorithm. * Move opencti.* field definitions to their own file. * Tolerate URLs that can't be parsed. e.g. '127.0.0.1:8080/path/'. * Extract file extensions from file names. * Don't extract a drive_letter or directory from a directory path. * Revert "Replace placeholder logo with OpenCTI logo." This reverts commit 7e27677d4b6ecb43601fe1ed0c7dec0b71baac06. * Replace placeholder logo with OpenCTI logo (this time as an SVG). * Remove config variable preserve_duplicate_custom_fields. * Update some fields. * Combine and test kill chain fields. * Merge opencti.observable.process.startup_info into one object, index as flattened. * Clean up createdBy after renaming the fields it contains. * Don't keep description if it's empty or null. * Clean up related.* lists properly. * Set threat.indicator.reference using _conf.url. * Clean up temporary variable usage: put them at the root with a _tmp prefix. * Tidy. * Handle preserve_original_event in the CEL expression. * Fix tags in pipeline tests. * Rename placeholder screenshot. * Tidy up manifest.yml files, options and option handling. * Remove the workout for CEL's encode_json() on nested data, since the fix is in 8.9.0. * Improve CEL program. * Fix formatting. * Fix mix up of handlebars vs CEL variables. * Bring back the CEL encode_json() workaround. * Remove the CEL encode_json() workaround. * Remove handlebars template logic from CEL expression. * Include generated sample event. * README. * Set the threat.indicator.url.full field. * Allow Kibana 8.9.0 (also 8.9.0 is the minimum beats for CEL encode_json() fix). * Add pipeline step to set ecs-indicator-detail tag. * Set value for threat.feed.dashboard_id. * Bring back expected_values overrides for threat.indicator.{name,type}, since they're now needed. * Add dashboards. * Add screenshots (wide versions). * Screenshots - narrow versions. * Screenshot of the "Add Integration" page on Fleet added. * New expected_values workaround. * README: ECS fields doc. * Map created to event.created and leave @timestamp to receive its default value. * Clean up timestamp documentation. * Update dashboards to use logs-* and bring back controls. * Improve URL for linking to OpenCTI documentation. * README: Additional info about data models and approach to mapping. * Populate threat.indicator.url.* fields for domain-name and hostname indicator types. * Mark as secret the vars for api_key, oauth2 and ssl. * Version bump. * Update package manifest.yml to format_version 3.0.0. * Ignore error 'expected filter in dashboard: no filter found', since it conflicts with the 'SEI Integration Dashboard styling guide' recommendation 'Only use filters on visualizations panels, not dashboards'. * Avoid failing pipeline benchmarks due to missing _conf.url field. * Revert "Mark as secret the vars for api_key, oauth2 and ssl." This reverts commit 9a90cdaa084c15cf54a81c29e08a5c02954220ea. * ECS 8.10 * Relocate redact config. * Use triple curly brackets to access field values in template snippets. * Single-quote string to avoid escaping. * Improved failure handler. * Don't keep empty hash fields. * Remove unused ad-hoc data views from dashboards. * Switch opencti field definitions to be nested rather than flattened. * Fix field type for hash group. * Do clean up observable fieldsets and observable itself at the end of observable processing. * Comment on which validation is excluded. * Fix typos.

chrisberkhout added the enhancement New feature or request label Aug 14, 2023

bhapas added the Team:Security-External Integrations label Aug 15, 2023

chrisberkhout added the New Integration Issue or pull request for creating a new integration package. label Aug 15, 2023

ebeahan requested a review from a team August 22, 2023 14:21

marc-gr self-assigned this Aug 22, 2023

efd6 reviewed Aug 22, 2023

View reviewed changes

efd6 reviewed Aug 23, 2023

View reviewed changes

Comment thread packages/ti_opencti/data_stream/indicator/agent/stream/cel.yml.hbs Outdated

ebeahan reviewed Aug 24, 2023

View reviewed changes

Comment thread packages/ti_opencti/LICENSE.txt Outdated

Comment thread packages/ti_opencti/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml Outdated

Comment thread packages/ti_opencti/img/sample-screenshot.png Outdated

Comment thread packages/ti_opencti/img/sample-logo.svg Outdated

bhapas reviewed Aug 28, 2023

View reviewed changes

ebeahan unassigned marc-gr Aug 30, 2023

chrisberkhout mentioned this pull request Sep 26, 2023

Tests fail due to ECS expected values elastic/elastic-package#1472

Closed

chrisberkhout marked this pull request as ready for review October 2, 2023 09:15

chrisberkhout requested a review from a team as a code owner October 2, 2023 09:15

efd6 reviewed Oct 2, 2023

View reviewed changes

jsoriano approved these changes Oct 3, 2023

View reviewed changes

efd6 mentioned this pull request Oct 3, 2023

ci: make testing more surgical when changes to links_table.yml are made #8077

Open

chrisberkhout requested a review from efd6 October 5, 2023 14:07

chrisberkhout changed the title ~~OpenCTI integration~~ Oct 5, 2023

efd6 reviewed Oct 8, 2023

View reviewed changes

Comment thread ...ges/ti_opencti/data_stream/indicator/_dev/test/pipeline/test-phone-number.json-expected.json Outdated

Comment thread packages/ti_opencti/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml Outdated

andrewkroh added the Integration:ti_opencti OpenCTI label Oct 10, 2023

chrisberkhout added 2 commits October 12, 2023 13:15

Initial commit of ti_opencti.

e9990c4

Fix cel program, add sys test, show vars.

8ca51ce

chrisberkhout added 20 commits October 12, 2023 13:15

Update dashboards to use logs-* and bring back controls.

b7949dc

Improve URL for linking to OpenCTI documentation.

1e86a5d

README: Additional info about data models and approach to mapping.

868c79e

Populate threat.indicator.url.* fields for domain-name and hostname i…

d72daab

…ndicator types.

Mark as secret the vars for api_key, oauth2 and ssl.

21e55ba

Version bump.

c99c773

Update package manifest.yml to format_version 3.0.0.

2085e86

Ignore error 'expected filter in dashboard: no filter found', since i…

dd11be4

…t conflicts with the 'SEI Integration Dashboard styling guide' recommendation 'Only use filters on visualizations panels, not dashboards'.

Avoid failing pipeline benchmarks due to missing _conf.url field.

6f6020b

Revert "Mark as secret the vars for api_key, oauth2 and ssl."

43983e1

This reverts commit 9a90cdaa084c15cf54a81c29e08a5c02954220ea.

ECS 8.10

b3dab9d

Relocate redact config.

04fd9a7

Use triple curly brackets to access field values in template snippets.

6b6e0f9

Single-quote string to avoid escaping.

4caa5a8

Improved failure handler.

daebeaf

Don't keep empty hash fields.

6ec056a

Remove unused ad-hoc data views from dashboards.

a8499b6

Switch opencti field definitions to be nested rather than flattened.

35b02f6

Fix field type for hash group.

1fcf229

Do clean up observable fieldsets and observable itself at the end of …

c7bd854

…observable processing.

taylor-swanson approved these changes Oct 12, 2023

View reviewed changes

Comment thread packages/ti_opencti/_dev/build/docs/README.md Outdated

Comment thread packages/ti_opencti/_dev/build/docs/README.md Outdated

Comment thread packages/ti_opencti/data_stream/indicator/fields/opencti.yml Outdated

chrisberkhout added 2 commits October 12, 2023 16:50

Comment on which validation is excluded.

ac3c516

Fix typos.

50c5039

ebeahan merged commit 96d2416 into elastic:main Oct 16, 2023

chrisberkhout deleted the ti_opencti branch October 17, 2023 09:11

Uh oh!

Conversation

chrisberkhout commented Aug 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Background

Status

What's working

Potential future enhancements

How to test this PR locally

Making dashboard changes

Checklist

Related issues

Screenshots

elasticmachine commented Aug 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Build stats

Test stats 🧪

Steps errors

Test integration: aws

Google Storage Download

🤖 GitHub comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

ebeahan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chrisberkhout commented Aug 31, 2023

chrisberkhout commented Sep 29, 2023

elasticmachine commented Oct 2, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented Oct 3, 2023

jsoriano commented Oct 3, 2023

jsoriano left a comment

Choose a reason for hiding this comment

efd6 commented Oct 8, 2023

Uh oh!

Uh oh!

taylor-swanson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ebeahan commented Oct 16, 2023

chrisberkhout commented Oct 16, 2023

elasticmachine commented Oct 16, 2023

colin-stubbs commented Nov 1, 2023

Labels

10 participants

chrisberkhout commented Aug 14, 2023 •

edited

Loading

elasticmachine commented Aug 14, 2023 •

edited

Loading

`Test integration: aws`

`Google Storage Download`

elasticmachine commented Aug 23, 2023 •

edited

Loading