[Enhancement] [integration] Add EclecticIQ Advanced CSV Outgoing feed integration by petarc-eiq · Pull Request #8465 · elastic/integrations

petarc-eiq · 2023-11-10T14:28:36Z

Proposed commit message

Add Eclectic IQ integration with Advanced CSV Observables Outgoing Feed data stream.

The integration's data stream downloads outgoing feed content blocks data in CSV format, parses the CSV data and submits it to the ingestion pipeline.

Ingestion pipeline takes the input data and populates appropriate ECS fields.

The integration has a transform that creates "latest" index and populates it with records from data stream index that doesn't have "deleted_at" date, which is populated only for input records that are deleted from source dataset. This way, result index has only the latest data and provides a retention policy.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Review documentation

How to test this PR locally

Prerequiseites

Advanced CSV Observables Outgoing Feed with populated dataset(s).
EIQ IC API key.

Steps:

Install the integration.
Set up the integration (EIQ IC URL, API token, outgoing feed ID).
Check "logs-ti_eclecticiq_latest.observables-*" index for data.

cla-checker-service · 2023-11-10T14:28:40Z

💚 CLA has been signed

elasticmachine · 2023-11-13T13:46:48Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-12-14T11:52:20.109+0000
Duration: 15 min 28 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5
Skipped	0
Total	5

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-11-14T11:54:28Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

kcreddy · 2023-12-01T07:48:00Z

/test

kcreddy · 2023-12-04T16:05:30Z

/test

kcreddy · 2023-12-05T14:11:42Z

@petarc-eiq Looks like the CI is failing on older README file. Can you run elastic-package build && elastic-package format && elastic-package lint && elastic-package check && elastic-package build to rebuild and update the README file.

- Removed reference to non-existent log reference section - Added one more way to filter TI results to show only EclecticIQ data

zed-eiq · 2023-12-05T14:41:55Z

@\petarc-eiq Looks like the CI is failing on older README file. Can you run elastic-package build && elastic-package format && elastic-package lint && elastic-package check && elastic-package build to rebuild and update the README file.

done!

kcreddy · 2023-12-05T15:20:41Z

/test

elasticmachine · 2023-12-05T15:42:09Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`1/1`)	💚 3.888
Classes	100.0% (`1/1`)	💚 3.888
Methods	87.5% (`14/16`)	👎 -4.806
Lines	81.407% (`324/398`)	👎 -7.337
Conditionals	100.0% (`0/0`)	💚

kcreddy · 2023-12-05T15:54:54Z

@petarc-eiq @zed-eiq The CI is now failing on pipeline tests. Can you please check if they are successful on your end and commit updated *-expected.json file.

petarc-eiq · 2023-12-06T16:19:18Z

/test

kcreddy · 2023-12-06T20:52:30Z

/test

kcreddy · 2023-12-07T15:44:21Z

/test

kcreddy · 2023-12-12T13:07:44Z

/test

kcreddy · 2023-12-13T07:01:03Z

/test

kcreddy · 2023-12-13T07:58:33Z

@petarc-eiq Looks like the system tests are failing now. Can you check on your local if they are successfully running?

jamiehynds · 2023-12-13T08:58:31Z

@@ -0,0 +1,41 @@
+format_version: 2.8.0
+name: ti_eclecticiq
+title: EclecticIQ Outgoing Feeds Integration


@petarc-eiq @kcreddy as discussed, can we shorten the title to EclecticIQ and also reflect those changes in data stream names, documentation, etc too.

jamiehynds · 2023-12-13T08:59:37Z

+version: 1.0.0
+source:
+  license: Elastic-2.0
+description: Download data from EclecticIQ IC outgoing feeds


@petarc-eiq @kcreddy can we adjust this description to ensure alignment with other integrations. Ingest threat intelligence indicators from EclecticIQ with Elastic Agent

kcreddy

@petarc-eiq Based on the changes Jamie suggested to the integration's title and description, I added some changes you need to make.

kcreddy · 2023-12-13T10:09:37Z

@@ -0,0 +1,143 @@
+# EclecticIQ Outgoing Feeds Integration


Suggested change

# EclecticIQ Outgoing Feeds Integration

# EclecticIQ Integration

kcreddy · 2023-12-13T10:09:56Z

@@ -0,0 +1,143 @@
+# EclecticIQ Outgoing Feeds Integration
+
+The EclecticIQ Outgoing Feeds integration


Suggested change

The EclecticIQ Outgoing Feeds integration

The EclecticIQ integration

kcreddy · 2023-12-13T10:18:08Z

+## Data streams
+
+The EclecticIQ Outgoing Feeds integration
+collects one type of data streams: logs.
+
+**Logs** collected from this integration
+are collections of threat intelligence observables
+ingested from the connected EclecticIQ Intelligence Center outgoing feed.


Suggested change

## Data streams

The EclecticIQ Outgoing Feeds integration

collects one type of data streams: logs.

**Logs** collected from this integration

are collections of threat intelligence observables

ingested from the connected EclecticIQ Intelligence Center outgoing feed.

## Data streams

The EclecticIQ integration

collects one type of data: logs.

**Logs** collected from this integration

are collections of threat intelligence observables

ingested from the connected EclecticIQ Intelligence Center outgoing feed. These are available in the datastream named `threat`.

kcreddy · 2023-12-13T10:18:48Z

+### Update strategies
+
+You must set the **same** _Update strategy_ for
+both the EclecticIQ Outgoing Feeds Integration instance


Suggested change

both the EclecticIQ Outgoing Feeds Integration instance

both the EclecticIQ Integration instance

kcreddy · 2023-12-13T10:21:38Z

@@ -0,0 +1,43 @@
+title: "Poll Outgoing feed"


Suggested change

title: "Poll Outgoing feed"

title: EclecticIQ Threat

(i didn't realise that github has a "commit suggestion" feature 😅 )

I'm leaving this as "Poll Outgoing feed" if that's okay! This refers to functionality on EIQ IC, and makes it clear that the outgoing feed feature is what's being used here (there are other ways to get data out of EIQ IC)

kcreddy · 2023-12-13T11:01:27Z

+- name: event.dataset
+  type: constant_keyword
+  description: Event dataset
+  value: ti_eclecticiq.outgoing_feed


Suggested change

value: ti_eclecticiq.outgoing_feed

value: ti_eclecticiq.threat

kcreddy · 2023-12-13T11:01:36Z

+- name: "@timestamp"
+  type: date
+  description: Event timestamp.
+- name: eclecticiq.outgoing_feed


Suggested change

- name: eclecticiq.outgoing_feed

- name: eclecticiq.threat

kcreddy · 2023-12-13T11:01:44Z

@@ -0,0 +1,31 @@
+source:
+  index:
+    - "logs-ti_eclecticiq.outgoing_feed-*"


Suggested change

- "logs-ti_eclecticiq.outgoing_feed-*"

- "logs-ti_eclecticiq.threat-*"

kcreddy · 2023-12-13T11:03:39Z

+latest:
+  unique_key:
+    - event.dataset
+    - eclecticiq.outgoing_feed.observable_id


Suggested change

- eclecticiq.outgoing_feed.observable_id

- eclecticiq.threat.observable_id

kcreddy · 2023-12-13T11:03:45Z

+    delay: 120s
+retention_policy:
+  time:
+    field: eclecticiq.outgoing_feed.deleted_at


Suggested change

field: eclecticiq.outgoing_feed.deleted_at

field: eclecticiq.threat.deleted_at

zed-eiq · 2023-12-13T11:41:24Z

we're pending one more README change to include the list of IoCs we support/don't support in this version 🙏

kcreddy · 2023-12-13T12:17:31Z

/test

kcreddy · 2023-12-14T10:28:20Z

/test

elasticmachine · 2023-12-14T10:43:03Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

kcreddy

@petarc-eiq @zed-eiq Looks like the CI is passing now 🚀
PR LGTM 👍🏼
Is there anything else you would like to add to this PR or can I proceed merging it?

petarc-eiq · 2023-12-14T11:09:30Z

@petarc-eiq @zed-eiq Looks like the CI is passing now 🚀 PR LGTM 👍🏼 Is there anything else you would like to add to this PR or can I proceed merging it?

Hello @kcreddy we're doing a round of smoke testing with the latest updates, should be done soon. When we get verification, we are good to go! We will send confirmation as soon as possible. Thanks!

petarc-eiq · 2023-12-14T15:23:12Z

@zed-eiq Are we done with documentation? Smoke tests went as expected, so if @zed-eiq confirms, we are good to go with the merge 🙂 @kcreddy

zed-eiq · 2023-12-15T09:20:27Z

i'm done with readme updates! we're good to go from our end 🙏

elasticmachine · 2023-12-18T08:42:42Z

Package ti_eclecticiq - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=ti_eclecticiq

jamiehynds requested a review from a team November 14, 2023 11:54

jamiehynds added the Team:Security-External Integrations label Nov 14, 2023

jamiehynds added the New Integration Issue or pull request for creating a new integration package. label Nov 14, 2023

Add EclecticIQ Advanced CSV Outgoing feed integration

704bb5e

kcreddy reviewed Dec 1, 2023

View reviewed changes

apply MR comments

2436198

zed-eiq added 3 commits December 5, 2023 15:30

Fixes for README

0315700

- Removed reference to non-existent log reference section - Added one more way to filter TI results to show only EclecticIQ data

Make package manifest version match changelog version

7f64aaa

Format and lint

39cb592

kcreddy reviewed Dec 5, 2023

View reviewed changes

Comment thread packages/ti_eclecticiq/elasticsearch/transform/latest_ioc/transform.yml Outdated

Comment thread packages/ti_eclecticiq/manifest.yml

Comment thread packages/ti_eclecticiq/_dev/build/docs/README.md

fix failing pipeline tests

47397df

uppercase mac addresses

254bbb2

kcreddy reviewed Dec 11, 2023

View reviewed changes

Comment thread packages/ti_eclecticiq/data_stream/outgoing_feed/fields/ecs.yml Outdated

petarc-eiq added 2 commits December 12, 2023 13:09

replace example domain with "example.com"

a9dd1e7

redefine threat.indicator.name field

8940328

fix update_strategy in system test

b4946c0

jamiehynds reviewed Dec 13, 2023

View reviewed changes

Update name and description of ti_eclecticiq in manifest.yml

846fd0d

kcreddy reviewed Dec 13, 2023

View reviewed changes

zed-eiq added 3 commits December 13, 2023 12:28

Update ti_eclecticiq README

6a98231

Update ti_eclecticiq built README

f72306e

Update ti_eclecticiq field names

579a308

petarc-eiq added 3 commits December 13, 2023 13:26

fix system test config

425c0c9

rename "outgoing_feed" to "threat" and fix system test

f18184c

update readme file

a387b10

kcreddy approved these changes Dec 14, 2023

View reviewed changes

zed-eiq added 3 commits December 14, 2023 12:48

Update ti_eclectiiciq README with supported EclecticIQ observables

383242a

Update ti_eclecticiq README template with correct stream name

cff5dae

Rebuild ti_eclecticiq README

f225db0

kcreddy merged commit 24da00d into elastic:main Dec 18, 2023

petarc-eiq deleted the integration-eclecticiq branch December 18, 2023 15:25

andrewkroh added the Integration:ti_eclecticiq EclecticIQ (Partner supported) label Jul 22, 2024

	# EclecticIQ Outgoing Feeds Integration
	# EclecticIQ Integration

		@@ -0,0 +1,143 @@
		# EclecticIQ Outgoing Feeds Integration

		The EclecticIQ Outgoing Feeds integration

	The EclecticIQ Outgoing Feeds integration
	The EclecticIQ integration

	both the EclecticIQ Outgoing Feeds Integration instance
	both the EclecticIQ Integration instance

	value: ti_eclecticiq.outgoing_feed
	value: ti_eclecticiq.threat

	- "logs-ti_eclecticiq.outgoing_feed-*"
	- "logs-ti_eclecticiq.threat-*"

	- eclecticiq.outgoing_feed.observable_id
	- eclecticiq.threat.observable_id

	field: eclecticiq.outgoing_feed.deleted_at
	field: eclecticiq.threat.deleted_at

Uh oh!

Conversation

petarc-eiq commented Nov 10, 2023

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

cla-checker-service Bot commented Nov 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

elasticmachine commented Nov 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Nov 14, 2023

kcreddy commented Dec 1, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy commented Dec 4, 2023

kcreddy commented Dec 5, 2023

zed-eiq commented Dec 5, 2023

kcreddy commented Dec 5, 2023

elasticmachine commented Dec 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

Uh oh!

Uh oh!

kcreddy commented Dec 5, 2023

petarc-eiq commented Dec 6, 2023

kcreddy commented Dec 6, 2023

kcreddy commented Dec 7, 2023

Uh oh!

kcreddy commented Dec 12, 2023

kcreddy commented Dec 13, 2023

kcreddy commented Dec 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kcreddy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zed-eiq commented Dec 13, 2023

kcreddy commented Dec 13, 2023

kcreddy commented Dec 14, 2023

elasticmachine commented Dec 14, 2023

🚀 Benchmarks report

kcreddy left a comment

Choose a reason for hiding this comment

petarc-eiq commented Dec 14, 2023

petarc-eiq commented Dec 14, 2023

zed-eiq commented Dec 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

elasticmachine commented Dec 18, 2023

Labels

6 participants

cla-checker-service Bot commented Nov 10, 2023 •

edited

Loading

elasticmachine commented Nov 13, 2023 •

edited

Loading

elasticmachine commented Dec 5, 2023 •

edited

Loading

zed-eiq commented Dec 15, 2023 •

edited

Loading