ti_*: Add missing fields in transform destination indices (2)

[kcreddy]

Proposed commit message

Add missing fields for detection rules.

Some fields such as event.module were missing from the 
destination index. In few packages, the field mappings are getting 
overwritten when same prefix exists in multiple files. To prevent this, 
all fields with same prefix are moved into single file, ecs.yml. 
This is only a temporary fix until kibana.version is updated to 
>= 8.14.0, in which the root issue is fixed.

  - Add comments inside ecs.yml describing the issue.

  - Add `threat.feed.name` field only if its not added inside 
    ingest pipelines.

  - Some packages have missing fields `event.module` and `event.dataset` 
    which are added to both source and destination.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• Manually verify destination mappings using system tests.

Following mappings validates that missing fields such as event.module are now mapped correctly in destination indices:
misp-threatattr-dest-8.13-1.33.1.json
misp-threatattr-source-8.13-1.33.1.json
opencti-indicator-dest-8.13-2.1.1.json
opencti-indicator-source-8.13-2.1.1.json
recordedfuture-threat-dest-8.13-1.25.1.json
recordedfuture-threat-source-8.13-1.25.1.json
threatconnect-indicator-dest-8.13-1.0.1.json
threatconnect-indicator-source-8.13-1.0.1.json
threatq-threat-dest-8.13-1.27.1.json
threatq-threat-source-8.13-1.27.1.json

How to test this PR locally

diff <(cat ~/misp-threatattr-source-8.13-1.33.1.json ) <(cat ~/misp-threatattr-dest-8.13-1.33.1.json )
Similarly for other files to find mapping diff between source and dest files. The diff shouldn't mainly contain event.module field which is required for detection rules.

• Note that for packages threatconnect and opencti there is large diff between source and destination mappings because of import_mappings: true on source adding multiple dynamic templates.

Related issues

• Relates [threat intelligence] logs-ti_abusech_latest.dest_threatfox-1 missing event.module field -> Threat Intel IP Match rule not working #10032
• Relates [Fleet] Transform mappings lost in shallow merge kibana#175331

Screenshots

Before:

After:

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 96f00bb

History

• 💚 Build #12246 succeeded eea000c
• 💔 Build #12227 failed 23d53dc

cc @kcreddy

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elasticmachine]

Package ti_misp - 1.33.1 containing this change is available at https://epr.elastic.co/search?package=ti_misp

[elasticmachine]

Package ti_opencti - 2.1.1 containing this change is available at https://epr.elastic.co/search?package=ti_opencti

[elasticmachine]

Package ti_recordedfuture - 1.25.1 containing this change is available at https://epr.elastic.co/search?package=ti_recordedfuture

[elasticmachine]

Package ti_threatconnect - 1.0.1 containing this change is available at https://epr.elastic.co/search?package=ti_threatconnect

[elasticmachine]

Package ti_threatq - 1.27.1 containing this change is available at https://epr.elastic.co/search?package=ti_threatq