ti_abusech: Adjust field mappings for transform

[kcreddy]

Proposed commit message

Adjust field mappings for transform.

Some fields such as event.module were missing from the destination index.
The field mappings are getting overwritten when same prefix exists in 
multiple files. To prevent this, all fields with same prefix are moved into 
single file, in this case ecs.yml. This is only a temporary fix until 
kibana.version is updated to >= 8.14.0, in which the root issue is fixed.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• Verify the destination mappings in all AbuseCH datastreams contains the missing fields event.module, event.dataset, threat.feed.name, and threat.feed.dashboard_id

Related issues

• Relates [threat intelligence] logs-ti_abusech_latest.dest_threatfox-1 missing event.module field -> Threat Intel IP Match rule not working #10032
• Relates [Fleet] Fix loading fields for transform destination index template  kibana#177608

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: e1e7a64

History

• 💚 Build #12173 succeeded 0919d85218a2473a2ebbfa87e9874b37934b2ecb

cc @kcreddy

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[andrewkroh]

Can you do a manual test where you export the mappings using 8.14 (where this problem is fixed) and compare with the mappings generated from this PR in 8.13. I want to double check that there are no other unexpected differences.

Otherwise LGTM.

[kcreddy]

Can you do a manual test where you export the mappings using 8.14 (where this problem is fixed) and compare with the mappings generated from this PR in 8.13. I want to double check that there are no other unexpected differences

Attached are the mappings I tested for AbuseCH.

• MalwareBazaar:
  abusech-malwarebazaar-dest-8.13-2.0.1.json
  abusech-malwarebazaar-source-8.13-2.0.1.json
  abusech-malwarebazaar-dest-8.14-2.0.0.json
  abusech-malwarebazaar-source-8.14-2.0.0.json
• ThreatFox:
  abusech-threatfox-dest-8.13-2.0.1.json
  abusech-threatfox-source-8.13-2.0.1.json
  abusech-threatfox-dest-8.14-2.0.0.json
  abusech-threatfox-source-8.14-2.0.0.json
• URL:
  abusech-url-dest-8.13-2.0.1.json
  abusech-url-source-8.13-2.0.1.json
  abusech-url-dest-8.14-2.0.0.json
  abusech-url-source-8.14-2.0.0.json
• Malware:
  abusech-malware-dest-8.13-2.0.1.json
  abusech-malware-source-8.13-2.0.1.json
  abusech-malware-dest-8.14-2.0.0.json
  abusech-malware-source-8.14-2.0.0.json

I verified source vs dest mappings for same version (8.13 and 8.14). I also verified dest 8.13 vs dest 8.14 mappings. All of them correctly match, except for Malware datastream.
On 8.14, the destination index of malware is incorrectly adding extra fields with malwarebazaar.* and threat.* that are actually part of malwarebazaar datastream. I am not sure why that is happening.

All of destination mappings are adding the missing fields event.module, event.dataset, threat.feed.name, and threat.feed.dashboard_id

[kcreddy]

Added few more ti_* packages to this PR: #10062

[andrewkroh]

On 8.14, the destination index of malware is incorrectly adding extra fields with malwarebazaar.*

That reproduces for me. I see the malwarebazaar fields in the logs-ti_abusech.latest_malware-template@package component template. There are no malwarebazaar fields in packages/ti_abusech/elasticsearch/transform/latest_malware/fields/*. This seems like it would have to be a bug in Fleet because it created the component template. Can you please open a Kibana bug for this.

The extra mappings won't break anything in this use-case, will they?

[kcreddy]

The extra mappings won't break anything in this use-case, will they?

Yeah, they shouldn't be a problem.

Can you please open a Kibana bug for this.

Created: elastic/kibana#184759

[elasticmachine]

Package ti_abusech - 2.0.1 containing this change is available at https://epr.elastic.co/search?package=ti_abusech

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)