Skip to content

[ti_*] Fix labels.is_ioc_transform_source values - #11231

Merged
chrisberkhout merged 5 commits into
elastic:mainfrom
chrisberkhout:fix-transform-source-label
Sep 26, 2024
Merged

[ti_*] Fix labels.is_ioc_transform_source values#11231
chrisberkhout merged 5 commits into
elastic:mainfrom
chrisberkhout:fix-transform-source-label

Conversation

@chrisberkhout

@chrisberkhout chrisberkhout commented Sep 24, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

[ti_*] Fix `labels.is_ioc_transform_source` values

The distinction between source and destination records was lost when
#11008 added a field definition for `labels.is_ioc_transform_source`
to the destination indices with a value of `"true"`.

Here it is restored by:
- Moving `labels.is_ioc_transform_source` field definitions into
  separate files.
- Setting the value to `"false"` in the transform destinations.
- Incrementing transform version numbers so new destination indices
  will be built with correct values.
- Adjusting dashboards to filter for not `"true"` rather than not
  exists.

Security rules already filter for not `"true"`.

Extra details

The relevant security rules can be seen here.

Dashboards using exists queries could be found with the following, but have all been removed:

ag --json -B5 -A5 labels.is_ioc_transform_source | grep -B5 -A5 exists

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues
@chrisberkhout chrisberkhout added Integration:ti_misp MISP Integration:ti_cybersixgill Cybersixgill Integration:ti_abusech abuse.ch Integration:ti_otx AlienVault OTX Integration:ti_recordedfuture Recorded Future Integration:ti_anomali Anomali ThreatStream bugfix Pull request that fixes a bug issue Integration:ti_maltiverse Maltiverse (Partner supported) Integration:ti_threatq ThreatQuotient (Partner supported) Integration:ti_cif3 Collective Intelligence Framework v3 (Community supported) Integration:ti_rapid7_threat_command Rapid7 Threat Command (Partner supported) Integration:ti_opencti OpenCTI Integration:ti_mandiant_advantage Mandiant Advantage (Partner supported) Integration:ti_crowdstrike CrowdStrike Falcon Intelligence Integration:ti_eclecticiq EclecticIQ (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Integration:ti_threatconnect ThreatConnect Integration:ti_eset ESET Threat Intelligence (Partner supported) labels Sep 24, 2024
@chrisberkhout chrisberkhout self-assigned this Sep 24, 2024
@chrisberkhout
chrisberkhout requested a review from a team as a code owner September 24, 2024 10:09
@elasticmachine

elasticmachine commented Sep 24, 2024

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
@andrewkroh andrewkroh added dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Integration:ti_custom Custom Threat Intelligence and removed Integration:ti_mandiant_advantage Mandiant Advantage (Partner supported) labels Sep 24, 2024
kcreddy
kcreddy reviewed Sep 24, 2024
View reviewed changes
Comment thread packages/ti_anomali/elasticsearch/transform/latest_ioc/transform.yml Outdated
Comment thread packages/ti_cif3/data_stream/feed/fields/is-ioc-transform-source-true.yml Outdated
Comment thread packages/ti_custom/elasticsearch/transform/latest_ioc/transform.yml Outdated
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 24, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
@chrisberkhout

chrisberkhout commented Sep 25, 2024

Copy link
Copy Markdown
Contributor Author

/test
@chrisberkhout
chrisberkhout enabled auto-merge (squash) September 26, 2024 09:36
@chrisberkhout
chrisberkhout merged commit f8d7f24 into elastic:main Sep 26, 2024
@elasticmachine

elasticmachine commented Sep 26, 2024

Copy link
Copy Markdown

💚 Build Succeeded

History

  • 💔 Build #16469 failed 76b5b613ec65eb5ea344d2135aa69255b22ed025
  • 💔 Build #16431 failed 2fcc94655164759ed9c3ed358bf589959e54054d
  • 💔 Build #16415 failed 2fcc94655164759ed9c3ed358bf589959e54054d
  • 💔 Build #16401 failed 2fcc94655164759ed9c3ed358bf589959e54054d
  • 💔 Build #16363 failed 86df85afbd2dc60ebedefd0866072b2ad7e225c8
  • 💔 Build #16356 failed 6f2347e21d1d254876dc18ff621e34b5f6b028a0

cc @chrisberkhout
@elastic-sonarqube

elastic-sonarqube Bot commented Sep 26, 2024

Copy link
Copy Markdown

Quality Gate failed Quality Gate failed

Failed conditions
52.5% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube
@chrisberkhout
chrisberkhout deleted the fix-transform-source-label branch September 26, 2024 10:22
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_abusech - 2.3.3 containing this change is available at https://epr.elastic.co/search?package=ti_abusech
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_anomali - 1.22.3 containing this change is available at https://epr.elastic.co/search?package=ti_anomali
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_cif3 - 1.14.3 containing this change is available at https://epr.elastic.co/search?package=ti_cif3
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_crowdstrike - 1.1.7 containing this change is available at https://epr.elastic.co/search?package=ti_crowdstrike
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_custom - 0.1.1 containing this change is available at https://epr.elastic.co/search?package=ti_custom
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_cybersixgill - 1.30.3 containing this change is available at https://epr.elastic.co/search?package=ti_cybersixgill
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_eclecticiq - 1.2.3 containing this change is available at https://epr.elastic.co/search?package=ti_eclecticiq
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_eset - 1.2.4 containing this change is available at https://epr.elastic.co/search?package=ti_eset
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_maltiverse - 1.2.3 containing this change is available at https://epr.elastic.co/search?package=ti_maltiverse
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_misp - 1.35.3 containing this change is available at https://epr.elastic.co/search?package=ti_misp
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_opencti - 2.3.4 containing this change is available at https://epr.elastic.co/search?package=ti_opencti
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_otx - 1.25.3 containing this change is available at https://epr.elastic.co/search?package=ti_otx
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_rapid7_threat_command - 2.0.3 containing this change is available at https://epr.elastic.co/search?package=ti_rapid7_threat_command
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_recordedfuture - 1.26.3 containing this change is available at https://epr.elastic.co/search?package=ti_recordedfuture
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_threatconnect - 1.2.3 containing this change is available at https://epr.elastic.co/search?package=ti_threatconnect
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Sep 26, 2024

Copy link
Copy Markdown
Contributor

Package ti_threatq - 1.28.3 containing this change is available at https://epr.elastic.co/search?package=ti_threatq
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@chrisberkhout
[ti_*] Fix labels.is_ioc_transform_source values (elastic#11231)
62f8d3e 
The distinction between source and destination records was lost when
elastic#11008 added a field definition for `labels.is_ioc_transform_source`
to the destination indices with a value of `"true"`.

Here it is restored by:
- Moving `labels.is_ioc_transform_source` field definitions into
  separate files.
- Setting the value to `"false"` in the transform destinations.
- Incrementing transform version numbers so new destination indices
  will be built with correct values.
- Adjusting dashboards to filter for not `"true"` rather than not
  exists.

Security rules already filter for not `"true"`.
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@chrisberkhout
[ti_*] Fix labels.is_ioc_transform_source values (elastic#11231)
9b83747 
The distinction between source and destination records was lost when
elastic#11008 added a field definition for `labels.is_ioc_transform_source`
to the destination indices with a value of `"true"`.

Here it is restored by:
- Moving `labels.is_ioc_transform_source` field definitions into
  separate files.
- Setting the value to `"false"` in the transform destinations.
- Incrementing transform version numbers so new destination indices
  will be built with correct values.
- Adjusting dashboards to filter for not `"true"` rather than not
  exists.

Security rules already filter for not `"true"`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Integration:ti_abusech abuse.ch Integration:ti_anomali Anomali ThreatStream Integration:ti_cif3 Collective Intelligence Framework v3 (Community supported) Integration:ti_crowdstrike CrowdStrike Falcon Intelligence Integration:ti_custom Custom Threat Intelligence Integration:ti_cybersixgill Cybersixgill Integration:ti_eclecticiq EclecticIQ (Partner supported) Integration:ti_eset ESET Threat Intelligence (Partner supported) Integration:ti_maltiverse Maltiverse (Partner supported) Integration:ti_misp MISP Integration:ti_opencti OpenCTI Integration:ti_otx AlienVault OTX Integration:ti_rapid7_threat_command Rapid7 Threat Command (Partner supported) Integration:ti_recordedfuture Recorded Future Integration:ti_threatconnect ThreatConnect Integration:ti_threatq ThreatQuotient (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

6 participants

@chrisberkhout @elasticmachine @jsoriano @kcreddy @efd6 @andrewkroh