ssi_some: remove duplicated field definitions in transforms by mrodm · Pull Request #16180 · elastic/integrations

mrodm · 2025-12-01T14:53:17Z

Proposed commit message

Update field definitions in transforms to avoid duplicated fields found while working on elastic/elastic-package#2558

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Relates Using 'external: ecs' without a _dev/build/build.yml should fail elastic-package#2558

mrodm · 2025-12-01T15:09:11Z

- external: ecs
-  name: event.created


Field definition duplicated just above:

Error:

Error: checking package failed: linting package failed: found 1 validation error:

field "event.created" is defined multiple times for transform "latest_offense", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239772963433816/elastic/integrations/packages/ibm_qradar/elasticsearch/transform/latest_offense/fields/ecs.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239772963433816/elastic/integrations/packages/ibm_qradar/elasticsearch/transform/latest_offense/fields/ecs.yml

mrodm · 2025-12-01T15:10:22Z

- name: package.name
-  external: ecs
- name: package.version
-  external: ecs


Keep just the field definitions in packages/rapid7_insightvm/elasticsearch/transform/latest_cdr_vulnerabilities/fields/package.yml:

integrations/packages/rapid7_insightvm/elasticsearch/transform/latest_cdr_vulnerabilities/fields/package.yml

Lines 1 to 11 in 52e8777

- name: package

type: group

fields:

- name: fixed_version

type: keyword

- name: name

type: keyword

external: ecs

- name: version

type: keyword

external: ecs

Error:

Error: checking package failed: linting package failed: found 2 validation errors:

field "package.name" is defined multiple times for transform "latest_cdr_vulnerabilities", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239778925107839/elastic/integrations/packages/rapid7_insightvm/elasticsearch/transform/latest_cdr_vulnerabilities/fields/ecs.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239778925107839/elastic/integrations/packages/rapid7_insightvm/elasticsearch/transform/latest_cdr_vulnerabilities/fields/package.yml

field "package.version" is defined multiple times for transform "latest_cdr_vulnerabilities", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239778925107839/elastic/integrations/packages/rapid7_insightvm/elasticsearch/transform/latest_cdr_vulnerabilities/fields/ecs.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239778925107839/elastic/integrations/packages/rapid7_insightvm/elasticsearch/transform/latest_cdr_vulnerabilities/fields/package.yml

mrodm · 2025-12-01T15:11:28Z

- name: package.name
-  external: ecs
- name: package.path
-  external: ecs
- name: package.version
-  external: ecs


Keep the field definitions from: packages/tenable_io/elasticsearch/transform/latest_cdr_vulnerabilities/fields/package.yml

integrations/packages/tenable_io/elasticsearch/transform/latest_cdr_vulnerabilities/fields/package.yml

Lines 1 to 14 in 52e8777

- name: package

type: group

fields:

- name: fixed_version

type: keyword

- name: name

type: keyword

external: ecs

- name: path

type: keyword

external: ecs

- name: version

type: keyword

external: ecs

Error:

Error: checking package failed: linting package failed: found 3 validation errors:

field "package.version" is defined multiple times for transform "latest_cdr_vulnerabilities", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239786045634364/elastic/integrations/packages/tenable_io/elasticsearch/transform/latest_cdr_vulnerabilities/fields/ecs.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239786045634364/elastic/integrations/packages/tenable_io/elasticsearch/transform/latest_cdr_vulnerabilities/fields/package.yml

field "package.path" is defined multiple times for transform "latest_cdr_vulnerabilities", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239786045634364/elastic/integrations/packages/tenable_io/elasticsearch/transform/latest_cdr_vulnerabilities/fields/ecs.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239786045634364/elastic/integrations/packages/tenable_io/elasticsearch/transform/latest_cdr_vulnerabilities/fields/package.yml

field "package.name" is defined multiple times for transform "latest_cdr_vulnerabilities", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239786045634364/elastic/integrations/packages/tenable_io/elasticsearch/transform/latest_cdr_vulnerabilities/fields/ecs.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239786045634364/elastic/integrations/packages/tenable_io/elasticsearch/transform/latest_cdr_vulnerabilities/fields/package.yml

mrodm · 2025-12-01T15:12:22Z

- name: tags
-  type: keyword
-  description: User defined tags.


Keep the field definition from: packages/ti_crowdstrike/elasticsearch/transform/latest_ioc/fields/ecs.yml

integrations/packages/ti_crowdstrike/elasticsearch/transform/latest_ioc/fields/ecs.yml

Lines 27 to 28 in 52e8777

- name: tags

type: keyword

Error:

Error: checking package failed: linting package failed: found 1 validation error:

field "tags" is defined multiple times for transform "latest_ioc", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239786988523423/elastic/integrations/packages/ti_crowdstrike/elasticsearch/transform/latest_ioc/fields/beats.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239786988523423/elastic/integrations/packages/ti_crowdstrike/elasticsearch/transform/latest_ioc/fields/ecs.yml

mrodm · 2025-12-01T15:13:19Z

-# Below fields to be moved into base-fields.yml after kibana.version changed to >= 8.14 
-# Related to fix: https://github.com/elastic/kibana/pull/177608
- name: event.module
-  type: constant_keyword
-  description: Event module.
-  value: ti_custom
- name: event.dataset
-  type: constant_keyword
-  description: Event dataset.
-  value: ti_custom.indicator


Keep the field definitions from: packages/ti_custom/elasticsearch/transform/latest_ioc/fields/base-fields.yml

integrations/packages/ti_custom/elasticsearch/transform/latest_ioc/fields/base-fields.yml

Lines 10 to 17 in 52e8777

- name: event.module

type: constant_keyword

description: Event module

value: ti_custom

- name: event.dataset

type: constant_keyword

description: Event dataset

value: ti_custom.indicator

Error:

Error: checking package failed: linting package failed: found 2 validation errors:

field "event.module" is defined multiple times for transform "latest_ioc", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239789143584049/elastic/integrations/packages/ti_custom/elasticsearch/transform/latest_ioc/fields/base-fields.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239789143584049/elastic/integrations/packages/ti_custom/elasticsearch/transform/latest_ioc/fields/ecs.yml

field "event.dataset" is defined multiple times for transform "latest_ioc", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239789143584049/elastic/integrations/packages/ti_custom/elasticsearch/transform/latest_ioc/fields/base-fields.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239789143584049/elastic/integrations/packages/ti_custom/elasticsearch/transform/latest_ioc/fields/ecs.yml

mrodm · 2025-12-01T15:14:08Z

- name: "@timestamp"
-  type: date
-  description: Event timestamp.


Keep the field definition from: packages/ti_eclecticiq/elasticsearch/transform/latest_ioc/fields/base-fields.yml

integrations/packages/ti_eclecticiq/elasticsearch/transform/latest_ioc/fields/base-fields.yml

Lines 13 to 15 in 52e8777

- name: "@timestamp"

type: date

description: Event timestamp.

Error:

Error: checking package failed: linting package failed: found 1 validation error:

field "@timestamp" is defined multiple times for transform "latest_ioc", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239787195394465/elastic/integrations/packages/ti_eclecticiq/elasticsearch/transform/latest_ioc/fields/base-fields.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764239787195394465/elastic/integrations/packages/ti_eclecticiq/elasticsearch/transform/latest_ioc/fields/fields.yml

mrodm · 2025-12-01T15:14:40Z

- external: ecs
-  name: organization.name


Field definition duplicated just above.

Error:

Error: checking package failed: linting package failed: found 1 validation error:

field "organization.name" is defined multiple times for transform "ip", found in: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764240052938457124/elastic/integrations/packages/ti_greynoise/elasticsearch/transform/ip/fields/ecs.yml, /opt/buildkite-agent/builds/bk-agent-prod-gcp-1764240052938457124/elastic/integrations/packages/ti_greynoise/elasticsearch/transform/ip/fields/ecs.yml

elastic-vault-github-plugin-prod · 2025-12-01T15:18:07Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

mrodm · 2025-12-01T15:59:53Z

      external: ecs
    - name: version
-      type: keyword
      external: ecs


External fields are already keyword.

package.version
https://github.com/elastic/ecs/blob/ce703ab38f0c128c0dfeeb2daf0893ff3021cdff/generated/ecs/ecs_nested.yml#L9478

package.path
https://github.com/elastic/ecs/blob/ce703ab38f0c128c0dfeeb2daf0893ff3021cdff/generated/ecs/ecs_nested.yml#L9430

package.name
https://github.com/elastic/ecs/blob/ce703ab38f0c128c0dfeeb2daf0893ff3021cdff/generated/ecs/ecs_nested.yml#L9419

mrodm · 2025-12-01T16:01:54Z

  type: keyword
 - name: tags
  type: keyword
+  description: User defined tags.


Added the same description as in the other file.

elasticmachine · 2025-12-01T17:01:44Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6

LGTM

Suggest the following commit message:

ssi_some: remove duplicated field definitions in transforms

Field validation tooling improvements revealed duplicate field definitions
across multiple transform field files, causing package linting failures.
This removes the duplicate definitions while preserving correct field
mappings and increments transform versions to ensure proper updates.

Affected packages: ibm_qradar, rapid7_insightvm, tenable_io, 
ti_crowdstrike, ti_custom, ti_eclecticiq, ti_greynoise

elasticmachine · 2025-12-02T11:50:29Z

💚 Build Succeeded

Buildkite Build
Commit: 6fee734

History

💚 Build #34874 succeeded afdd33f
💚 Build #34872 succeeded 673cfd5
💚 Build #34870 succeeded 2efc3dd

cc @mrodm

elastic-vault-github-plugin-prod · 2025-12-02T12:21:39Z

Package ibm_qradar - 0.1.1 containing this change is available at https://epr.elastic.co/package/ibm_qradar/0.1.1/

elastic-vault-github-plugin-prod · 2025-12-02T12:21:40Z

Package rapid7_insightvm - 2.5.1 containing this change is available at https://epr.elastic.co/package/rapid7_insightvm/2.5.1/

elastic-vault-github-plugin-prod · 2025-12-02T12:21:41Z

Package tenable_io - 4.6.1 containing this change is available at https://epr.elastic.co/package/tenable_io/4.6.1/

elastic-vault-github-plugin-prod · 2025-12-02T12:21:42Z

Package ti_crowdstrike - 2.6.1 containing this change is available at https://epr.elastic.co/package/ti_crowdstrike/2.6.1/

elastic-vault-github-plugin-prod · 2025-12-02T12:21:43Z

Package ti_custom - 1.2.3 containing this change is available at https://epr.elastic.co/package/ti_custom/1.2.3/

elastic-vault-github-plugin-prod · 2025-12-02T12:21:44Z

Package ti_eclecticiq - 1.4.2 containing this change is available at https://epr.elastic.co/package/ti_eclecticiq/1.4.2/

elastic-vault-github-plugin-prod · 2025-12-02T12:21:45Z

Package ti_greynoise - 0.7.2 containing this change is available at https://epr.elastic.co/package/ti_greynoise/0.7.2/

mrodm self-assigned this Dec 1, 2025

mrodm commented Dec 1, 2025

View reviewed changes

mrodm added 2 commits December 1, 2025 16:27

Update transform fields

caf6868

Update destination index and fleet_transform_version

afdd33f

mrodm force-pushed the update_transforms_fields branch from 673cfd5 to afdd33f Compare December 1, 2025 15:48

mrodm commented Dec 1, 2025

View reviewed changes

mrodm marked this pull request as ready for review December 1, 2025 16:24

mrodm requested a review from a team as a code owner December 1, 2025 16:24

mrodm mentioned this pull request Dec 1, 2025

Using 'external: ecs' without a _dev/build/build.yml should fail elastic/elastic-package#2558

Closed

mrodm requested a review from a team December 1, 2025 16:54

efd6 approved these changes Dec 1, 2025

View reviewed changes

mrodm changed the title ~~Remove duplicated fields in transforms~~ Dec 2, 2025

Update URL from changelog entries

6fee734

mrodm merged commit 1ed3776 into elastic:main Dec 2, 2025
7 checks passed

mrodm deleted the update_transforms_fields branch December 2, 2025 12:00

github-actions Bot mentioned this pull request Mar 19, 2026

[ssi integrations] Use elastic-package links for identical files #17904

Open

kcreddy mentioned this pull request Aug 7, 2026

aws_securityhub: latest_findings transform fails once total_fields.limit (2000) is exhausted #20604

Closed

Navigation Menu

Uh oh!

ssi_some: remove duplicated field definitions in transforms - #16180

ssi_some: remove duplicated field definitions in transforms#16180
mrodm merged 3 commits into
elastic:mainfrom
mrodm:update_transforms_fields

mrodm commented Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented Dec 1, 2025

mrodm Dec 1, 2025

mrodm Dec 1, 2025

elasticmachine commented Dec 1, 2025

efd6 left a comment

elasticmachine commented Dec 2, 2025

Uh oh!

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

Labels

4 participants

	- name: package
	type: group
	fields:
	- name: fixed_version
	type: keyword
	- name: name
	type: keyword
	external: ecs
	- name: version
	type: keyword
	external: ecs

	- name: event.module
	type: constant_keyword
	description: Event module
	value: ti_custom
	- name: event.dataset
	type: constant_keyword
	description: Event dataset
	value: ti_custom.indicator

Uh oh!

Conversation

mrodm commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

mrodm Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

mrodm Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

mrodm Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

mrodm Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

mrodm Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

mrodm Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

mrodm Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

elastic-vault-github-plugin-prod Bot commented Dec 1, 2025

🚀 Benchmarks report

mrodm Dec 1, 2025

Choose a reason for hiding this comment

mrodm Dec 1, 2025

Choose a reason for hiding this comment

elasticmachine commented Dec 1, 2025

efd6 left a comment

Choose a reason for hiding this comment

elasticmachine commented Dec 2, 2025

💚 Build Succeeded

History

Uh oh!

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

elastic-vault-github-plugin-prod Bot commented Dec 2, 2025

Labels

4 participants

mrodm commented Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading

mrodm Dec 1, 2025 •

edited

Loading