[trend_micro_vision_one] Improve ingest pipeline structure, error handling, and ECS field mappings

[brijesh-elastic]

Proposed commit message

trend_micro_vision_one: restructure ingest pipelines and improve ECS mappings

Restructure ingest pipelines across all six data streams (alert,
audit, detection, endpoint_activity, network_activity, telemetry)
for better organization, error handling, and maintainability.
Add processor tags throughout all pipelines for error tracing.
Improve on_failure handlers to include processor type, tag,
pipeline, and failure message. Fix request tracer config for
httpjson inputs. Bump ECS version from 8.11.0 to
9.3.0.

Alert: normalize event.severity from string labels
(low/medium/high/critical) to numeric values (21/47/73/99),
populate event.created from createdDateTime, extract
threat.technique.id from matched MITRE rules, and collect
related.ip from impact scope entities.

Audit and detection: replace individual field renames with a
camelCase-to-snake_case conversion script. Detection also adds
mappings for fields present in pipeline test logs but not
previously parsed (risk_level_value, sender_ip, tactic_id, and
others).

Endpoint activity: add human-readable *_value fields that
resolve numeric codes to descriptive names for event.id,
event.sub_id, object.integrity_level, object.true_type,
object.sub_true_type, and win_event_id.

Network activity: normalize eventTime to UNIX_MS before the
date processor (which only honours the first listed format), and
add an act_value field resolving numeric action codes to text.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/trend_micro_vision_one directory.
• Run the following command to run tests.

elastic-package test -v

Related issues

• Closes trend_micro_vision_one: Normalise event.severity handling #17395

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[github-actions]

Vale Linting Results

Summary: 17 warnings, 2 suggestions found

⚠️ Warnings (17)

File	Line	Rule	Message
packages/trend_micro_vision_one/docs/README.md	267	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/trend_micro_vision_one/docs/README.md	269	Elastic.QuotesPunctuation	Place punctuation inside closing quotation marks.
packages/trend_micro_vision_one/docs/README.md	270	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/trend_micro_vision_one/docs/README.md	271	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/trend_micro_vision_one/docs/README.md	439	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/trend_micro_vision_one/docs/README.md	441	Elastic.QuotesPunctuation	Place punctuation inside closing quotation marks.
packages/trend_micro_vision_one/docs/README.md	442	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/trend_micro_vision_one/docs/README.md	443	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/trend_micro_vision_one/docs/README.md	784	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/trend_micro_vision_one/docs/README.md	786	Elastic.QuotesPunctuation	Place punctuation inside closing quotation marks.
packages/trend_micro_vision_one/docs/README.md	787	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/trend_micro_vision_one/docs/README.md	788	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/trend_micro_vision_one/docs/README.md	804	Elastic.BritishSpellings	Use American English spelling 'gray' instead of British English 'Grey'.
packages/trend_micro_vision_one/docs/README.md	1771	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.
packages/trend_micro_vision_one/docs/README.md	1773	Elastic.QuotesPunctuation	Place punctuation inside closing quotation marks.
packages/trend_micro_vision_one/docs/README.md	1774	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.
packages/trend_micro_vision_one/docs/README.md	1775	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.

💡 Suggestions (2)

File	Line	Rule	Message
packages/trend_micro_vision_one/docs/README.md	881	Elastic.WordChoice	Consider using 'run, start' instead of 'execute', unless the term is in the UI.
packages/trend_micro_vision_one/docs/README.md	906	Elastic.WordChoice	Consider using 'open' instead of 'launch', unless the term is in the UI.

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package trend_micro_vision_one 👍(6) 💚(3) 💔(3)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
alert	6289.31	3216.47	-3072.84 (-48.86%)	💔
detection	3875.97	2534.85	-1341.12 (-34.6%)	💔
endpoint_activity	4719.21	3822.63	-896.58 (-19%)	💔

To see the full report comment with /test benchmark fullreport

[brijesh-elastic]

We can't normalize event.severity according to the issue because the API returns integer values rather than the expected numeric format.

The logs contain two severity-related fields: cat (documented) and severity (not documented but present in the pipeline test logs). We are using cat to populate the ECS field event.severity. In the pipeline we map cat to severity_level so existing ingest pipelines are not broken.

[efd6]

Can you provide a link for the documentation of the cat field? I cannot find any docs for it except as part of CEF.

[kcreddy]

If the cat contains values between 0-100, then we can derive them as per this, which is the basis for arriving at our current event.severity values.

[brijesh-elastic]

Can you provide a link for the documentation of the cat field?

It isn’t publicly available; you need to log in to access it. Attaching a screenshot.

If the cat contains values between 0-100, then we can derive them as per elastic/detection-rules#1 (comment), which is the basis for arriving at our current event.severity values.

We aren’t sure whether the value will always fall in the 0–100 range. In live logs we’re only seeing it as 100 so far.

[kcreddy]

I found a conflicting page which refers different values 100-200 : https://github.com/trendmicro/tm-v1-schema/blob/main/doc/Detections.yaml#L2121-L2129

[brijesh-elastic]

I found a conflicting page which refers different values 100-200 : https://github.com/trendmicro/tm-v1-schema/blob/main/doc/Detections.yaml#L2121-L2129

Aren't they example values?

[efd6]

I think "The weighted priority of the incident" means that we should not be using this to populate the event.severity field at all. It should stay as a vendor-namespaced field with that description.

[brijesh-elastic]

Removed event.severity ECS mapping, 45363b5.

[efd6]

Suggest the following commit message:

trend_micro_vision_one: restructure ingest pipelines and improve ECS mappings

Restructure ingest pipelines across all six data streams (alert,
audit, detection, endpoint_activity, network_activity, telemetry)
for better organization, error handling, and maintainability.
Add processor tags throughout all pipelines for error tracing.
Improve on_failure handlers to include processor type, tag,
pipeline, and failure message. Bump ECS version from 8.11.0 to
9.3.0.

Alert: normalize event.severity from string labels
(low/medium/high/critical) to numeric values (21/47/73/99),
populate event.created from createdDateTime, extract
threat.technique.id from matched MITRE rules, and collect
related.ip from impact scope entities.

Audit and detection: replace individual field renames with a
camelCase-to-snake_case conversion script. Detection also adds
mappings for fields present in pipeline test logs but not
previously parsed (risk_level_value, sender_ip, tactic_id, and
others).

Endpoint activity: add human-readable *_value fields that
resolve numeric codes to descriptive names for event.id,
event.sub_id, object.integrity_level, object.true_type,
object.sub_true_type, and win_event_id.

Network activity: normalize eventTime to UNIX_MS before the
date processor (which only honours the first listed format), and
add an act_value field resolving numeric action codes to text.

I think it would be helpful in future to break large changes like this into a number of targeted and well explained commits so that it is easier to review.

[efd6]

Suggested change

	# convert aptRelated ('0'/'1') to related_apt (boolean)
	# convert apt_related ('0'/'1') to related_apt (boolean)

[efd6]

Suggested change

	description: Resolved numeric risk level to text value.
	description: Resolve numeric risk level to text value.

[efd6]

Suggested change

	description: Resolved numeric action id to text value.
	description: Resolve numeric action id to text value.

[efd6]

I'm not sure we can do this; process.entity_id is a "Unique identifier for the process." Process IDs are reused and so not globally unique.

[brijesh-elastic]

Right, I'll remove it.

[efd6]

Can you provide a link for the documentation of the cat field? I cannot find any docs for it except as part of CEF.

[efd6]

Why do we drop fields containing @. This is surprising enough to warrant a comment.

Also, prefer

Suggested change

	if (!entry.getKey().contains("@")) {
	String newKey = camelToSnake(entry.getKey());
	newObj[newKey] = convertToSnakeCase(entry.getValue());
	}
	if (entry.getKey().contains("@")) {
	continue;
	}
	newObj[camelToSnake(entry.getKey())] = convertToSnakeCase(entry.getValue());

[brijesh-elastic]

I took the script from other references. I don't think we need to skip this.

[github-actions]

Buildkite failure is caused by policy golden files not being updated after renaming tracer config keys in the HTTPJSON stream templates.

Build: https://buildkite.com/elastic/integrations/builds/39416

What failed: Job Check integrations trend_micro_vision_one (test-integrations-trend_micro_vision_one) in package test execution.

Error:
ERROR unexpected content in policy: --- want +++ got

Key failing excerpt

2026/03/10 15:04:22 ERROR unexpected content in policy: --- want
+++ got
@@ -23,2 +23,6 @@
           request.timeout: 30s
+          request.tracer:
+            enabled: false
+            filename: ../../logs/httpjson/http-request-trace-*.ndjson
+            maxbackups: 5
           request.transforms:
@@ -49,6 +53,2 @@
           request.url: (example.visionone.trendmicro.com/redacted)
-          resource.tracer:
-            enabled: false
-            filename: ../../logs/httpjson/http-request-trace-*.ndjson
-            maxbackups: 5

Root cause:
The PR changes three stream templates from resource.tracer to request.tracer, but corresponding policy expected snapshots still assert resource.tracer.

Changed templates in PR:

• packages/trend_micro_vision_one/data_stream/alert/agent/stream/httpjson.yml.hbs (line ~3 in patch)
• packages/trend_micro_vision_one/data_stream/audit/agent/stream/httpjson.yml.hbs (line ~3 in patch)
• packages/trend_micro_vision_one/data_stream/detection/agent/stream/httpjson.yml.hbs (line ~3 in patch)

Current expected snapshots still use old key:

• packages/trend_micro_vision_one/data_stream/alert/_dev/test/policy/test-default.expected:38
• packages/trend_micro_vision_one/data_stream/audit/_dev/test/policy/test-default.expected:41
• packages/trend_micro_vision_one/data_stream/detection/_dev/test/policy/test-default.expected:50

Recommended fix:
Regenerate/update the affected policy expected files (test-default.expected and likely test-all.expected) for alert, audit, and detection so they expect request.tracer.

Practical approach:

1. Run package policy tests for trend_micro_vision_one and accept updated snapshots.
2. Verify no remaining resource.tracer references in these three HTTPJSON policy expectations.
3. Re-run CI for the package.

Verification:
I did not run local tests for the PR branch; this diagnosis is based on Buildkite failed job logs plus PR diff inspection.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[efd6]

The policy tests need to be updated.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 45363b5

History

• 💔 Build #39416 failed b3d2afd
• 💚 Build #39400 succeeded 87d1eab
• 💚 Build #39281 succeeded 397d025
• 💚 Build #39271 succeeded 778a3e9
• 💔 Build #39270 failed 3d9cb2a

cc @brijesh-elastic

[kcreddy]

LGTM, please wait for @efd6

[efd6]

We should be able to handle both the original and the new timestamp, but I guess we're handling the old form in the next event so I suppose a replacement is OK.

[elastic-vault-github-plugin-prod]

Package trend_micro_vision_one - 2.9.0 containing this change is available at https://epr.elastic.co/package/trend_micro_vision_one/2.9.0/