Skip to content

[ti_eset] Add smsscam data stream - #19127

Open
polakovicp wants to merge 2 commits into
elastic:ti_set_polakovicp_enhancementsfrom
polakovicp:ti_eset_smsscam
Open

[ti_eset] Add smsscam data stream#19127
polakovicp wants to merge 2 commits into
elastic:ti_set_polakovicp_enhancementsfrom
polakovicp:ti_eset_smsscam

Conversation

@polakovicp

@polakovicp polakovicp commented May 21, 2026

Copy link
Copy Markdown
Contributor

[ti_eset] Add smsscam data stream

PR adds new SMS scam data stream to ti_eset integration. Documentation for this data stream is submitted as a separate PR

Related issues

Relates to PR #18709

> elastic-package test                                                                                                  

2026/08/07 10:14:34  INFO elastic-package v0.125.1 version-hash undefined (build time: unknown)
2026/08/07 10:14:34  INFO elastic-stack: 9.4.2
--- Test results for package: ti_eset - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                        │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ ti_eset │             │ asset     │ dashboard ti_eset-402bf6b0-8194-11ee-a704-a3364df0a052 is loaded │ PASS   │           0s │
│ ti_eset │             │ asset     │ dashboard ti_eset-4fc91930-819a-11ee-a704-a3364df0a052 is loaded │ PASS   │           0s │
│ ti_eset │             │ asset     │ dashboard ti_eset-c27d91e0-819a-11ee-a704-a3364df0a052 is loaded │ PASS   │           0s │
│ ti_eset │             │ asset     │ dashboard ti_eset-f608e350-8192-11ee-a704-a3364df0a052 is loaded │ PASS   │           0s │
│ ti_eset │ apt         │ asset     │ index_template logs-ti_eset.apt is loaded                        │ PASS   │           0s │
│ ti_eset │ apt         │ asset     │ ingest_pipeline logs-ti_eset.apt-1.10.0 is loaded                │ PASS   │           0s │
│ ti_eset │ botnet      │ asset     │ index_template logs-ti_eset.botnet is loaded                     │ PASS   │           0s │
│ ti_eset │ botnet      │ asset     │ ingest_pipeline logs-ti_eset.botnet-1.10.0 is loaded             │ PASS   │           0s │
│ ti_eset │ cc          │ asset     │ index_template logs-ti_eset.cc is loaded                         │ PASS   │           0s │
│ ti_eset │ cc          │ asset     │ ingest_pipeline logs-ti_eset.cc-1.10.0 is loaded                 │ PASS   │           0s │
│ ti_eset │ domains     │ asset     │ index_template logs-ti_eset.domains is loaded                    │ PASS   │           0s │
│ ti_eset │ domains     │ asset     │ ingest_pipeline logs-ti_eset.domains-1.10.0 is loaded            │ PASS   │           0s │
│ ti_eset │ files       │ asset     │ index_template logs-ti_eset.files is loaded                      │ PASS   │           0s │
│ ti_eset │ files       │ asset     │ ingest_pipeline logs-ti_eset.files-1.10.0 is loaded              │ PASS   │           0s │
│ ti_eset │ ip          │ asset     │ index_template logs-ti_eset.ip is loaded                         │ PASS   │           0s │
│ ti_eset │ ip          │ asset     │ ingest_pipeline logs-ti_eset.ip-1.10.0 is loaded                 │ PASS   │           0s │
│ ti_eset │ smsscam     │ asset     │ index_template logs-ti_eset.smsscam is loaded                    │ PASS   │           0s │
│ ti_eset │ smsscam     │ asset     │ ingest_pipeline logs-ti_eset.smsscam-1.10.0 is loaded            │ PASS   │           0s │
│ ti_eset │ url         │ asset     │ index_template logs-ti_eset.url is loaded                        │ PASS   │           0s │
│ ti_eset │ url         │ asset     │ ingest_pipeline logs-ti_eset.url-1.10.0 is loaded                │ PASS   │           0s │
╰─────────┴─────────────┴───────────┴──────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_eset - END   ---
Done
Run pipeline tests for the package
--- Test results for package: ti_eset - START ---
╭─────────┬─────────────┬───────────┬─────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                       │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼─────────────────────────────────────────────────┼────────┼──────────────┤
│ ti_eset │ apt         │ pipeline  │ (ingest pipeline warnings test-eset-ndjson.log) │ PASS   │   1.2871349s │
│ ti_eset │ apt         │ pipeline  │ test-eset-ndjson.log                            │ PASS   │   486.9944ms │
│ ti_eset │ botnet      │ pipeline  │ (ingest pipeline warnings test-eset-ndjson.log) │ PASS   │   956.3832ms │
│ ti_eset │ botnet      │ pipeline  │ test-eset-ndjson.log                            │ PASS   │   156.5129ms │
│ ti_eset │ cc          │ pipeline  │ (ingest pipeline warnings test-eset-ndjson.log) │ PASS   │   927.7941ms │
│ ti_eset │ cc          │ pipeline  │ test-eset-ndjson.log                            │ PASS   │   159.0035ms │
│ ti_eset │ domains     │ pipeline  │ (ingest pipeline warnings test-eset-ndjson.log) │ PASS   │   990.1571ms │
│ ti_eset │ domains     │ pipeline  │ test-eset-ndjson.log                            │ PASS   │     84.096ms │
│ ti_eset │ files       │ pipeline  │ (ingest pipeline warnings test-eset-ndjson.log) │ PASS   │   993.9527ms │
│ ti_eset │ files       │ pipeline  │ test-eset-ndjson.log                            │ PASS   │   234.0291ms │
│ ti_eset │ ip          │ pipeline  │ (ingest pipeline warnings test-eset-ndjson.log) │ PASS   │   1.0061225s │
│ ti_eset │ ip          │ pipeline  │ test-eset-ndjson.log                            │ PASS   │   130.1589ms │
│ ti_eset │ smsscam     │ pipeline  │ (ingest pipeline warnings test-eset-ndjson.log) │ PASS   │   1.0281994s │
│ ti_eset │ smsscam     │ pipeline  │ test-eset-ndjson.log                            │ PASS   │    74.6241ms │
│ ti_eset │ url         │ pipeline  │ (ingest pipeline warnings test-eset-ndjson.log) │ PASS   │   1.0152789s │
│ ti_eset │ url         │ pipeline  │ test-eset-ndjson.log                            │ PASS   │   162.4506ms │
╰─────────┴─────────────┴───────────┴─────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_eset - END   ---
Done
Run static tests for the package
2026/08/07 10:15:04  INFO elastic-package v0.125.1 version-hash undefined (build time: unknown)
--- Test results for package: ti_eset - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ ti_eset │ apt         │ static    │ Verify sample_event.json │ PASS   │    80.8624ms │
│ ti_eset │ botnet      │ static    │ Verify sample_event.json │ PASS   │    73.3995ms │
│ ti_eset │ cc          │ static    │ Verify sample_event.json │ PASS   │    64.4114ms │
│ ti_eset │ domains     │ static    │ Verify sample_event.json │ PASS   │    72.1927ms │
│ ti_eset │ files       │ static    │ Verify sample_event.json │ PASS   │    71.1082ms │
│ ti_eset │ ip          │ static    │ Verify sample_event.json │ PASS   │    65.9286ms │
│ ti_eset │ smsscam     │ static    │ Verify sample_event.json │ PASS   │    69.7198ms │
│ ti_eset │ url         │ static    │ Verify sample_event.json │ PASS   │    65.4011ms │
╰─────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_eset - END   ---
Done
Run system tests for the package
--- Test results for package: ti_eset - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼──────────────┤
│ ti_eset │ apt         │ system    │ httpjson  │ PASS   │  48.3440943s │
│ ti_eset │ botnet      │ system    │ httpjson  │ PASS   │  50.7877475s │
│ ti_eset │ cc          │ system    │ httpjson  │ PASS   │  45.8434489s │
│ ti_eset │ domains     │ system    │ httpjson  │ PASS   │  50.9207716s │
│ ti_eset │ files       │ system    │ httpjson  │ PASS   │  49.8628262s │
│ ti_eset │ ip          │ system    │ httpjson  │ PASS   │  51.4722095s │
│ ti_eset │ smsscam     │ system    │ cel       │ PASS   │  50.4508311s │
│ ti_eset │ url         │ system    │ httpjson  │ PASS   │  44.6062436s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴──────────────╯
--- Test results for package: ti_eset - END   ---
Done
@polakovicp
polakovicp requested a review from a team as a code owner May 21, 2026 14:46
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Reviewers

Buildkite won't run for external contributors automatically; you need to add a comment:

  • /test : will kick off a build in Buildkite.

NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details.

@andrewkroh andrewkroh added Integration:ti_eset ESET Threat Intelligence (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels May 21, 2026
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@botelastic

botelastic Bot commented Jun 25, 2026

Copy link
Copy Markdown

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic Bot added the Stalled label Jun 25, 2026
@mergify

mergify Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request
@vera-review-bot

Copy link
Copy Markdown

👀 I have started reviewing the PR

delay: 120s
field: event.ingested
settings:
num_failure_retries: -1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM elasticsearch/transform/smsscam_latest_ioc/transform.yml:28

New transform omits settings.unattended: true

The smsscam_latest_ioc transform sets only settings.num_failure_retries: -1 and omits settings.unattended: true. Every other transform in this package (url_latest_ioc, ip_latest_ioc, cc_latest_ioc, botnet_latest_ioc, domains_latest_ioc, files_latest_ioc, apt_latest_ioc) sets unattended: true, and changelog 1.9.0 ("Allow transforms to run in unattended mode") deliberately enabled it package-wide. Without unattended mode the transform will not auto-recover from transient failures and will stop on errors during normal operation and package upgrades, leaving the latest-IoC destination index stale. num_failure_retries controls retry count but does not restore the auto-recovery behavior that unattended provides.

Recommendation:

Add unattended: true to the settings block to match the other ti_eset transforms (it can coexist with num_failure_retries):

settings:
  unattended: true
  num_failure_retries: -1

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@botelastic botelastic Bot removed the Stalled label Jun 29, 2026
@vera-review-bot

Copy link
Copy Markdown

👀 I have started reviewing the PR

type: logs
ilm_policy: logs-ti_eset.smsscam-default_policy
streams:
- input: cel

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 HIGH data_stream/smsscam/manifest.yml:5

New cel data stream has no cel input declared; credentials unwired

This stream declares input: cel, but the root manifest's only policy template (eset) declares a single httpjson input and nothing of type cel. The seven existing data streams all use input: httpjson with run_as_cel: true and the httpjson_as_cel.yml.hbs template, drawing username/password from the policy-template httpjson input vars. The new cel.yml.hbs references {{username}} and {{password}} in its state: block, yet neither var is defined in this data stream's streams[].vars, and no cel input exists at the policy-template level to supply them. As a result the Basic auth header renders from empty credentials (Basic + base64 of :) and the feed cannot authenticate; the undeclared cel input also causes elastic-package check to fail because the stream input is not referenced by any policy template.

Recommendation:

Add a cel input to the eset policy template in the root manifest.yml carrying the same credential vars the httpjson input defines, so the new stream's {{username}}/{{password}} resolve:

policy_templates:
  - name: eset
    inputs:
      - type: httpjson
        # ...existing httpjson input...
      - type: cel
        title: ETI feeds (CEL)
        description: Collect data from ETI feeds (TAXII version 2) via CEL
        vars:
          - name: username
            type: text
            title: ETI Username
            required: true
            show_user: true
          - name: password
            type: password
            title: ETI Password
            secret: true
            required: true
            show_user: true

Alternatively, keep parity with the other seven streams by using input: httpjson with run_as_cel: true and the existing httpjson_as_cel.yml.hbs template instead of a native cel input.


🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

data_stream:
vars:
preserve_original_event: true
url: http://{{Hostname}}:{{Port}}/taxii2/643f4eb5-f8b7-46a3-a606-6d61d5ce223a/collections/6e20217a2e1246b8ab11be29f759f716/objects/

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 HIGH data_stream/smsscam/.../test-cel-config.yml:9

System test targets smsscam collection with no mock rule

The system test points url at the smsscam TAXII collection 6e20217a2e1246b8ab11be29f759f716 and asserts hit_count: 1, but the shared mock served by the eti service (_dev/deploy/docker/files/config.yml) defines no rule for that collection path — it only has rules for the botnet, cc, domains, files, ip, and MISP/url collections. With no matching rule the mock returns a non-2xx response, the CEL program takes its error branch and emits zero events, so the hit_count: 1 assertion fails when the system test runs.

Recommendation:

Add a rule for the smsscam collection to _dev/deploy/docker/files/config.yml returning at least one indicator object so the test produces a hit:

rules:
  # SMS scam
  - path: /taxii2/643f4eb5-f8b7-46a3-a606-6d61d5ce223a/collections/6e20217a2e1246b8ab11be29f759f716/objects/
    methods: ["GET"]
    request_headers:
      Authorization:
        - "Basic dGVzdDp0ZXN0"
    query_params:
      match[type]: "indicator"
      added_after: "{added_after:.*}"
    responses:
      - status_code: 200
        headers:
          Content-Type: "application/taxii+json;version=2.1"
        body: |-
          {"more": false, "objects": [{"type": "indicator", "id": "indicator--9da47616-6288-475e-99e2-6827e77a7c17", "created": "2024-03-19T15:20:09.000Z", "modified": "2024-03-19T15:20:09.000Z", "name": "Unwanted", "description": "Host is known source of active fraudulent content.", "pattern": "[url:value='www.example.com']", "pattern_type": "stix", "valid_from": "2024-03-19T15:20:09Z", "valid_until": "2024-03-21T15:20:09Z", "labels": ["unwanted-activity"]}]}

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Comment thread packages/ti_eset/manifest.yml Outdated
title: Sample logo
size: 96x96
type: image/svg+xml
vars:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟠 High confidence: medium path: packages/ti_eset/manifest.yml:33

Moving username/password from the httpjson input to package-level vars changes their scope, so credentials saved by existing 1.9.0 policies are not carried over on upgrade; declare the two vars on both the httpjson and cel inputs instead of hoisting them.

Details

In 1.9.0 username and password were declared under policy_templates[0].inputs[0].vars (the httpjson input). This PR deletes them from that input and re-declares them as package-level vars. Fleet stores and migrates variable values per scope: input-scoped values live under inputs[].vars in the saved package policy, package-scoped values under the policy's top-level vars. Because no package-level vars existed in 1.9.0, the values entered by existing users stay behind on the httpjson input and the new package-level username/password come up empty. Both are required: true with no default, so upgrading an existing ti_eset policy to 1.10.0 yields a package policy that fails required-var validation until the credentials are re-entered by hand. Declaring the same two vars on each input keeps existing httpjson policies intact and still wires credentials into the new cel input that the smsscam stream uses.

Recommendation:

Drop the package-level vars block and declare the credential vars on both inputs:

policy_templates:
  - name: eset
    title: ETI feeds (TAXII version 2.1)
    description: Collect data from ETI feeds (TAXII version 2.1)
    inputs:
      - type: httpjson
        title: ETI Legacy Feeds (TAXII 2.1)
        description: Collection of ETI legacy feeds
        vars:
          - name: username
            type: text
            title: ETI Username
            required: true
            show_user: true
          - name: password
            type: password
            title: ETI Password
            multi: false
            required: true
            secret: true
            show_user: true
      - type: cel
        title: ETI New-Generation Feeds (TAXII 2.1)
        description: Collection of ETI new-generation feeds
        vars:
          - name: username
            type: text
            title: ETI Username
            required: true
            show_user: true
          - name: password
            type: password
            title: ETI Password
            multi: false
            required: true
            secret: true
            show_user: true

If hoisting to package level is intentional, the changelog entry should be typed breaking-change and state that credentials must be re-entered after upgrade.


🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

# newer versions go on top
- version: "1.10.0"
changes:
- description: PLACEHOLDER - specified once all related PRs will be merged to branch ti_set_polakovicp_enhancements

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: high path: packages/ti_eset/changelog.yml:4

The 1.10.0 changelog entry ships a placeholder description and link: TODO; replace both with the real summary and the PR URL before this feature branch merges to main.

Details

The new 1.10.0 entry has description: PLACEHOLDER - specified once all related PRs will be merged to branch ti_set_polakovicp_enhancements and link: TODO. The changelog link field is validated as a GitHub pull/issue URL by elastic-package lint, and TODO is not one. The description is also user-facing release-note text. This PR targets the ti_set_polakovicp_enhancements feature branch rather than main, so the placeholder is understood to be temporary, but it must not survive the merge to main - flagging so it is not lost.

Recommendation:

Replace the placeholder with a real description and PR link:

- version: "1.10.0"
  changes:
    - description: Add SMS scam data stream.
      type: enhancement
      link: https://github.com/elastic/integrations/pull/19127

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@@ -0,0 +1,103 @@
title: SMS scam

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: high path: packages/ti_eset/data_stream/smsscam/manifest.yml:1

The new smsscam data stream is absent from the README template, so it ships with no documentation, no fields table and no sample event; add it to the three tables and a ### SMS scam section.

Details

packages/ti_eset/_dev/build/docs/README.md is unchanged by this PR and mentions smsscam nowhere. That template drives the generated docs/README.md and contains three tables that enumerate every feed (dataset/collection names, transform destination indices and aliases, ILM retention) plus one ### <Name> section per data stream with {{fields "<name>"}} and {{event "<name>"}}. Because smsscam is missing from all of them, the new data stream's field reference and sample event are never rendered and users get no documentation for the feed, its collection URL or its 7d/48h retention - even though elasticsearch/ilm/default_policy.json, lifecycle.yml and the smsscam_latest_ioc transform were all added.

Recommendation:

Add smsscam to the three tables and add a docs section in _dev/build/docs/README.md:

| smsscam | SMS scam stix 2.1      |

| `logs-ti_eset.smsscam-*` | logs-ti_eset_latest.dest_smsscam-* | logs-ti_eset_latest.smsscam |

| `logs-ti_eset.smsscam` | 7d            | 48h           |

### SMS scam

{{fields "smsscam"}}

{{event "smsscam"}}

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@@ -0,0 +1 @@
{"type": "indicator", "spec_version": "2.1", "id": "indicator--9da47616-6288-475e-99e2-6827e77a7c17", "created": "2024-03-19T15:20:09.000Z", "modified": "2024-03-19T15:20:09.000Z", "name": "Unwanted", "description": "Host is known source of active fraudulent content.", "pattern": "[url:value='www.candycasino79.com']", "pattern_type": "stix", "pattern_version": "2.1", "valid_from": "2024-03-19T15:20:09Z", "valid_until": "2024-03-21T15:20:09Z", "labels": ["unwanted-activity"]}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: medium path: packages/ti_eset/data_stream/smsscam/_dev/test/pipeline/test-eset-ndjson.log:1

The pipeline test fixture is a single indicator that exercises only one of the four confidence labels and never the non-indicator drop branch; add fixture lines covering the remaining branches.

Details

The fixture contains one document with labels: ["unwanted-activity"]. The new pipeline has branches that no test covers: the script_bce6660a confidence map has four entries (malicious-activity/phishing-activity -> High, unwanted-activity -> Medium, benign -> Low) and only the Medium branch runs; drop_5a415bbc (ctx.eti?.type != 'indicator') is never exercised; split_eti_pattern_to_eti__patterns_7811368c never sees a multi-pattern ' OR ' value; and the foreach_eti__patterns_605d59c7 grok never sees a pattern it fails to match. The sibling url data stream ships a 10-line fixture covering multiple label values, so this is below the package's own bar for a brand-new stream.

Recommendation:

Extend the fixture with the uncovered branches and regenerate the expected file with elastic-package test pipeline --generate:

{"type": "indicator", "spec_version": "2.1", "id": "indicator--0c1d2e3f-4a5b-4c6d-8e9f-0a1b2c3d4e5f", "created": "2024-03-19T15:20:09.000Z", "modified": "2024-03-19T15:20:09.000Z", "name": "Malicious", "description": "Host actively distributes high-severity threat.", "pattern": "[url:value='https://example.com/a'] OR [url:value='https://example.org/b']", "pattern_type": "stix", "pattern_version": "2.1", "valid_from": "2024-03-19T15:20:09Z", "valid_until": "2024-03-21T15:20:09Z", "labels": ["malicious-activity"]}
{"type": "indicator", "spec_version": "2.1", "id": "indicator--1d2e3f4a-5b6c-4d7e-8f9a-1b2c3d4e5f60", "created": "2024-03-19T15:20:09.000Z", "modified": "2024-03-19T15:20:09.000Z", "name": "Benign", "description": "Host is of uncertain reputation.", "pattern": "[url:value='https://example.net/c']", "pattern_type": "stix", "pattern_version": "2.1", "valid_from": "2024-03-19T15:20:09Z", "valid_until": "2024-03-21T15:20:09Z", "labels": ["benign"]}
{"type": "marking-definition", "spec_version": "2.1", "id": "marking-definition--2e3f4a5b-6c7d-4e8f-9a0b-2c3d4e5f6071", "created": "2024-03-19T15:20:09.000Z"}

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

ilm_policy: logs-ti_eset.smsscam-default_policy
streams:
- input: cel
enabled: false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🔵 Low confidence: medium path: packages/ti_eset/data_stream/smsscam/manifest.yml:6

smsscam is the only data stream in the package declared enabled: false, so the new feed collects nothing until a user finds and enables it; drop the line unless the opt-in default is deliberate.

Details

All seven existing feeds (apt, botnet, cc, domains, files, ip, url) declare no enabled key and therefore default to enabled, and the README's setup steps tell users to "Enable data streams you are interested in and have access to" - i.e. the package's model is enabled-by-default with the user turning feeds off. enabled: false makes smsscam the sole exception and it is also the only stream on the new cel input, so that input produces no data out of the box. If the opt-in default is intentional (for example because access to the SMS scam collection is separately licensed), that is worth stating in the data stream description so users understand they must switch it on.

Recommendation:

Match the other feeds by removing the key:

streams:
  - input: cel
    title: SMS scam
    description: Collect data from ETI SMS scam feed
    template_path: cel.yml.hbs

Or, if opt-in is deliberate, keep it and say so in the description:

streams:
  - input: cel
    enabled: false
    title: SMS scam
    description: Collect data from ETI SMS scam feed. Disabled by default; enable it if your ETI subscription includes the SMS scam collection.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

request.proxy_url: {{proxy_url}}
{{/if}}
request.tracer:
enabled: {{enable_request_tracer}}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟠 High confidence: high path: packages/ti_eset/data_stream/apt/agent/stream/httpjson_as_cel.yml.hbs:14

The new template interpolates enabled: {{enable_request_tracer}}, but these seven data streams never declare a default for that var, so the rendered enabled: is null and Beats treats a null enabled as ON — add default: false to each manifest (as smsscam already does).

Details

The template this PR removes gated the tracer with {{#if enable_request_tracer}}, so an unset var meant no tracer. The replacement interpolates the var directly into a value position. enable_request_tracer is declared type: bool, required: false with NO default in packages/ti_eset/data_stream/{apt,botnet,cc,domains,files,ip,url}/manifest.yml, so when the user leaves it alone Fleet renders nothing and the agent receives enabled: (YAML null). In Beats the tracer config is Enabled *bool and func (t *tracerConfig) enabled() bool { return t != nil && (t.Enabled == nil || *t.Enabled) } (x-pack/filebeat/input/cel/config.go, same shape in httpjson) — a nil Enabled means ENABLED. Every default install of these seven streams therefore starts writing full HTTP request/response traces to the agent's disk, which the var's own description calls out as compromising security. The new smsscam manifest already sets default: false, so the seven migrated manifests are the outliers. This same line is identical in the botnet, cc, domains, files, ip and url copies of httpjson_as_cel.yml.hbs.

Recommendation:

Add an explicit default to the enable_request_tracer var in each of the seven migrated data stream manifests:

      - name: enable_request_tracer
        type: bool
        title: Enable request tracing
        default: false
        multi: false
        required: false
        show_user: false
        description: The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_enable) for details.

Alternatively, keep the guard in the template so an unset var cannot enable tracing:

{{#if enable_request_tracer}}
request.tracer:
  enabled: true
  filename: "../../logs/cel/http-request-trace-*.ndjson"
  maxbackups: 5
{{/if}}

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.


- version: "1.10.0"
changes:
- description: PLACEHOLDER - specified once all related PRs will be merged to branch ti_set_polakovicp_enhancements

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟠 High confidence: high path: packages/ti_eset/changelog.yml:5

The 1.10.0 changelog entry is a literal PLACEHOLDER with link: TODO, which is not a valid URL — replace it with real change descriptions and the PR link before merge.

Details

packages/ti_eset/changelog.yml declares version 1.10.0 with description: PLACEHOLDER - specified once all related PRs will be merged to branch ti_set_polakovicp_enhancements and link: TODO. link must be a URL (every other entry in this file points at a github.com/elastic/integrations PR), and the description is what users see in Fleet's package changelog. The entry also does not mention any of the changes actually in this PR: the new smsscam data stream and its transform, the migration of all seven existing data streams from the httpjson template to run_as_cel, the Kibana constraint bump, and the relocation of the username/password vars.

Recommendation:

Replace the placeholder with the real entries before merge:

- version: "1.10.0"
  changes:
    - description: Add SMS scam data stream.
      type: enhancement
      link: https://github.com/elastic/integrations/pull/19127
    - description: Migrate the apt, botnet, cc, domains, files, ip and url data streams to the CEL execution engine (run_as_cel).
      type: enhancement
      link: https://github.com/elastic/integrations/pull/19127

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

delay: 120s
field: event.ingested
settings:
num_failure_retries: -1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟠 High confidence: medium path: packages/ti_eset/elasticsearch/transform/smsscam_latest_ioc/transform.yml:28

The new smsscam transform sets num_failure_retries: -1 but drops unattended: true, which all seven sibling transforms have — add unattended: true so it still starts when the source index does not yet exist.

Details

All seven existing transforms in this package (apt, botnet, cc, domains, files, ip, url _latest_ioc/transform.yml) end with settings:\n unattended: true, added deliberately in 1.9.0 ("Allow transforms to run in unattended mode"). The new smsscam transform replaces that with num_failure_retries: -1. Without unattended mode a transform that cannot resolve its source index fails and stops instead of retrying. That matters here specifically: start: true is set in the transform manifest, while the smsscam stream is shipped with enabled: false (see finding 5), so logs-ti_eset.smsscam-* will not exist on a default install and the transform has nothing to recover from.

Recommendation:

Match the sibling transforms and keep the retry setting if it is wanted:

settings:
  unattended: true
  num_failure_retries: -1

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Comment thread packages/ti_eset/manifest.yml Outdated
conditions:
kibana:
version: "^8.19.4 || ~9.0.7 || ^9.1.4"
version: "^9.4.0"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: medium path: packages/ti_eset/manifest.yml:12

This bump drops every 8.x and 9.0-9.3 stack and moves username/password from the input to package level, both of which are breaking for existing installs — record them as type: breaking-change in the changelog.

Details

Two user-visible breaking changes land here but the 1.10.0 changelog entry is typed enhancement. (1) conditions.kibana.version narrows from ^8.19.4 || ~9.0.7 || ^9.1.4 to ^9.4.0, so every stack below 9.4 stops receiving updates for this package. The floor itself is justified — run_as_cel is absent from the httpjson input on the 8.19 and 9.3 branches of elastic/beats, so the migrated templates genuinely need a newer agent — but the compatibility drop still needs to be declared. (2) The username and password vars move out of the httpjson input in policy_templates up to package-level vars. Var identity in Fleet is scoped to where it is declared, so existing policies carrying these values at input level do not carry over to the new package-level location on upgrade.

Recommendation:

Declare both in the changelog so Fleet surfaces them at upgrade time:

- version: "1.10.0"
  changes:
    - description: Raise the minimum Kibana version to 9.4.0; the data streams now run on the CEL execution engine, which is not available on earlier stacks.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/19127
    - description: Move the ETI username and password variables from the httpjson input to the package level; these credentials must be re-entered after upgrading.
      type: breaking-change
      link: https://github.com/elastic/integrations/pull/19127

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

ilm_policy: logs-ti_eset.smsscam-default_policy
streams:
- input: cel
enabled: false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🟡 Medium confidence: low path: packages/ti_eset/data_stream/smsscam/manifest.yml:6

The smsscam stream is the only one in the package shipped with enabled: false, so the new feed collects nothing until a user finds and turns it on — drop the key unless the default-off behaviour is deliberate.

Details

None of the seven existing data stream manifests in packages/ti_eset/data_stream/*/manifest.yml set enabled, so they all default to enabled. The new smsscam stream sets enabled: false. The README, the ILM/expiration tables and the transform all present smsscam as a first-class feed alongside the others, so the disabled default is inconsistent with how the rest of the package (and this PR's own docs) describe it. It also means the smsscam_latest_ioc transform, which has start: true, has no source index on a default install.

Recommendation:

Remove the key so the stream matches its siblings:

title: SMS scam
type: logs
ilm_policy: logs-ti_eset.smsscam-default_policy
streams:
  - input: cel
    title: SMS scam
    description: Collect data from ETI SMS scam feed
    template_path: cel.yml.hbs

If the feed genuinely requires a separate ETI entitlement and should stay opt-in, say so in the stream description so the reason is visible in Fleet.


🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

}
}
}
} No newline at end of file

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity: 🔵 Low confidence: medium path: packages/ti_eset/data_stream/smsscam/elasticsearch/ilm/default_policy.json:23

Three files added here end without a trailing newline, which elastic-package format rewrites and the repo's source-format CI check then fails on — run elastic-package format in the package directory.

Details

The diff reports \ No newline at end of file for packages/ti_eset/data_stream/smsscam/elasticsearch/ilm/default_policy.json, packages/ti_eset/data_stream/smsscam/fields/agent.yml, and packages/ti_eset/_dev/deploy/docker/files/config.yml. The equivalent existing file (packages/ti_eset/data_stream/url/elasticsearch/ilm/default_policy.json) has the same content, and .buildkite/scripts/check_sources.sh fails the build when elastic-package format produces a diff.

Recommendation:

Run the formatter from the package root and commit the result:

cd packages/ti_eset
elastic-package format

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@mergify

mergify Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request
@vera-review-bot

Copy link
Copy Markdown

🟢 No issues across the latest commits 7655419.

Review summary

Issues found across earlier commits cd0c221 — 3 high, 2 medium, 1 low
  • 🟠 The new template interpolates enabled: {{enable_request_tracer}}, but these seven data streams never declare a default for that var, so the rendered enabled: is null and Beats treats a null enabled as ON — add default: false to each manifest (as smsscam already does). (link) (Unresolved)
  • 🟠 The 1.10.0 changelog entry is a literal PLACEHOLDER with link: TODO, which is not a valid URL — replace it with real change descriptions and the PR link before merge. (link) (Unresolved)
  • 🟠 The new smsscam transform sets num_failure_retries: -1 but drops unattended: true, which all seven sibling transforms have — add unattended: true so it still starts when the source index does not yet exist. (link) (Unresolved)
  • 🟡 This bump drops every 8.x and 9.0-9.3 stack and moves username/password from the input to package level, both of which are breaking for existing installs — record them as type: breaking-change in the changelog. (link) (Unresolved)
  • 🟡 The smsscam stream is the only one in the package shipped with enabled: false, so the new feed collects nothing until a user finds and turns it on — drop the key unless the default-off behaviour is deliberate. (link) (Unresolved)
  • 🔵 Three files added here end without a trailing newline, which elastic-package format rewrites and the repo's source-format CI check then fails on — run elastic-package format in the package directory. (link) (Unresolved)
Issues found across earlier commits ac41745f426381 (3 commits) — 1 high, 3 medium, 1 low
  • 🟠 Moving username/password from the httpjson input to package-level vars changes their scope, so credentials saved by existing 1.9.0 policies are not carried over on upgrade (link) (Unresolved)
  • 🟡 The 1.10.0 changelog entry ships a placeholder description and link: TODO (link) (Unresolved)
  • 🟡 The new smsscam data stream is absent from the README template, so it ships with no documentation, no fields table and no sample event (link) (Unresolved)
  • 🟡 The pipeline test fixture is a single indicator that exercises only one of the four confidence labels and never the non-indicator drop branch (link) (Unresolved)
  • 🔵 smsscam is the only data stream in the package declared enabled: false, so the new feed collects nothing until a user finds and enables it (link) (Unresolved)

Since this is a community PR, a new commit triggers another review — at most once every 30 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.2.7 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration:ti_eset ESET Threat Intelligence (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

2 participants