[ti_eset] Add new feeds

[polakovicp]

Proposed commit message

PR add additional data streams to ti_eset integration: androidthreats, androidinfostealer, cryptoscam, emailattachments, phishingurl, puaadware, puadualapps, scamurl, smishing, smsscam, ransomware

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

[elastic-vault-github-plugin-prod]

Reviewers

Buildkite won't run for external contributors automatically; you need to add a comment:

• /test : will kick off a build in Buildkite.

NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details.

[macroscopeapp]

🟡 Medium ingest_pipeline/default.yml:93

The foreach processor on eti._patterns at line 93 is missing an if condition. The preceding split processor has ignore_missing: true, so when eti.pattern is null, eti._patterns is never created. This causes the foreach to fail with a missing-field error, triggering on_failure and incorrectly marking the event as a pipeline error. Consider adding if: ctx.eti?._patterns != null to the foreach processor, matching the pattern used in the cc data stream.

- foreach:
+ foreach:
     field: eti._patterns
     processor:
       grok:
         field: _ingest._value
         patterns:
         - "^\\[?file:hashes.'MD5'%{SPACE}=%{SPACE}'%{DATA:threat.indicator.file.hash.md5}'\\]?"
         - "^\\[?file:hashes.'SHA-1'%{SPACE}=%{SPACE}'%{DATA:threat.indicator.file.hash.sha1}'\\]?"
         - "^\\[?file:hashes.'SHA-256'%{SPACE}=%{SPACE}'%{DATA:threat.indicator.file.hash.sha256}'\\]?"
         ignore_failure: true
+    if: ctx.eti?._patterns != null

Also found in 7 other location(s)

packages/ti_eset/data_stream/smishing/elasticsearch/ingest_pipeline/default.yml:93

The foreach processor at line 93 operates on eti._patterns without an if guard or ignore_missing: true. If eti.pattern is missing or null on an incoming document, the split at line 88 silently skips (due to ignore_missing: true), leaving eti._patterns undefined. The foreach processor then fails because the field doesn't exist, sending the document to the on_failure handler and incorrectly marking it as a pipeline error. Other pipelines in the same integration (e.g., cc, domains) correctly guard this with if: "ctx.eti?._patterns != null" on the foreach. This pipeline should add the same guard.

packages/ti_eset/data_stream/scamurl/elasticsearch/ingest_pipeline/default.yml:93

The foreach processor at line 93 iterates over eti._patterns but does not specify ignore_missing: true. The split processor at line 88 that creates eti._patterns has ignore_missing: true, meaning if eti.pattern is absent, eti._patterns will not be created. When the foreach processor then tries to iterate over the non-existent eti._patterns field, it will throw an error and trigger the on_failure handler, marking the document as a pipeline error. Adding ignore_missing: true to the foreach processor would fix this.

packages/ti_eset/data_stream/ransomware/elasticsearch/ingest_pipeline/default.yml:93

The foreach processor at line 93 iterating over eti._patterns is missing an if guard (e.g., if: ctx.eti?._patterns != null). The preceding split processor at line 88 uses ignore_missing: true, so if eti.pattern is absent, eti._patterns will not be created. The foreach processor will then fail because the field does not exist, sending the document to the on_failure handler. Other existing pipelines in this integration (e.g., cc, domains) correctly include if: "ctx.eti?._patterns != null" on their equivalent foreach processor. The fix is to add if: ctx.eti?._patterns != null to the foreach processor.

packages/ti_eset/data_stream/androidthreats/elasticsearch/ingest_pipeline/default.yml:93

The foreach processor at line 93 iterates over eti._patterns but does not have ignore_missing: true. If eti.pattern is missing/null, the split processor at line 88 (which has ignore_missing: true) will silently skip without creating eti._patterns. The subsequent foreach will then fail because the field does not exist, causing the pipeline's on_failure handler to fire. Add ignore_missing: true to the foreach at line 93.

packages/ti_eset/data_stream/puadualapps/elasticsearch/ingest_pipeline/default.yml:93

The foreach processor at line 93 iterating over eti._patterns does not have ignore_missing: true. If eti.pattern is missing from the document, the split processor at line 88 will be a no-op (due to its own ignore_missing: true), meaning eti._patterns will not be created. The subsequent foreach on the non-existent eti._patterns field will then fail, sending the document to the on_failure handler. An ignore_missing: true should be added to the foreach at line 93.

packages/ti_eset/data_stream/puaadware/elasticsearch/ingest_pipeline/default.yml:93

The foreach processor at line 93 iterating over eti._patterns has no if condition and no ignore_missing: true. If the eti.pattern field is missing from a document (the split processor at line 88 uses ignore_missing: true so it silently skips), eti._patterns will not exist. The foreach processor will then fail because the field it is trying to iterate is missing, triggering the pipeline's on_failure handler. An if: ctx.eti?._patterns != null condition or ignore_missing: true should be added to the foreach.

packages/ti_eset/data_stream/emailattachments/elasticsearch/ingest_pipeline/default.yml:93

The foreach processor at line 93 iterating over eti._patterns does not specify ignore_missing: true. The split processor at line 88 that creates eti._patterns has ignore_missing: true, meaning if eti.pattern is missing, eti._patterns will not be created. This causes the foreach to fail with an error for any document where eti.pattern is absent, sending it to the on_failure handler. Adding ignore_missing: true to the foreach or adding an if condition would fix this.

🤖 Copy this AI Prompt to have your agent fix this:

In file packages/ti_eset/data_stream/androidinfostealer/elasticsearch/ingest_pipeline/default.yml around lines 93-103:

The `foreach` processor on `eti._patterns` at line 93 is missing an `if` condition. The preceding `split` processor has `ignore_missing: true`, so when `eti.pattern` is null, `eti._patterns` is never created. This causes the `foreach` to fail with a missing-field error, triggering `on_failure` and incorrectly marking the event as a pipeline error. Consider adding `if: ctx.eti?._patterns != null` to the `foreach` processor, matching the pattern used in the `cc` data stream.

Also found in 7 other location(s):
- packages/ti_eset/data_stream/smishing/elasticsearch/ingest_pipeline/default.yml:93 -- The `foreach` processor at line 93 operates on `eti._patterns` without an `if` guard or `ignore_missing: true`. If `eti.pattern` is missing or null on an incoming document, the `split` at line 88 silently skips (due to `ignore_missing: true`), leaving `eti._patterns` undefined. The `foreach` processor then fails because the field doesn't exist, sending the document to the `on_failure` handler and incorrectly marking it as a pipeline error. Other pipelines in the same integration (e.g., `cc`, `domains`) correctly guard this with `if: "ctx.eti?._patterns != null"` on the `foreach`. This pipeline should add the same guard.
- packages/ti_eset/data_stream/scamurl/elasticsearch/ingest_pipeline/default.yml:93 -- The `foreach` processor at line 93 iterates over `eti._patterns` but does not specify `ignore_missing: true`. The `split` processor at line 88 that creates `eti._patterns` has `ignore_missing: true`, meaning if `eti.pattern` is absent, `eti._patterns` will not be created. When the `foreach` processor then tries to iterate over the non-existent `eti._patterns` field, it will throw an error and trigger the `on_failure` handler, marking the document as a pipeline error. Adding `ignore_missing: true` to the `foreach` processor would fix this.
- packages/ti_eset/data_stream/ransomware/elasticsearch/ingest_pipeline/default.yml:93 -- The `foreach` processor at line 93 iterating over `eti._patterns` is missing an `if` guard (e.g., `if: ctx.eti?._patterns != null`). The preceding `split` processor at line 88 uses `ignore_missing: true`, so if `eti.pattern` is absent, `eti._patterns` will not be created. The `foreach` processor will then fail because the field does not exist, sending the document to the `on_failure` handler. Other existing pipelines in this integration (e.g., `cc`, `domains`) correctly include `if: "ctx.eti?._patterns != null"` on their equivalent `foreach` processor. The fix is to add `if: ctx.eti?._patterns != null` to the `foreach` processor.
- packages/ti_eset/data_stream/androidthreats/elasticsearch/ingest_pipeline/default.yml:93 -- The `foreach` processor at line 93 iterates over `eti._patterns` but does not have `ignore_missing: true`. If `eti.pattern` is missing/null, the `split` processor at line 88 (which has `ignore_missing: true`) will silently skip without creating `eti._patterns`. The subsequent `foreach` will then fail because the field does not exist, causing the pipeline's `on_failure` handler to fire. Add `ignore_missing: true` to the `foreach` at line 93.
- packages/ti_eset/data_stream/puadualapps/elasticsearch/ingest_pipeline/default.yml:93 -- The `foreach` processor at line 93 iterating over `eti._patterns` does not have `ignore_missing: true`. If `eti.pattern` is missing from the document, the `split` processor at line 88 will be a no-op (due to its own `ignore_missing: true`), meaning `eti._patterns` will not be created. The subsequent `foreach` on the non-existent `eti._patterns` field will then fail, sending the document to the `on_failure` handler. An `ignore_missing: true` should be added to the `foreach` at line 93.
- packages/ti_eset/data_stream/puaadware/elasticsearch/ingest_pipeline/default.yml:93 -- The `foreach` processor at line 93 iterating over `eti._patterns` has no `if` condition and no `ignore_missing: true`. If the `eti.pattern` field is missing from a document (the `split` processor at line 88 uses `ignore_missing: true` so it silently skips), `eti._patterns` will not exist. The `foreach` processor will then fail because the field it is trying to iterate is missing, triggering the pipeline's `on_failure` handler. An `if: ctx.eti?._patterns != null` condition or `ignore_missing: true` should be added to the `foreach`.
- packages/ti_eset/data_stream/emailattachments/elasticsearch/ingest_pipeline/default.yml:93 -- The `foreach` processor at line 93 iterating over `eti._patterns` does not specify `ignore_missing: true`. The `split` processor at line 88 that creates `eti._patterns` has `ignore_missing: true`, meaning if `eti.pattern` is missing, `eti._patterns` will not be created. This causes the `foreach` to fail with an error for any document where `eti.pattern` is absent, sending it to the `on_failure` handler. Adding `ignore_missing: true` to the `foreach` or adding an `if` condition would fix this.

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

This PR is too large to reliably review. It would be better if you could send this in a sequence of PRs, possibly to a long lived feature branch. If you would like to use the feature branch approach, I can create the target for you.

Also, I see that you are using HTTP JSON. The current agent templates use httpjson, but generally we will use cel when adding new integrations/data streams.

[polakovicp]

@efd6 These data streams parses IoC with URLs or files. Differences between them are only in one grok processor and naming.
If you still recommend splitting this PR into multiple, then please create feature branch for tracking.

Also, I see that you are using HTTP JSON. The current agent templates use httpjson, but generally we will use cel when adding new integrations/data streams.

Should this be done in this PR or we can implement this in future version? Do you have any example integration?
Thank you

[efd6]

I have created a ti_set_polakovicp_enhancements branch that you can target.

Should this be done in this PR or we can implement this in future version?

In these additions; migrating is harder than starting with CEL from the outset.

Do you have any example integration?

Here's a CEL input agent configuration that will work.

interval: {{interval}}
{{#if http_client_timeout}}
resource.timeout: {{http_client_timeout}}
{{/if}}
resource.url: {{url}}
{{#if ssl}}
resource.ssl: {{ssl}}
{{/if}}
{{#if proxy_url}}
resource.proxy_url: {{proxy_url}}
{{/if}}

resource.tracer:
  enabled: {{enable_request_tracer}}
  filename: "../../logs/cel/http-request-trace-*.ndjson"
  maxbackups: 5

state:
  url: {{url}}
  username: {{username}}
  password: {{password}}
  page_size: {{page_size}}
  initial_interval: {{initial_interval}}
  want_more: false
redact:
  fields:
    - username
    - password

program: |
  state.with({
    "added_after": state.?want_more.orValue(false) ?
      state.added_after
    :
      state.?cursor.last_timestamp.orValue(
        (now - duration(state.initial_interval)).format(time_layout.RFC3339)
      ),
  }).as(state, state.with(
    request(
      "GET",
      state.url.trim_right("/") + "/?" + {
        "match[type]": ["indicator"],
        "limit": [string(int(state.page_size))],
        "added_after": [state.added_after],
      }.format_query()
    ).with({
      "Header": {
        "Accept": ["application/taxii+json;version=2.1"],
        "Content-Type": ["application/taxii+json;version=2.1"],
        "Authorization": ["Basic " + (state.username + ":" + state.password).base64()],
      },
    }).do_request().as(resp, (resp.StatusCode == 200 || resp.StatusCode == 206) ?
      resp.Body.decode_json().as(body,
        body.?objects.orValue([]).as(objects, {
          "events": (size(objects) > 0) ?
            objects.map(e, {"message": dyn(e.encode_json())})
          :
            [{"retry": dyn(true)}],
          "added_after": resp.Header[?"X-Taxii-Date-Added-Last"][0].orValue(state.added_after),
          "want_more": body.?more.orValue(false) && size(objects) > 0,
          "cursor": {
            "last_timestamp": resp.Header[?"X-Taxii-Date-Added-Last"][0].orValue(state.added_after),
          },
        })
      )
    :
      {
        "events": {
          "error": {
            "code": string(resp.StatusCode),
            "id": string(resp.Status),
            "message": "GET " + state.url.trim_right("/") + ": " + (
              (size(resp.Body) != 0) ?
                string(resp.Body)
              :
                string(resp.Status) + " (" + string(resp.StatusCode) + ")"
            ),
          },
        },
        "want_more": false,
      }
    )
  ))

tags:
{{#if preserve_original_event}}
  - preserve_original_event
{{/if}}
{{#each tags as |tag|}}
  - {{tag}}
{{/each}}
{{#contains "forwarded" tags}}
publisher_pipeline.disable_host: true
{{/contains}}
processors:
  - drop_event.when.equals.retry: true
{{#if processors}}
{{processors}}
{{/if}}

You will need to make the input in the data stream be input: cel instead of input: httpjson and you will need to add an input to the root manifest.