Skip to content

ti_*: Use num_failure_retries instead of unattended mode - #18404

Merged
kcreddy merged 1 commit into
elastic:mainfrom
kcreddy:ti_all-num_failure_retries
May 12, 2026
Merged

ti_*: Use num_failure_retries instead of unattended mode#18404
kcreddy merged 1 commit into
elastic:mainfrom
kcreddy:ti_all-num_failure_retries

Conversation

@kcreddy

@kcreddy kcreddy commented Apr 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

ti_*: use num_failure_retries instead of unattended mode

Replace settings.unattended: true with settings.num_failure_retries: -1
in all ti_* managed transforms. Unlike unattended mode which retries
all failures indefinitely (masking irrecoverable errors),
num_failure_retries: -1 retries only recoverable failures while still
surfacing genuinely irrecoverable ones to users.

Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added
after the original unattended PR (#16535) had no failure resilience at
all and now get num_failure_retries: -1 added.

[git-generate]
for transform in $(find packages/ti_*/ -type f -name transform.yml \
  -path '*/elasticsearch/transform/*'); do
    yq -i 'del(.settings.unattended)' "$transform"
    yq -i '.settings.num_failure_retries = -1' "$transform"
done

for transform in $(git diff --name-only packages/ | \
  grep 'transform\.yml$'); do
    current=$(yq '._meta.fleet_transform_version' "$transform")
    next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}')
    yq -i "._meta.fleet_transform_version = \"$next\"" "$transform"
done

for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \
  sort -u); do
    cd "$pkg"
    elastic-package changelog add \
      --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \
      --type enhancement --next minor \
      --link "https://github.com/elastic/integrations/pull/18404"
    cd ../../
done

Summary

Switches all ti_* managed transforms from settings.unattended: true to settings.num_failure_retries: -1.

  • unattended: true retries all failures indefinitely, including irrecoverable ones, which masks real problems from users.
  • num_failure_retries: -1 retries only recoverable failures indefinitely (network blips, transient cluster instability) while still surfacing irrecoverable errors.

This covers 52 transforms across 23 packages. Three packages (ti_anyrun, ti_flashpoint, ti_strider) were added after the original unattended PR (#16535) and had no failure resilience at all -- they now get num_failure_retries: -1 for the first time.

Changes per package

For each affected transform:

  1. Removed settings.unattended: true
  2. Added settings.num_failure_retries: -1
  3. Bumped _meta.fleet_transform_version (minor bump triggers reinstall)
  4. Added a changelog entry

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues
@kcreddy
kcreddy requested a review from a team as a code owner April 15, 2026 06:47
@kcreddy
kcreddy marked this pull request as draft April 15, 2026 06:47
@kcreddy kcreddy self-assigned this Apr 15, 2026
@kcreddy kcreddy added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Apr 15, 2026
kcreddy added a commit to kcreddy/integrations that referenced this pull request Apr 15, 2026
@kcreddy
ti_*: use num_failure_retries instead of unattended mode
f441298 
Replace settings.unattended: true with settings.num_failure_retries: -1
in all ti_* managed transforms. Unlike unattended mode which retries
all failures indefinitely (masking irrecoverable errors),
num_failure_retries: -1 retries only recoverable failures while still
surfacing genuinely irrecoverable ones to users.

Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added
after the original unattended PR (elastic#16535) had no failure resilience at
all and now get num_failure_retries: -1 added.

Requires elastic/package-spec#1124 (add num_failure_retries to the
transform settings schema).

[git-generate]
for transform in $(find packages/ti_*/ -type f -name transform.yml \
  -path '*/elasticsearch/transform/*'); do
    yq -i 'del(.settings.unattended)' "$transform"
    yq -i '.settings.num_failure_retries = -1' "$transform"
done

for transform in $(git diff --name-only packages/ | \
  grep 'transform\.yml$'); do
    current=$(yq '._meta.fleet_transform_version' "$transform")
    next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}')
    yq -i "._meta.fleet_transform_version = \"$next\"" "$transform"
done

for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \
  sort -u); do
    cd "$pkg"
    elastic-package changelog add \
      --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \
      --type enhancement --next minor \
      --link "elastic#18404"
    cd ../../
done

Made-with: Cursor
@kcreddy
kcreddy force-pushed the ti_all-num_failure_retries branch from 2fc72f0 to f441298 Compare April 15, 2026 06:53
@andrewkroh andrewkroh added Integration:ti_recordedfuture Recorded Future Integration:ti_rapid7_threat_command Rapid7 Threat Command (Partner supported) Integration:ti_domaintools DomainTools Feeds (Partner supported) Integration:ti_otx AlienVault OTX Integration:ti_strider Strider Shield (Partner supported) Integration:ti_anyrun ANY.RUN Threat Intelligence Feeds (Partner supported) Integration:ti_misp MISP Integration:ti_opencti OpenCTI Integration:ti_cybersixgill Cybersixgill Integration:ti_anomali Anomali ThreatStream Integration:ti_maltiverse Maltiverse (Partner supported) Integration:ti_eset ESET Threat Intelligence (Partner supported) Integration:ti_eclecticiq EclecticIQ (Partner supported) Integration:ti_abusech abuse.ch Integration:ti_greynoise GreyNoise (Community supported) Integration:ti_google_threat_intelligence Google Threat Intelligence (Partner supported) Integration:ti_threatq ThreatQuotient (Partner supported) Integration:ti_crowdstrike CrowdStrike Falcon Intelligence Integration:ti_custom Custom Threat Intelligence Integration:ti_threatconnect ThreatConnect Integration:ti_cif3 Collective Intelligence Framework v3 (Community supported) Integration:ti_flashpoint Flashpoint Integration:ti_cyware_intel_exchange Cyware Intel Exchange labels Apr 15, 2026
@kcreddy
kcreddy marked this pull request as ready for review May 11, 2026 17:47
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented May 11, 2026

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
efd6
efd6 approved these changes May 12, 2026
View reviewed changes

@efd6 efd6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that for

for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \
  sort -u); do
    cd "$pkg"
    elastic-package changelog add \
      --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \
      --type enhancement --next minor \
      --link "https://github.com/elastic/integrations/pull/18404"
    cd ../../
done

you can do this with subshell to avoid the need to move back to the root.

for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \
  sort -u); do
  (
    cd "$pkg"
    elastic-package changelog add \
      --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \
      --type enhancement --next minor \
      --link "https://github.com/elastic/integrations/pull/18404"
  )
done
@kcreddy
kcreddy merged commit eb9375c into elastic:main May 12, 2026
8 checks passed
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_abusech - 3.7.0 containing this change is available at https://epr.elastic.co/package/ti_abusech/3.7.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_anomali - 2.7.0 containing this change is available at https://epr.elastic.co/package/ti_anomali/2.7.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_anyrun - 1.1.0 containing this change is available at https://epr.elastic.co/package/ti_anyrun/1.1.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_cif3 - 1.19.0 containing this change is available at https://epr.elastic.co/package/ti_cif3/1.19.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_crowdstrike - 2.8.0 containing this change is available at https://epr.elastic.co/package/ti_crowdstrike/2.8.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_custom - 1.7.0 containing this change is available at https://epr.elastic.co/package/ti_custom/1.7.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_cybersixgill - 1.35.0 containing this change is available at https://epr.elastic.co/package/ti_cybersixgill/1.35.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_cyware_intel_exchange - 0.3.0 containing this change is available at https://epr.elastic.co/package/ti_cyware_intel_exchange/0.3.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_domaintools - 1.4.0 containing this change is available at https://epr.elastic.co/package/ti_domaintools/1.4.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_eclecticiq - 1.6.0 containing this change is available at https://epr.elastic.co/package/ti_eclecticiq/1.6.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_eset - 1.10.0 containing this change is available at https://epr.elastic.co/package/ti_eset/1.10.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_flashpoint - 0.2.0 containing this change is available at https://epr.elastic.co/package/ti_flashpoint/0.2.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_google_threat_intelligence - 0.9.0 containing this change is available at https://epr.elastic.co/package/ti_google_threat_intelligence/0.9.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_greynoise - 0.8.0 containing this change is available at https://epr.elastic.co/package/ti_greynoise/0.8.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_maltiverse - 1.7.0 containing this change is available at https://epr.elastic.co/package/ti_maltiverse/1.7.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_misp - 1.43.0 containing this change is available at https://epr.elastic.co/package/ti_misp/1.43.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_opencti - 2.14.0 containing this change is available at https://epr.elastic.co/package/ti_opencti/2.14.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_otx - 1.31.0 containing this change is available at https://epr.elastic.co/package/ti_otx/1.31.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_rapid7_threat_command - 2.9.0 containing this change is available at https://epr.elastic.co/package/ti_rapid7_threat_command/2.9.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_recordedfuture - 2.5.0 containing this change is available at https://epr.elastic.co/package/ti_recordedfuture/2.5.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_strider - 0.1.0 containing this change is available at https://epr.elastic.co/package/ti_strider/0.1.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_threatconnect - 2.1.0 containing this change is available at https://epr.elastic.co/package/ti_threatconnect/2.1.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

Package ti_threatq - 1.38.0 containing this change is available at https://epr.elastic.co/package/ti_threatq/1.38.0/
@github-actions github-actions Bot mentioned this pull request May 12, 2026
Open
herrBez pushed a commit to herrBez/integrations that referenced this pull request Jun 1, 2026
@kcreddy @herrBez
ti_*: use num_failure_retries instead of unattended mode (elastic#18404)
7d4209e 
Replace settings.unattended: true with settings.num_failure_retries: -1
in all ti_* managed transforms. Unlike unattended mode which retries
all failures indefinitely (masking irrecoverable errors),
num_failure_retries: -1 retries only recoverable failures while still
surfacing genuinely irrecoverable ones to users.

Three packages (ti_anyrun, ti_flashpoint, ti_strider) that were added
after the original unattended PR (elastic#16535) had no failure resilience at
all and now get num_failure_retries: -1 added.

[git-generate]
for transform in $(find packages/ti_*/ -type f -name transform.yml \
  -path '*/elasticsearch/transform/*'); do
    yq -i 'del(.settings.unattended)' "$transform"
    yq -i '.settings.num_failure_retries = -1' "$transform"
done

for transform in $(git diff --name-only packages/ | \
  grep 'transform\.yml$'); do
    current=$(yq '._meta.fleet_transform_version' "$transform")
    next=$(echo "$current" | awk -F. '{printf "%d.%d.%d",$1,$2+1,0}')
    yq -i "._meta.fleet_transform_version = \"$next\"" "$transform"
done

for pkg in $(git diff --name-only packages/ | cut -d/ -f1,2 | \
  sort -u); do
    cd "$pkg"
    elastic-package changelog add \
      --description "Use num_failure_retries instead of unattended mode for transform failure recovery." \
      --type enhancement --next minor \
      --link "elastic#18404"
    cd ../../
done
@github-actions github-actions Bot mentioned this pull request Jun 10, 2026
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:ti_abusech abuse.ch Integration:ti_anomali Anomali ThreatStream Integration:ti_anyrun ANY.RUN Threat Intelligence Feeds (Partner supported) Integration:ti_cif3 Collective Intelligence Framework v3 (Community supported) Integration:ti_crowdstrike CrowdStrike Falcon Intelligence Integration:ti_custom Custom Threat Intelligence Integration:ti_cybersixgill Cybersixgill Integration:ti_cyware_intel_exchange Cyware Intel Exchange Integration:ti_domaintools DomainTools Feeds (Partner supported) Integration:ti_eclecticiq EclecticIQ (Partner supported) Integration:ti_eset ESET Threat Intelligence (Partner supported) Integration:ti_flashpoint Flashpoint Integration:ti_google_threat_intelligence Google Threat Intelligence (Partner supported) Integration:ti_greynoise GreyNoise (Community supported) Integration:ti_maltiverse Maltiverse (Partner supported) Integration:ti_misp MISP Integration:ti_opencti OpenCTI Integration:ti_otx AlienVault OTX Integration:ti_rapid7_threat_command Rapid7 Threat Command (Partner supported) Integration:ti_recordedfuture Recorded Future Integration:ti_strider Strider Shield (Partner supported) Integration:ti_threatconnect ThreatConnect Integration:ti_threatq ThreatQuotient (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

4 participants

@kcreddy @elasticmachine @efd6 @andrewkroh