Skip to content

Add new gap fill status for rules#242595

Merged
nkhristinin merged 60 commits intoelastic:mainfrom
nkhristinin:rules-with-gap-status
Dec 8, 2025
Merged

Add new gap fill status for rules#242595
nkhristinin merged 60 commits intoelastic:mainfrom
nkhristinin:rules-with-gap-status

Conversation

@nkhristinin
Copy link
Contributor

@nkhristinin nkhristinin commented Nov 11, 2025 •
edited
Loading

Add new gap fill status for rules

Short summary

  • Add a new “Gap fill status” column to the Rules table showing the gap fill status.
  • Remove the “Show only rules with gaps” button and replace it with a Gap fill status filter on the Rules table.
  • Remove the gaps date picker. The Rules table now defaults to a 90-day window.
Screenshot 2025-11-20 at 09 34 29

What is gap fill status?

Gap fill status summarises all gaps fill for a rule within a time window using the following precedence:

unfilled > in_progress > filled

  • If any unfilled gap exists, the rule is marked unfilled.
  • Otherwise, if any gap is in progress, it is in_progress.
  • Otherwise, the rule is filled.

We plan to add an “error” status in the future for cases where a gap fill attempt failed.

API changes

Find rules : GET /api/detection_engine/rules/_find

  • Added optional gap_fill_statuses query param (unfilled | in_progress | filled).
  • Gap filtering now occurs only when all of the following are provided:
    gap_fill_statuses, gaps_range_start, gaps_range_end.

Get rule IDs with gaps: POST /internal/alerting/rules/gaps/_get_rules

  • Request now supports highest_priority_gap_fill_statuses to filter by per-rule gap fill status.

Get gaps summary by rule IDs:

POST /internal/alerting/rules/gaps/_get_gaps_summary_by_rule_ids

  • Response now includes gap_fill_status (gap fill status) per rule.

How to test

1. Ensure you have rules with gaps

There are two ways to create gaps:

Manual method

  1. Create and enable a security rule with a 1-minute interval and 0-second lookback.
  2. After the first run, disable the rule, wait 5 minutes, then enable it again.
  3. You should see an execution error about gaps, and the gap should appear in the gaps table on the Execution tab.

Using the tool

Run the following command to generate multiple rules and gaps (100 rules, 10 gaps each, 30-minute interval, remove all rules before):

npm run start -- rules --rules 100 -c -g 10 -i "30m"

2. UI

  • Open the Rules table: confirm there is no “Show rules with gaps” button and no gaps date picker.
  • Confirm a new “Gap fill status” column is visible.
  • Use the Gap fill status filter to show rules by unfilled, in_progress, or filled.
  • Verify results reflect the last 90 days by default.
  • Bulk-fill gaps and confirm statuses transition in_progress → filled.
@nkhristinin
Copy link
Contributor Author

nkhristinin commented Nov 11, 2025

/ci
@nkhristinin
Copy link
Contributor Author

nkhristinin commented Nov 11, 2025

/ci
@nkhristinin
Copy link
Contributor Author

nkhristinin commented Nov 12, 2025

/ci
@nkhristinin
Copy link
Contributor Author

nkhristinin commented Nov 12, 2025

/ci
@nkhristinin
Copy link
Contributor Author

nkhristinin commented Nov 13, 2025

/ci
@nkhristinin nkhristinin changed the title Add aggregated gap status Nov 13, 2025
@nkhristinin nkhristinin self-assigned this Nov 13, 2025
@nkhristinin nkhristinin added release_note:enhancement backport:skip This PR does not require backporting labels Nov 13, 2025
@nkhristinin nkhristinin marked this pull request as ready for review November 13, 2025 10:07
@nkhristinin nkhristinin requested review from a team as code owners November 13, 2025 10:07
@nkhristinin nkhristinin requested a review from maximpn December 3, 2025 10:43
@nkhristinin
Copy link
Contributor Author

nkhristinin commented Dec 3, 2025

Hey, @maximpn, @approksiu

I make gap fill status filter available only on rule monitoring tab. With those changes I removed ability to store this filter in URL, so when page reload or we leave rule monitoring tab - gap fill status will be reseted
@nkhristinin nkhristinin requested a review from ymao1 December 3, 2025 13:37
@nkhristinin
Copy link
Contributor Author

nkhristinin commented Dec 3, 2025

@maximpn

Fill gaps button on the rule's detail page doesn't fill all the gaps. It requires manual time range adjustment. Is it expected behavior?

yes, its how it works right now
@nkhristinin
Copy link
Contributor Author

nkhristinin commented Dec 4, 2025

@elasticmachine merge upstream
maximpn
maximpn approved these changes Dec 4, 2025
View reviewed changes
Copy link
Contributor

@maximpn maximpn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nkhristinin Thanks for addressing my critical comments and removing Gap fill status filter from the Installed Rules table 🙏

Some of my comments are still relevant. For example total number of rules with gaps number is flickering upon loading. Most probably we should have a task to track them.
@elasticmachine
Copy link
Contributor

elasticmachine commented Dec 4, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 8462 8463 +1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
alerting 827 828 +1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 11.1MB 11.1MB +3.4KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
alerting 23.9KB 24.0KB +89.0B
Unknown metric groups

API count

id before after diff
alerting 864 867 +3

History

cc @nkhristinin
ymao1
ymao1 approved these changes Dec 4, 2025
View reviewed changes
Copy link
Contributor

@ymao1 ymao1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Response ops changes LGTM. Code review only
@nkhristinin nkhristinin changed the title Add new aggregated gap status for rules Dec 5, 2025
@nkhristinin
Copy link
Contributor Author

nkhristinin commented Dec 8, 2025

@elasticmachine merge upstream
@nkhristinin nkhristinin merged commit 2bb238c into elastic:main Dec 8, 2025
12 checks passed
mbondyra added a commit to mbondyra/kibana that referenced this pull request Dec 8, 2025
@mbondyra
Merge commit '6647f813c9fa03ac0378e3d4756246e8dc4b4c76' into lens_cps
60dea28 
* commit '6647f813c9fa03ac0378e3d4756246e8dc4b4c76': (33 commits)
  [Detection Engine] Extracts Rules/Alerts/Exceptions permission to new Rules feature privileges (elastic#239634)
  [Agent Builder] Add Intro Tour (elastic#245551)
  Add datastream lifecycle support to indices metadata (elastic#245548)
  [Serverless] Update preconfigured connectors (elastic#245445)
  [Metrics][Discover] Discover to prefer line chars for time series data (elastic#244595)
  Update dependency @elastic/ebt to ^1.4.1 (main) (elastic#241629)
  [One Workflow] fix: request bodies with oneof schemas (`kibana.SetAlertsStatus`, etc) (elastic#245344)
  Update dependency ai to v5 (elastic#244675)
  Fix Discover trace waterfall behavior with duplicate spans (elastic#244984)
  [FSH] Migrated fs usage to kbn/fs for sample ingest (elastic#244163)
  Streamlang: Unskip type coercion test (elastic#245519)
  [Response Ops][Reporting] Fixing error in calculating delay value between retries (elastic#245431)
  Add TopNavMenuBeta to navigation plugin (elastic#243578)
  [scout] support custom servers configuration (elastic#244306)
  [Fleet] Run agentless background sync without dry run (elastic#245286)
  Fix Change Password Flaky Test (elastic#245443)
  Add new gap fill status for rules (elastic#242595)
  [Kibana Search] Move SLOs higher up in search results (elastic#245518)
  feat(slo): introduce find SLO instances internal route (elastic#245333)
  [FSH] Dropped unnecessary `fs` persistence for synthetics project code (elastic#244338)
  ...
@nkhristinin nkhristinin added the Team:Detection Engine Security Solution Detection Engine Area label Jan 12, 2026
@elasticmachine
Copy link
Contributor

elasticmachine commented Jan 12, 2026

Pinging @elastic/security-detection-engine (Team:Detection Engine)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:enhancement Team:Detection Engine Security Solution Detection Engine Area v9.3.0

9 participants

@nkhristinin @banderror @elasticmachine @rylnd @maximpn @ymao1 @approksiu @nastasha-solomon @kibanamachine