Skip to content

Support rrule for task scheduling#217728

Merged
ersin-erdal merged 43 commits intoelastic:mainfrom
ersin-erdal:216308-support-rrule-for-task-scheduling
May 7, 2025
Merged

Support rrule for task scheduling#217728
ersin-erdal merged 43 commits intoelastic:mainfrom
ersin-erdal:216308-support-rrule-for-task-scheduling

Conversation

@ersin-erdal
Copy link
Contributor

@ersin-erdal ersin-erdal commented Apr 9, 2025 •
edited by kibanamachine
Loading

Resolves: #216308

This PR adds rrule notation support to the task scheduling.

Currently we use a simple interval string (such as 1h or 45m) to calculate the next runAt that we use to decide if the task should be picked up or not. With this PR, we will be able to use rrule as well.

We set the runAt field in 3 places:

  • At task creation: when the task is created for the first time, this is always now if a specific runAt is not provided.
  • At task update. We update runAt on task update as well but only when it is provided. We don't update it when the schedule config changes.
  • After task run: to decide when the next run will be after a task is run.

This PR modifies:

  • At task creation: It still uses the runAt when it is provided but calculates a runAt by using rrule if the config has a fixed time (like every day at 13:00) config. Becausethe task should not run immediately when there is a fixed time to run.
  • After task run: It calculates the next runAt with rrule when it is provided.

This PR doesn't apply any change on At task update as it requires a further investigation and consideration. We will handle it with a follow-on issue.

To verify:

You can use the test task sampleRecurringTask to test the feature.
Just create a task with rrule in the schedule config.
Then check the runAt field in the task SO to see if it is the expected datetime.
Then run the task one more time and check the new runAt

I also used the usage reporting task in the actions plugin to test the feature.
@ersin-erdal ersin-erdal added Team:ResponseOps Platform ResponseOps team (formerly the Cases and Alerting teams) t// release_note:feature Makes this part of the condensed release notes backport:version Backport to applied version labels v9.1.0 v8.19.0 labels Apr 9, 2025
ersin-erdal and others added 11 commits April 11, 2025 17:54
@ersin-erdal
add functional tests
20ecece
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/styled_components_m…
c29a08e 
…apping'
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
de20db1 
…-fix'
@ersin-erdal
fix linting
65f0960
@ersin-erdal
Merge branch '216308-support-rrule-for-task-scheduling' of github.com…
2dde5f6 
…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling
@ersin-erdal
fix linter
717cadb
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…
72b108e 
…te --fix'
@ersin-erdal
Merge branch 'main' into 216308-support-rrule-for-task-scheduling
38e69ef
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/jest_integration -u…
c9e71ba 
… src/core/server/integration_tests/ci_checks'
@ersin-erdal
Merge branch 'main' into 216308-support-rrule-for-task-scheduling
5f6fad9
@ersin-erdal
Merge branch 'main' into 216308-support-rrule-for-task-scheduling
4a04b90
ersin-erdal
ersin-erdal commented Apr 14, 2025
View reviewed changes
x-pack/platform/plugins/shared/task_manager/server/task_store.ts
if (
get(taskInstance, 'schedule.interval', null) == null &&
get(taskInstance, 'schedule.rrule', null) == null
) {
Copy link
Contributor Author

@ersin-erdal ersin-erdal Apr 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably checking for schedule is enough.
@ersin-erdal ersin-erdal marked this pull request as ready for review April 16, 2025 16:51
@ersin-erdal ersin-erdal requested review from a team as code owners April 16, 2025 16:51
@elasticmachine
Copy link
Contributor

elasticmachine commented Apr 16, 2025

Pinging @elastic/response-ops (Team:ResponseOps)
ymao1
ymao1 approved these changes May 5, 2025
View reviewed changes
Copy link
Contributor

@ymao1 ymao1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Left a question about having a fallback for Rrrule.after and a request to add a comment about the logic behind testing if an rrule has a fixed interval.
@elasticmachine
Copy link
Contributor

elasticmachine commented May 5, 2025

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
taskManager 64 67 +3

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id before after diff
taskManager 8 9 +1
Unknown metric groups

API count

id before after diff
taskManager 110 113 +3

History
@ersin-erdal ersin-erdal merged commit 0878a09 into elastic:main May 7, 2025
9 checks passed
@kibanamachine
Copy link
Contributor

kibanamachine commented May 7, 2025

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/14883566346
@kibanamachine
Copy link
Contributor

kibanamachine commented May 7, 2025

💔 All backports failed

Status Branch Result
❌ 8.19 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 217728

Questions ?

Please refer to the Backport tool documentation
This was referenced May 7, 2025
Merged
Merged
@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label May 9, 2025
@kibanamachine
Copy link
Contributor

kibanamachine commented May 9, 2025

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 217728 locally
cc: @ersin-erdal
@ersin-erdal ersin-erdal mentioned this pull request May 9, 2025
Merged
@ersin-erdal
Copy link
Contributor Author

ersin-erdal commented May 9, 2025

💚 All backports created successfully

Status Branch Result
✅ 8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation
ersin-erdal added a commit to ersin-erdal/kibana that referenced this pull request May 9, 2025
@ersin-erdal @kibanamachine
Support rrule for task scheduling (elastic#217728)
a5c086b 
Resolves: elastic#216308

This PR adds rrule notation support to the task scheduling.

Currently we use a simple interval string (such as `1h` or `45m`) to
calculate the next `runAt` that we use to decide if the task should be
picked up or not. With this PR, we will be able to use rrule as well.

We set the `runAt` field in 3 places:
- At task creation: when the task is created for the first time, this is
always `now` if a specific runAt is not provided.
- At task update. We update runAt on task update as well but only when
it is provided. We don't update it when the schedule config changes.
- After task run: to decide when the next run will be after a task is
run.

This PR modifies:
- At task creation: It still uses the `runAt` when it is provided but
calculates a runAt by using rrule if the config has a fixed time (like
every day at 13:00) config. Becausethe task should not run immediately
when there is a fixed time to run.
- After task run: It calculates the next `runAt` with rrule when it is
provided.

This PR doesn't apply any change on `At task update` as it requires a
further investigation and consideration. We will handle it with a
follow-on issue.
You can use the test task `sampleRecurringTask` to test the feature.
Just create a task with rrule in the schedule config.
Then check the runAt field in the task SO to see if it is the expected
datetime.
Then run the task one more time and check the new `runAt`

I also used the usage reporting task in the actions plugin to test the
feature.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 0878a09)
@kibanamachine
Copy link
Contributor

kibanamachine commented May 12, 2025

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @ersin-erdal
2 similar comments
@kibanamachine
Copy link
Contributor

kibanamachine commented May 13, 2025

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @ersin-erdal
@kibanamachine
Copy link
Contributor

kibanamachine commented May 14, 2025

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @ersin-erdal
ersin-erdal added a commit that referenced this pull request May 14, 2025
@ersin-erdal @kibanamachine
[8.19] Support rrule for task scheduling (#217728) (#220657)
aa17aeb 
# Backport

This will backport the following commits from `main` to `8.19`:
- [Support rrule for task scheduling
(#217728)](#217728)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Ersin
Erdal","email":"92688503+ersin-erdal@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-05-07T12:39:11Z","message":"Support
rrule for task scheduling (#217728)\n\nResolves:
https://github.com/elastic/kibana/issues/216308\n\nThis PR adds rrule
notation support to the task scheduling.\n\nCurrently we use a simple
interval string (such as `1h` or `45m`) to\ncalculate the next `runAt`
that we use to decide if the task should be\npicked up or not. With this
PR, we will be able to use rrule as well.\n\nWe set the `runAt` field in
3 places:\n- At task creation: when the task is created for the first
time, this is\nalways `now` if a specific runAt is not provided.\n- At
task update. We update runAt on task update as well but only when\nit is
provided. We don't update it when the schedule config changes.\n- After
task run: to decide when the next run will be after a task
is\nrun.\n\nThis PR modifies:\n- At task creation: It still uses the
`runAt` when it is provided but\ncalculates a runAt by using rrule if
the config has a fixed time (like\nevery day at 13:00) config.
Becausethe task should not run immediately\nwhen there is a fixed time
to run.\n- After task run: It calculates the next `runAt` with rrule
when it is\nprovided.\n\nThis PR doesn't apply any change on `At task
update` as it requires a\nfurther investigation and consideration. We
will handle it with a\nfollow-on issue.\n\n## To verify:\nYou can use
the test task `sampleRecurringTask` to test the feature.\nJust create a
task with rrule in the schedule config.\nThen check the runAt field in
the task SO to see if it is the expected\ndatetime.\nThen run the task
one more time and check the new `runAt`\n\nI also used the usage
reporting task in the actions plugin to test
the\nfeature.\n\n---------\n\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"0878a095a556b09bd05695469539c25be2c3423d","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:ResponseOps","backport
missing","release_note:feature","backport:version","v9.1.0","v8.19.0"],"title":"Support
rrule for task
scheduling","number":217728,"url":"https://github.com/elastic/kibana/pull/217728","mergeCommit":{"message":"Support
rrule for task scheduling (#217728)\n\nResolves:
https://github.com/elastic/kibana/issues/216308\n\nThis PR adds rrule
notation support to the task scheduling.\n\nCurrently we use a simple
interval string (such as `1h` or `45m`) to\ncalculate the next `runAt`
that we use to decide if the task should be\npicked up or not. With this
PR, we will be able to use rrule as well.\n\nWe set the `runAt` field in
3 places:\n- At task creation: when the task is created for the first
time, this is\nalways `now` if a specific runAt is not provided.\n- At
task update. We update runAt on task update as well but only when\nit is
provided. We don't update it when the schedule config changes.\n- After
task run: to decide when the next run will be after a task
is\nrun.\n\nThis PR modifies:\n- At task creation: It still uses the
`runAt` when it is provided but\ncalculates a runAt by using rrule if
the config has a fixed time (like\nevery day at 13:00) config.
Becausethe task should not run immediately\nwhen there is a fixed time
to run.\n- After task run: It calculates the next `runAt` with rrule
when it is\nprovided.\n\nThis PR doesn't apply any change on `At task
update` as it requires a\nfurther investigation and consideration. We
will handle it with a\nfollow-on issue.\n\n## To verify:\nYou can use
the test task `sampleRecurringTask` to test the feature.\nJust create a
task with rrule in the schedule config.\nThen check the runAt field in
the task SO to see if it is the expected\ndatetime.\nThen run the task
one more time and check the new `runAt`\n\nI also used the usage
reporting task in the actions plugin to test
the\nfeature.\n\n---------\n\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"0878a095a556b09bd05695469539c25be2c3423d"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/217728","number":217728,"mergeCommit":{"message":"Support
rrule for task scheduling (#217728)\n\nResolves:
https://github.com/elastic/kibana/issues/216308\n\nThis PR adds rrule
notation support to the task scheduling.\n\nCurrently we use a simple
interval string (such as `1h` or `45m`) to\ncalculate the next `runAt`
that we use to decide if the task should be\npicked up or not. With this
PR, we will be able to use rrule as well.\n\nWe set the `runAt` field in
3 places:\n- At task creation: when the task is created for the first
time, this is\nalways `now` if a specific runAt is not provided.\n- At
task update. We update runAt on task update as well but only when\nit is
provided. We don't update it when the schedule config changes.\n- After
task run: to decide when the next run will be after a task
is\nrun.\n\nThis PR modifies:\n- At task creation: It still uses the
`runAt` when it is provided but\ncalculates a runAt by using rrule if
the config has a fixed time (like\nevery day at 13:00) config.
Becausethe task should not run immediately\nwhen there is a fixed time
to run.\n- After task run: It calculates the next `runAt` with rrule
when it is\nprovided.\n\nThis PR doesn't apply any change on `At task
update` as it requires a\nfurther investigation and consideration. We
will handle it with a\nfollow-on issue.\n\n## To verify:\nYou can use
the test task `sampleRecurringTask` to test the feature.\nJust create a
task with rrule in the schedule config.\nThen check the runAt field in
the task SO to see if it is the expected\ndatetime.\nThen run the task
one more time and check the new `runAt`\n\nI also used the usage
reporting task in the actions plugin to test
the\nfeature.\n\n---------\n\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"0878a095a556b09bd05695469539c25be2c3423d"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label May 14, 2025
ymao1 added a commit that referenced this pull request May 20, 2025
@ymao1 @ersin-erdal
@kibanamachine @elasticmachine
[Response Ops][Reporting] Scheduled Reports - Schedule API (#219771)
fd0a43d 
Towards #216313

> [!Note]
> This PR will be merged into a feature branch -
`elastic:scheduled-reports`
> Once this is merged, I will open a draft PR from the feature branch
into `main`

## Summary

### Schedule API

- Added internal API `/internal/reporting/schedule/{exportTypeId}`

```
POST /internal/reporting/schedule/{exportTypeId}
{
  "jobParams": <same jobParams format as existing generate single report API>,
  "schedule": {
    "rrule": <modified Task Manager rRule schema from #217728>
  }
  "notification": {
    "email": {
      "to": <optional list of email addresses>,
      "cc": <optional list of email addresses>,
      "bcc": <optional list of email addresses>
    }
}
```

- Scheduling a report creates a new `scheduled_report` type saved object
in the `.kibana-alerting-cases` index.
- Uses the same role privileges as the current generate report API, so
user must have the reporting subfeature privilege for one of the
Analytics feature privileges (Discover/Dashboard/Visualize)

## Verify

1. Run ES and Kibana with examples and install the sample web logs data
set.
2. In the DEV console, call the schedule API using

```
POST kbn:/internal/reporting/schedule/printablePdfV2
{
    "schedule": {
        "rrule": {
            "freq": 3,
            "interval": 3,
            "byhour": [12],
            "byminute": [0]
        }
    },
    "jobParams": "(browserTimezone:America/New_York,layout:(dimensions:(height:2220,width:1364),id:preserve_layout),locatorParams:!((id:DASHBOARD_APP_LOCATOR,params:(dashboardId:edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b,preserveSavedFilters:!t,timeRange:(from:now-7d/d,to:now),useHash:!f,viewMode:view))),objectType:dashboard,title:'[Logs] Web Traffic',version:'9.1.0')"
}
```

3. Verify a saved object has been created using `GET
.kibana_alerting_cases/_search?q=type:scheduled_report`
4. Try different requests with different schedules and notifications. It
should respect the email domain allowlist if configured and reject
invalid rrule schedules and email addresses.

---------

Co-authored-by: Ersin Erdal <ersin.erdal@elastic.co>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Ersin Erdal <92688503+ersin-erdal@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
akowalska622 pushed a commit to akowalska622/kibana that referenced this pull request May 29, 2025
@ersin-erdal @kibanamachine @akowalska622
Add rrule schema to task schedule with intermediate release (elastic#…
045495f 
…219429)

Towards: elastic#216308

This PR adds the `rrule` schema to the task SO with an intermediate
release for the following PR:
elastic#217728

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
akowalska622 pushed a commit to akowalska622/kibana that referenced this pull request May 29, 2025
@ersin-erdal @kibanamachine @akowalska622
Support rrule for task scheduling (elastic#217728)
3e789bd 
Resolves: elastic#216308

This PR adds rrule notation support to the task scheduling.

Currently we use a simple interval string (such as `1h` or `45m`) to
calculate the next `runAt` that we use to decide if the task should be
picked up or not. With this PR, we will be able to use rrule as well.

We set the `runAt` field in 3 places:
- At task creation: when the task is created for the first time, this is
always `now` if a specific runAt is not provided.
- At task update. We update runAt on task update as well but only when
it is provided. We don't update it when the schedule config changes.
- After task run: to decide when the next run will be after a task is
run.

This PR modifies:
- At task creation: It still uses the `runAt` when it is provided but
calculates a runAt by using rrule if the config has a fixed time (like
every day at 13:00) config. Becausethe task should not run immediately
when there is a fixed time to run.
- After task run: It calculates the next `runAt` with rrule when it is
provided.

This PR doesn't apply any change on `At task update` as it requires a
further investigation and consideration. We will handle it with a
follow-on issue.

## To verify:
You can use the test task `sampleRecurringTask` to test the feature.
Just create a task with rrule in the schedule config.
Then check the runAt field in the task SO to see if it is the expected
datetime.
Then run the task one more time and check the new `runAt`

I also used the usage reporting task in the actions plugin to test the
feature.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
qn895 pushed a commit to qn895/kibana that referenced this pull request Jun 3, 2025
@ersin-erdal @kibanamachine @qn895
Add rrule schema to task schedule with intermediate release (elastic#…
5489904 
…219429)

Towards: elastic#216308

This PR adds the `rrule` schema to the task SO with an intermediate
release for the following PR:
elastic#217728

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
qn895 pushed a commit to qn895/kibana that referenced this pull request Jun 3, 2025
@ersin-erdal @kibanamachine @qn895
Support rrule for task scheduling (elastic#217728)
827094b 
Resolves: elastic#216308

This PR adds rrule notation support to the task scheduling.

Currently we use a simple interval string (such as `1h` or `45m`) to
calculate the next `runAt` that we use to decide if the task should be
picked up or not. With this PR, we will be able to use rrule as well.

We set the `runAt` field in 3 places:
- At task creation: when the task is created for the first time, this is
always `now` if a specific runAt is not provided.
- At task update. We update runAt on task update as well but only when
it is provided. We don't update it when the schedule config changes.
- After task run: to decide when the next run will be after a task is
run.

This PR modifies:
- At task creation: It still uses the `runAt` when it is provided but
calculates a runAt by using rrule if the config has a fixed time (like
every day at 13:00) config. Becausethe task should not run immediately
when there is a fixed time to run.
- After task run: It calculates the next `runAt` with rrule when it is
provided.

This PR doesn't apply any change on `At task update` as it requires a
further investigation and consideration. We will handle it with a
follow-on issue.

## To verify:
You can use the test task `sampleRecurringTask` to test the feature.
Just create a task with rrule in the schedule config.
Then check the runAt field in the task SO to see if it is the expected
datetime.
Then run the task one more time and check the new `runAt`

I also used the usage reporting task in the actions plugin to test the
feature.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@fbaligand fbaligand mentioned this pull request Aug 3, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:version Backport to applied version labels release_note:feature Makes this part of the condensed release notes Team:ResponseOps Platform ResponseOps team (formerly the Cases and Alerting teams) t// v8.19.0 v9.1.0

6 participants

@ersin-erdal @elasticmachine @kibanamachine @pmuellr @jloleysens @ymao1