[ResponseOps][Reporting] Send email with report attachment for scheduled reporting task (merging into feature branch) by doakalexi · Pull Request #220539 · elastic/kibana

doakalexi · 2025-05-08T15:51:21Z

Note

This PR will be merged into a feature branch

Summary

This PR updates the schedule reporting to flow to notify users once the report is generated successfully and adds support for attachments in the email connector. There is a 10 MB limit for schedule report attachments, if the attachment is larger than 10MB it will throw an error and save it in the scheduled report.

The flow for sending attachments using Nodemailer is simple, we just append them to the request.
For Exchange, the flow is more complicated:

Create a draft email
If the size of the attachment is 3MB or less we can just upload the base64 attachment to the draft email.
If the size of the attachment is > 3MB, we need to create an upload session and convert the attachment to bytes and upload it in 2MB chunks. After all the chunks have been uploaded we can close the upload session.
Send the draft email.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

To verify

Add a preconfigured connector kibana.yml and set notifications.connectors.default.email. We need to test this for Exchange and Nodemailer. Slack me for the config and secrets to test Exchange. For nodemailer you can use maildev.
Start kibana and add sample data.
Create a scheduled report for the sample data dashboard in dev tools
Verify that the scheduled report is emailed as an attachment and opens correctly.
Increase of the sample data dashboard to > 10MB and verify that there is an error and the scheduled report is not emailed.

…apping'

…-fix'

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

…te --fix'

… src/core/server/integration_tests/ci_checks'

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

… src/core/server/integration_tests/ci_checks'

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

afharo

Core changes LGTM

…ents

ymao1

Looks good! I tested sending the sample dashboard which is ~4Mb with nodemailer and it worked 🎉 .

Left a few more comments and ~~I wasn't able to receive a bcc only email using the exchange API.~~ the spam filter caught it

I see this warning in the logs:

[2025-06-06T13:15:13.486-04:00][WARN ][plugins.actions] Calling "execute" in UnsecuredActionsClient without any relatedSavedObjects data. Consider including this for traceability.

I think we should be able to include the scheduled report SO information

ymao1 · 2025-06-06T15:55:19Z

x-pack/platform/plugins/private/reporting/server/lib/tasks/run_scheduled_report.ts

+        throw new Error('Reporting notification service has not been initialized.');
+      }
+
+      if (reportSO && reportSO.attributes.notification && reportSO.attributes.notification.email) {


i guess if the reportSO is undefined for some reason, we could try to load it here?

Resolved in this commit, 894e62f

...atform/plugins/private/reporting/server/services/notifications/email_notification_service.ts

ymao1 · 2025-06-06T16:02:54Z

x-pack/platform/plugins/shared/stack_connectors/server/connector_types/email/send_email.ts

    subject,
    html: messageHTML,
    text: message,
+    ...(attachments && { attachments }),


should this be ...(attachments.length > 0 && { attachments }),? i don't know if it matters if we're passing an empty array?

Resolved in this commit, 894e62f

x-pack/platform/plugins/shared/stack_connectors/server/connector_types/email/send_email.ts

x-pack/platform/plugins/private/reporting/server/lib/tasks/run_scheduled_report.ts

...latform/plugins/shared/stack_connectors/server/connector_types/email/send_email_graph_api.ts

…_scheduled_report.ts Co-authored-by: Ying Mao <ying.mao@elastic.co>

…alexi/kibana into scheduled-reports-email-attachments

doakalexi · 2025-06-06T21:55:53Z

I see this warning in the logs:
[2025-06-06T13:15:13.486-04:00][WARN ][plugins.actions] Calling "execute" in UnsecuredActionsClient without any relatedSavedObjects data. Consider including this for traceability.
I think we should be able to include the scheduled report SO information

Resolved in this commit, 894e62f

...atform/plugins/private/reporting/server/services/notifications/email_notification_service.ts

…heduled-reports-email-attachments

x-pack/platform/plugins/private/reporting/server/lib/tasks/run_scheduled_report.ts

…ents

ymao1

LGTM!

ymao1 · 2025-06-16T16:02:07Z

x-pack/platform/plugins/private/reporting/server/lib/tasks/run_scheduled_report.ts

+      }
+    } catch (error) {
+      const message = `Error sending notification for scheduled report: ${error.message}`;
+      await this.saveExecutionWarning(


does using await here work with the .catch? I think we don't need this await.

Resolved in this commit, f3eb0fd

ymao1 · 2025-06-16T16:03:44Z

x-pack/platform/plugins/private/reporting/server/lib/tasks/run_scheduled_report.test.ts

+      expect(reportStore.setReportWarning).toHaveBeenCalledWith(savedReport, {
+        output: { content_type: 'application/pdf', size: 2097152 },
+        warning: 'Error sending notification for scheduled report: This is a test error!',
+      });


maybe a test for if saveExecutionWarning throws an error?

and a test that nothing is called if notifications are undefined?

Resolved in this commit, f3eb0fd

ymao1 · 2025-06-16T17:08:38Z

...latform/plugins/shared/stack_connectors/server/connector_types/email/send_email_graph_api.ts

+    const size = Buffer.byteLength(attachment.content);
+    if (size < smallAttachmentLimit) {
+      // If attachment is smaller than the limit, add the attachment to the draft email
+      logger.debug('[MS Exchange] attachment is smaller than 2Mb, attaching to draft');


nit: you could do logger = logger.get('ms-exchange') to add another context to the logger and then it would log as plugins.actions.email.ms-exchange and you woudn't need to add your own [MS Exchange]. logger tags might be useful too...we've been using those more in alerting.

Resolved in this commit, f3eb0fd

…alexi/kibana into scheduled-reports-email-attachments

elasticmachine · 2025-06-16T22:51:23Z

💛 Build succeeded, but was flaky

Failed CI Steps

Jest Tests #10

Test Failures

[job] [logs] Jest Tests #10 / RedirectWithDefaultEnvironment eventually renders the child element

Metrics [docs]

✅ unchanged

History

💔 Build #309071 failed e3944a9
💚 Build #308886 succeeded 018dcce
💚 Build #307539 succeeded d4f9afd
💚 Build #307472 succeeded e382f34
💚 Build #307179 succeeded 0473a06

Resolves #216313 ## Summary This is a feature branch that contains the following commits. Each individual linked PR contains a summary and verification instructions. * Schedule API - #219771 * Scheduled report task runner - #219770 * List and disable API - #220922 * Audit logging - #221846 * Send scheduled report emails - #220539 * Commit to check license - f5f9d9d * Update to list API response format - #224262 --------- Co-authored-by: Ersin Erdal <ersin.erdal@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Ersin Erdal <92688503+ersin-erdal@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Alexi Doak <109488926+doakalexi@users.noreply.github.com>

Resolves elastic#216313 ## Summary This is a feature branch that contains the following commits. Each individual linked PR contains a summary and verification instructions. * Schedule API - elastic#219771 * Scheduled report task runner - elastic#219770 * List and disable API - elastic#220922 * Audit logging - elastic#221846 * Send scheduled report emails - elastic#220539 * Commit to check license - elastic@f5f9d9d * Update to list API response format - elastic#224262 --------- Co-authored-by: Ersin Erdal <ersin.erdal@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Ersin Erdal <92688503+ersin-erdal@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Alexi Doak <109488926+doakalexi@users.noreply.github.com> (cherry picked from commit a409627) # Conflicts: # src/platform/packages/private/kbn-reporting/common/routes.ts # x-pack/platform/plugins/private/canvas/server/feature.test.ts # x-pack/platform/plugins/private/reporting/server/core.ts # x-pack/platform/plugins/private/reporting/server/features.ts # x-pack/platform/plugins/shared/features/server/__snapshots__/oss_features.test.ts.snap # x-pack/platform/test/api_integration/apis/features/features/features.ts # x-pack/test_serverless/api_integration/test_suites/chat/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/observability/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/search/platform_security/authorization.ts # x-pack/test_serverless/api_integration/test_suites/security/platform_security/authorization.ts

# Backport This will backport the following commits from `main` to `8.19`: - [[Response Ops][Reporting] Scheduled Reports (#221028)](#221028)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

ersin-erdal and others added 30 commits April 9, 2025 18:38

Support rrule for task scheduling

a74e49d

remove redundant return

6163f7e

add functional tests

20ecece

[CI] Auto-commit changed files from 'node scripts/styled_components_m…

c29a08e

…apping'

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

de20db1

…-fix'

fix linting

65f0960

Merge branch '216308-support-rrule-for-task-scheduling' of github.com…

2dde5f6

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

fix linter

717cadb

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

72b108e

…te --fix'

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

38e69ef

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

c9e71ba

… src/core/server/integration_tests/ci_checks'

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

5f6fad9

Allowing shared concurrency task types

d05d9c8

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

4a04b90

set second to [0]

249f14c

Merge branch '216308-support-rrule-for-task-scheduling' of github.com…

4c60d8e

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

testing schedule is more resilient

4a14142

remove "only"

4ec1bca

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

1275f1e

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

8aff2c9

Merge branch 'main' into shared-concurrency-tasks

04b70a2

Merge branch 'main' into shared-concurrency-tasks

c9547ae

throw when interval is not provided for the next run calculation

850ec64

Merge branch '216308-support-rrule-for-task-scheduling' of github.com…

10c40e5

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

Merge branch 'main' into 216308-support-rrule-for-task-scheduling

af75dfb

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

95cf333

… src/core/server/integration_tests/ci_checks'

update read me

d5227aa

Merge branch '216308-support-rrule-for-task-scheduling' of github.com…

2a1482e

…:ersin-erdal/kibana into 216308-support-rrule-for-task-scheduling

Merge branch 'main' into shared-concurrency-tasks

372d4c7

update read me

62fe072

afharo approved these changes Jun 3, 2025

View reviewed changes

Updates from pr comments

778c551

doakalexi requested a review from a team as a code owner June 3, 2025 22:30

doakalexi and others added 4 commits June 4, 2025 10:20

Adding notify() tests

e5ef677

Fixing failing test

85b8878

Fixing test failures

dfeaf86

Merge branch 'scheduled-reports' into scheduled-reports-email-attachm…

579966d

…ents

ymao1 reviewed Jun 6, 2025

View reviewed changes

doakalexi and others added 3 commits June 6, 2025 11:51

Update x-pack/platform/plugins/private/reporting/server/lib/tasks/run…

0f05f25

…_scheduled_report.ts Co-authored-by: Ying Mao <ying.mao@elastic.co>

More updates from pr comments

894e62f

Merge branch 'scheduled-reports-email-attachments' of github.com:doak…

52956ba

…alexi/kibana into scheduled-reports-email-attachments

ymao1 reviewed Jun 9, 2025

View reviewed changes

...atform/plugins/private/reporting/server/services/notifications/email_notification_service.ts Outdated Show resolved Hide resolved

doakalexi added 2 commits June 9, 2025 12:59

Updating for csv

7a8c6a3

Merge branch 'scheduled-reports' of github.com:elastic/kibana into sc…

0473a06

…heduled-reports-email-attachments

ymao1 reviewed Jun 10, 2025

View reviewed changes

x-pack/platform/plugins/private/reporting/server/lib/tasks/run_scheduled_report.ts Show resolved Hide resolved

doakalexi and others added 3 commits June 11, 2025 07:55

Moving around notificaton check

e382f34

Updating for pr comments

d4f9afd

Merge branch 'scheduled-reports' into scheduled-reports-email-attachm…

018dcce

…ents

ymao1 approved these changes Jun 16, 2025

View reviewed changes

doakalexi and others added 5 commits June 16, 2025 13:17

Updating for pr comments

f3eb0fd

Merge branch 'scheduled-reports-email-attachments' of github.com:doak…

3bea61e

…alexi/kibana into scheduled-reports-email-attachments

[CI] Auto-commit changed files from 'node scripts/yarn_deduplicate'

e3944a9

Fixing type check

179a35e

Merge branch 'scheduled-reports-email-attachments' of github.com:doak…

d9537f3

…alexi/kibana into scheduled-reports-email-attachments

doakalexi merged commit 94d67af into elastic:scheduled-reports Jun 17, 2025
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ResponseOps][Reporting] Send email with report attachment for scheduled reporting task (merging into feature branch)#220539

[ResponseOps][Reporting] Send email with report attachment for scheduled reporting task (merging into feature branch)#220539
doakalexi merged 146 commits intoelastic:scheduled-reportsfrom
doakalexi:scheduled-reports-email-attachments

doakalexi commented May 8, 2025 •

edited

Loading

afharo left a comment

ymao1 left a comment •

edited

Loading

ymao1 Jun 6, 2025

doakalexi Jun 6, 2025

Uh oh!

ymao1 Jun 6, 2025

doakalexi Jun 6, 2025

Uh oh!

Uh oh!

Uh oh!

doakalexi commented Jun 6, 2025

Uh oh!

Uh oh!

ymao1 left a comment

ymao1 Jun 16, 2025

doakalexi Jun 16, 2025

ymao1 Jun 16, 2025

ymao1 Jun 16, 2025

doakalexi Jun 16, 2025

ymao1 Jun 16, 2025

doakalexi Jun 16, 2025

elasticmachine commented Jun 16, 2025 •

edited

Loading

Uh oh!

Labels

6 participants

Conversation

doakalexi commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

To verify

afharo left a comment

Choose a reason for hiding this comment

ymao1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

doakalexi commented Jun 6, 2025

Uh oh!

Uh oh!

ymao1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

Uh oh!

Labels

6 participants

doakalexi commented May 8, 2025 •

edited

Loading

ymao1 left a comment •

edited

Loading

elasticmachine commented Jun 16, 2025 •

edited

Loading