Skip to content

[AWS S3] Introduce start timestamp and ignore older timespan to AWS S3 based integrations - #12645

Merged
Kavindu-Dodan merged 2 commits into
elastic:mainfrom
Kavindu-Dodan:feat/s3-start-and-ignore-older-timestamps
Mar 4, 2025
Merged

[AWS S3] Introduce start timestamp and ignore older timespan to AWS S3 based integrations#12645
Kavindu-Dodan merged 2 commits into
elastic:mainfrom
Kavindu-Dodan:feat/s3-start-and-ignore-older-timestamps

Conversation

@Kavindu-Dodan

@Kavindu-Dodan Kavindu-Dodan commented Feb 6, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

Introduce Ignore Older Timespan and Start Timestamp properties to integrations backed by AWS S3 input,

  • Ignore Older Timespan: Accepts a timespan in which entries are accepted for processing
  • Start Timestamp: Accepts a timestamp from which objects are accepted for processing

Configuring these properties allows S3 input to efficiently manage its internal registry. For example, setting Ignore Older Timespan to 2h makes the S3 input registry only track entries within the last 2 hours. Once entries are beyond the timespan, input can remove them from the registry, thus reducing memory consumption.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Screenshots

Configuration rendered (Title matching existing format)

image
@Kavindu-Dodan
Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch 3 times, most recently from 888b269 to 2540178 Compare February 7, 2025 18:02
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Feb 7, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
@Kavindu-Dodan Kavindu-Dodan added the Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] label Feb 7, 2025
@Kavindu-Dodan
Kavindu-Dodan marked this pull request as ready for review February 7, 2025 19:38
@Kavindu-Dodan
Kavindu-Dodan requested review from a team as code owners February 7, 2025 19:38
@Kavindu-Dodan Kavindu-Dodan changed the title [AWS S3] Introduce start timestamp and ignore older duration to AWS S3 based integrations Feb 7, 2025
gizas
gizas reviewed Feb 10, 2025
View reviewed changes
Comment thread packages/cloudflare_logpush/data_stream/audit/agent/stream/aws-s3.yml.hbs Outdated
gizas
gizas reviewed Feb 10, 2025
View reviewed changes
Comment thread packages/cloudflare_logpush/data_stream/casb/agent/stream/aws-s3.yml.hbs Outdated
@MichaelKatsoulis
MichaelKatsoulis self-requested a review February 10, 2025 16:01
@agithomas

agithomas commented Feb 11, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

@Kavindu-Dodan , could you please consider backporting the beats changes so that we can bump up the package versions only to 8.16.5 instead of 8.18.0?

cc @zmoog
zmoog
zmoog reviewed Feb 11, 2025
View reviewed changes

@zmoog zmoog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider a backporting the Beats change to keep the min stack version in the 8.16.x series.
Comment thread packages/aws/manifest.yml Outdated
@Kavindu-Dodan

Kavindu-Dodan commented Feb 14, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Pending backports to 8.16.x 1 (next - 8.16.5) & 8.17.x 2 (next - 8.17.3) tracks. I will update this PR accordingly

Footnotes

  1. https://github.com/elastic/beats/pull/42716 ↩

  2. https://github.com/elastic/beats/pull/42717 ↩
@Kavindu-Dodan
Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch from 2540178 to 58cdb90 Compare February 14, 2025 21:23
@Kavindu-Dodan
Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch 3 times, most recently from 9fc363c to 93ea04f Compare February 19, 2025 21:16
@agithomas

agithomas commented Feb 20, 2025

Copy link
Copy Markdown
Contributor

@Kavindu-Dodan , we have a great illustration of this feature's behaviour as part of the description of this issue.

Do you think that it will be good to include this illustration in AWS documentation? Reference: Azure Integration I think, if we include this illustration, it would avoid misconfiguration of this setting.

@andrewkroh, could you please share your opinion - if it is a good idea to include the illustration as part of the AWS documentation?

The changes look good to me. As discussed, kindly merge the changes after the 8.16.5 is available. Also, let us get the approval from the security-integrations team.
@Kavindu-Dodan

Kavindu-Dodan commented Feb 20, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@agithomas good point on the documentation. I thought of adding some diagrams, but when checking the current AWS integration documentation, I couldn't find a suitable place to add the details. So I went with detailed descriptions for the integration input fields instead.

Anyway, let me know if there's better placement for detailed documentation.
@Kavindu-Dodan
Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch from 93ea04f to c7c60dc Compare February 20, 2025 17:53
andrewkroh
andrewkroh reviewed Feb 20, 2025
View reviewed changes

@andrewkroh andrewkroh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewkroh, could you please share your opinion - if it is a good idea to include the illustration as part of the AWS documentation?

Personally, I don't think they are necessary to convey the meaning of the configuration options. If we need to provide additional context, then I would suggest linking to the reference docs like, "See the [input reference documentation] for more details." And add images in the reference docs.
Comment thread packages/amazon_security_lake/changelog.yml Outdated
Comment thread packages/amazon_security_lake/data_stream/event/manifest.yml Outdated
Comment thread packages/amazon_security_lake/data_stream/event/manifest.yml Outdated
@agithomas

agithomas commented Feb 21, 2025

Copy link
Copy Markdown
Contributor

If we need to provide additional context, then I would suggest linking to the reference docs like, "See the [input reference documentation] for more details." And add images in the reference docs.

Thanks @andrewkroh for your inputs. Yes, i agree that it would be the best approach.

@Kavindu-Dodan , it need not be part of this PR, it could be a separate PR by including the documentation team as reviewers.
@Kavindu-Dodan
Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch 2 times, most recently from 1b23f2f to 241160f Compare February 21, 2025 15:33
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package carbon_black_cloud - 2.9.0 containing this change is available at https://epr.elastic.co/package/carbon_black_cloud/2.9.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package cisco_umbrella - 1.30.0 containing this change is available at https://epr.elastic.co/package/cisco_umbrella/1.30.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package cloudflare_logpush - 1.35.0 containing this change is available at https://epr.elastic.co/package/cloudflare_logpush/1.35.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package f5_bigip - 1.27.0 containing this change is available at https://epr.elastic.co/package/f5_bigip/1.27.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package imperva_cloud_waf - 1.9.0 containing this change is available at https://epr.elastic.co/package/imperva_cloud_waf/1.9.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package jamf_protect - 2.11.0 containing this change is available at https://epr.elastic.co/package/jamf_protect/2.11.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package sentinel_one_cloud_funnel - 1.10.0 containing this change is available at https://epr.elastic.co/package/sentinel_one_cloud_funnel/1.10.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package servicenow - 0.11.0 containing this change is available at https://epr.elastic.co/package/servicenow/0.11.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package sublime_security - 1.7.0 containing this change is available at https://epr.elastic.co/package/sublime_security/1.7.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package symantec_endpoint_security - 1.9.0 containing this change is available at https://epr.elastic.co/package/symantec_endpoint_security/1.9.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package tanium - 1.16.0 containing this change is available at https://epr.elastic.co/package/tanium/1.16.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 4, 2025

Copy link
Copy Markdown
Contributor

Package trellix_edr_cloud - 1.8.0 containing this change is available at https://epr.elastic.co/package/trellix_edr_cloud/1.8.0/
flexitrev pushed a commit that referenced this pull request Mar 20, 2025
@Kavindu-Dodan @flexitrev
[AWS S3] Introduce start timestamp and ignore older timespan to AWS S…
7791199 
…3 based integrations (#12645)

* add s3 start timestamp and ignore older duration to integrations

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* review changes

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

---------

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment