Skip to content

feat: improve s3 access log parsing along with documentation updates - #15309

Merged
Kavindu-Dodan merged 2 commits into
elastic:mainfrom
Kavindu-Dodan:feat/improve-firehose-access-log-parsing
Sep 15, 2025
Merged

feat: improve s3 access log parsing along with documentation updates#15309
Kavindu-Dodan merged 2 commits into
elastic:mainfrom
Kavindu-Dodan:feat/improve-firehose-access-log-parsing

Conversation

@Kavindu-Dodan

@Kavindu-Dodan Kavindu-Dodan commented Sep 12, 2025

Copy link
Copy Markdown
Contributor

Proposed commit message

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Build package and deploy to see rendered documentation
  • Run integration tests to see correct log stream detection

Related issues

Fixes #15157

Screenshots

Improved documentation to highlight firehose integration only need asset intallation,

image

Footnotes

  1. https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html

@Kavindu-Dodan
Kavindu-Dodan requested a review from a team as a code owner September 12, 2025 17:59
@Kavindu-Dodan Kavindu-Dodan added the enhancement New feature or request label Sep 12, 2025
@Kavindu-Dodan
Kavindu-Dodan force-pushed the feat/improve-firehose-access-log-parsing branch from 138fa37 to 40313af Compare September 12, 2025 18:11
@andrewkroh andrewkroh added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:awsfirehose Amazon Data Firehose Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Sep 12, 2025

@kaiyan-sheng kaiyan-sheng left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for adding the clarification in the documentation!

return;
}

// Check for operation field content. Refer - https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html#log-record-fields

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Check for operation field content. Refer - https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html#log-record-fields
// Check for Operation field content. Refer - https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html#log-record-fields

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done with 5d35266 :)

def tokenCount = tokens.length;

// Check for S3 Access logs first using a more reliable token count
// Check for S3 Access logs first using a more reliable token content check

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit : this comment not needed at all I think

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I improved this comment only to show what it does - 5d35266

@MichaelKatsoulis MichaelKatsoulis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
@Kavindu-Dodan
Kavindu-Dodan force-pushed the feat/improve-firehose-access-log-parsing branch from 5d35266 to d5c5ce9 Compare September 15, 2025 15:53
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@elasticmachine

Copy link
Copy Markdown

💚 Build Succeeded

History

@Kavindu-Dodan
Kavindu-Dodan merged commit f7d18ff into elastic:main Sep 15, 2025
9 checks passed
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package awsfirehose - 1.9.0 containing this change is available at https://epr.elastic.co/package/awsfirehose/1.9.0/

tehbooom pushed a commit to tehbooom/integrations that referenced this pull request Nov 19, 2025
…lastic#15309)

* feat: improve s3 access log parsing along with documentation updates

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

* review changes

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

---------

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:awsfirehose Amazon Data Firehose Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services]

6 participants