Skip to content

[AWS] Add EMR metrics data stream - #6120

Merged
13 commits merged into
mainfrom
unknown repository
Jun 16, 2023
Merged

[AWS] Add EMR metrics data stream#6120
13 commits merged into
mainfrom
unknown repository

Conversation

@ghost

@ghost ghost commented May 9, 2023 •
edited by ghost
Loading

Copy link
Copy Markdown

What does this PR do?

Adds AWS EMR (ElasticMapReduce) metrics data stream.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots
@elasticmachine

elasticmachine commented May 9, 2023 •
edited
Loading

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-06-15T12:28:22.305+0000

  • Duration: 52 min 0 sec

Test stats 🧪

Test Results
Failed 0
Passed 192
Skipped 4
Total 196

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.
@elasticmachine

elasticmachine commented May 9, 2023 •
edited
Loading

Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (15/15) 💚
Files 93.75% (15/16) 👎 -6.25
Classes 93.75% (15/16) 👎 -6.25
Methods 85.714% (240/280) 👍 60.714
Lines 85.925% (7387/8597) 👎 -14.075
Conditionals 100.0% (0/0) 💚
@ghost
ghost force-pushed the add_aws-emr_metrics branch from 5f0cef1 to 9b91ea9 Compare May 15, 2023 18:03
@ghost
ghost force-pushed the add_aws-emr_metrics branch from 9b91ea9 to b6a4e7d Compare May 15, 2023 21:16
@ghost
ghost marked this pull request as ready for review May 16, 2023 00:30
@ghost
ghost self-requested a review as a code owner May 16, 2023 00:30
ritalwar
ritalwar reviewed May 16, 2023
View reviewed changes

@ritalwar ritalwar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please define the metric type as either counter or gauge for applicable fields? This would be beneficial from a TSDB perspective in the future.
@rameshelastic rameshelastic assigned ghost May 18, 2023
@ghost
ghost force-pushed the add_aws-emr_metrics branch from 6dc09e7 to 694620e Compare May 29, 2023 23:40
@ghost
ghost force-pushed the add_aws-emr_metrics branch from a32b736 to bd23a90 Compare May 29, 2023 23:42
@ghost
ghost requested a review from ritalwar May 30, 2023 14:29
lucian-ioan
lucian-ioan reviewed Jun 6, 2023
View reviewed changes
Comment thread packages/aws/_dev/build/docs/emr.md
Comment on lines +1 to +7
# Amazon EMR (ElasticMapReduce)

The Amazon EMR integration allows you to monitor [Amazon EMR](https://aws.amazon.com/emr/) — a fully managed big data processing and analytics service.

Use the Amazon EMR integration to collect metrics related to your EMR instances. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference the metrics when troubleshooting an issue.

For example, you could use this data to track Amazon EMR cluster progress and cluster storage. Then you can alert when utilization for an instance crosses a predefined threshold.

@lucian-ioan lucian-ioan Jun 6, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about clarifying the Hadoop version for this data stream in this doc?

Idea: Creating a meta issue for AWS integrations that we own, to improve documentation clarity. What are your thoughts on this? @zmoog @lalit-satapathy

@zmoog zmoog Jun 13, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is a good idea. @lucianpy, could you create an issue with a detailed list of information to add to this integration?

This way, we can ship the integration as it is and start working on an actionable list right after the current iteration ends.
zmoog
zmoog reviewed Jun 14, 2023
View reviewed changes
Comment thread packages/aws/data_stream/emr_metrics/agent/stream/stream.yml.hbs

@zmoog zmoog Jun 14, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recently we added a new cloudwatch metricset setting called include_linked_accounts to collect data from linked account. I guess we could add this to this integration as well.

See #6459 for more context about this topic.
Comment thread packages/aws/data_stream/emr_metrics/manifest.yml
@ghost

ghost commented Jun 15, 2023

Copy link
Copy Markdown
Author

/test
@elasticmachine

elasticmachine commented Jun 15, 2023

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @gpop63
zmoog
zmoog approved these changes Jun 16, 2023
View reviewed changes

@zmoog zmoog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, @gpop63! Could you create an issue and add the steps required to move this integration from BETA to GA? Many thanks!
@ghost
ghost merged commit a2f0dd9 into elastic:main Jun 16, 2023
@elasticmachine

elasticmachine commented Jun 16, 2023

Copy link
Copy Markdown

Package aws - 1.45.0 containing this change is available at https://epr.elastic.co/search?package=aws
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
Gabriel Pop @zmoog
[AWS] Add EMR metrics data stream (elastic#6120)
cc55e3c 
* add emr_metrics data stream

* bump package version

* fix aws package version

* fix field types

* add unit and metric_types

* add docs

* fix typo in docs

* Update packages/aws/data_stream/emr_metrics/manifest.yml

Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

* add tags_filter and include_linked_accounts

* add beta release

---------

Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration:aws AWS

6 participants

@elasticmachine @zmoog @lucian-ioan @ritalwar @andrewkroh @mrg-elastic