Skip to content

Update (most) obs-cloud-monitoring owned packages to format_version 3.0.0 - #8041

Merged
tommyers-elastic merged 22 commits into
elastic:mainfrom
tommyers-elastic:update-cloud-packages-to-format-version-3
Oct 16, 2023
Merged

Update (most) obs-cloud-monitoring owned packages to format_version 3.0.0#8041
tommyers-elastic merged 22 commits into
elastic:mainfrom
tommyers-elastic:update-cloud-packages-to-format-version-3

Conversation

@tommyers-elastic

@tommyers-elastic tommyers-elastic commented Oct 2, 2023

Copy link
Copy Markdown
Contributor

This PR updates all obs-cloud-monitoring owned packages, except azure and azure_metrics, to package spec v3.

These changes are mostly autogenerated in three stages.

  1. Firstly we run elastic-package format on everything. This commit can be excluded from review to make the diff cleaner.

  2. Then the following yq commands are run across all the affected packages:

find $1/data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].release" {} \;
find $1/data_stream -type f -path "*fields*" -name "ecs.yml" -exec yq e -i "unique_by(.name)" {} \;
yq e -i 'del .release' manifest.yml
yq e -i "with(select(.license != null); .conditions.elastic.subscription = .license) | del .license" manifest.yml
  1. Next the ecs-update tool is run as follows:
go run github.com/andrewkroh/go-examples/ecs-update@main -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-cloudnative-monitoring packages/*

Any remaining issues were fixed manually. Even though most of the changes here were generated by tooling, this still needs careful review (hide whitespace changes to make your life easier).

There should be no functional changes to any of the packages here.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/aws_logs
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/awsfargate
The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/awsfirehose
The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_application_insights
The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_billing
The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/gcp_metrics
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_metrics
@tommyers-elastic tommyers-elastic added the enhancement New feature or request label Oct 2, 2023
@tommyers-elastic
tommyers-elastic requested a review from a team as a code owner October 2, 2023 12:25
Comment thread packages/azure/data_stream/activitylogs/fields/ecs.yml
Comment thread packages/azure_metrics/data_stream/storage_account/fields/agent.yml
@tommyers-elastic
tommyers-elastic requested a review from zmoog October 2, 2023 12:30
@elasticmachine

elasticmachine commented Oct 2, 2023

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-13T15:44:02.024+0000

  • Duration: 25 min 58 sec

Test stats 🧪

Test Results
Failed 0
Passed 37
Skipped 0
Total 37

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine

elasticmachine commented Oct 2, 2023

Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (1/1) 💚
Files 100.0% (1/1) 💚
Classes 100.0% (1/1) 💚
Methods 66.667% (14/21) 👍 3.252
Lines 100.0% (135/135) 💚 0.137
Conditionals 100.0% (0/0) 💚
@tommyers-elastic tommyers-elastic changed the title Update (most) obs-cloud-monitoring owned packages to format version 3 Oct 2, 2023

@zmoog zmoog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@tommyers-elastic, do you plan to address the platformlogs' _dev/test/pipeline/test-platformlogs-invalid-raw.log malformed JSON issues on in a different PR?

IIRC, the test fails on stack versions > 8.6 because the json processor in Elasticsearch has become stricter by default starting from 8.7.

More on this issue at zmoog/public-notes#54

@tommyers-elastic

Copy link
Copy Markdown
Contributor Author

hey @zmoog yeh i have another PR just for azure because it contains quite a few more changes. i haven't addressed the malformed log issue there yet, but i will get round to it soon.

@zmoog

zmoog commented Oct 5, 2023

Copy link
Copy Markdown
Contributor

@tommyers-elastic, I mentioned it so we do not forget it. Happy to help with a review!

Comment thread packages/awsfirehose/data_stream/logs/routing_rules.yml Outdated
@tommyers-elastic
tommyers-elastic force-pushed the update-cloud-packages-to-format-version-3 branch 4 times, most recently from 462fc6a to ed1b1fc Compare October 13, 2023 13:50
@tommyers-elastic
tommyers-elastic force-pushed the update-cloud-packages-to-format-version-3 branch from ed1b1fc to f98d4e6 Compare October 13, 2023 14:13
@elasticmachine

Copy link
Copy Markdown

Package aws_logs - 0.6.0 containing this change is available at https://epr.elastic.co/search?package=aws_logs

@elasticmachine

Copy link
Copy Markdown

Package awsfargate - 0.4.0 containing this change is available at https://epr.elastic.co/search?package=awsfargate

@elasticmachine

Copy link
Copy Markdown

Package awsfirehose - 0.5.0 containing this change is available at https://epr.elastic.co/search?package=awsfirehose

@elasticmachine

Copy link
Copy Markdown

Package azure_application_insights - 1.2.0 containing this change is available at https://epr.elastic.co/search?package=azure_application_insights

@elasticmachine

Copy link
Copy Markdown

Package azure_billing - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=azure_billing

@elasticmachine

Copy link
Copy Markdown

Package gcp_metrics - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=gcp_metrics

@andrewkroh andrewkroh added Integration:aws_logs Custom AWS Logs Integration:awsfargate AWS Fargate (for ECS clusters) Integration:awsfirehose Amazon Data Firehose Integration:azure_application_insights Azure Application Insights Metrics Overview Integration:azure_billing Azure Billing Metrics Integration:gcp_metrics GCP Metrics Input labels Jul 22, 2024
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
….0.0 (elastic#8041)

* run 'elastic-package format' across all obs-cloud-monitoring owned packages

* remove duplicate and unsupported fields from obs-cloud-monitoring owned packages

* [aws_logs] - removed dotted YAML keys from manifest

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/aws_logs

* [awsfargate] - removed dotted YAML keys from manifest

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/awsfargate

* [awsfirehose] - removed dotted YAML keys from manifest

The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/awsfirehose

* [azure_application_insights] - removed dotted YAML keys from manifest

The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_application_insights

* [azure_billing] - removed dotted YAML keys from manifest

The format_version in the package manifest changed from 2.9.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_billing

* [gcp_metrics] - removed dotted YAML keys from manifest

The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/gcp_metrics

* additional manual fixes for 'azure_metrics' package

* [azure_metrics] - removed dotted YAML keys from manifest

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@v0.0.0-20230926022833-8e5b97998cc0 -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type packages/azure_metrics

* update PR number on affected packages

* revert changes to 'azure' package - these will be done in a sepearate PR

* fix weird auto-formatting introduced into embedded painless scripts in yaml files.

* update awsfargate mappings for package-spec v3 compatibility

* update awsfirehose mappings for package-spec v3 compatibility

* update azure_application_insights validation.yml for package-spec v3 compatibility

* update azure_billing validation.yml for package-spec v3 compatibility

* revert changes to azure_metrics

* remove ingest pipeline that seems to have somehow creeped its way into this PR by mistake
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:aws_logs Custom AWS Logs Integration:awsfargate AWS Fargate (for ECS clusters) Integration:awsfirehose Amazon Data Firehose Integration:azure_application_insights Azure Application Insights Metrics Overview Integration:azure_billing Azure Billing Metrics Integration:gcp_metrics GCP Metrics Input

5 participants