Goal: migrate all currently implemented Metricbeat gcp module Metricsets to Data Streams within gcp integration package and add system tests.

TODO:

• gcp.billing data stream:

  • migrate metricset: [GCP] Add GCP Billing Metricset #2141
  • add data stream tests: GCP package billing datastream tests #2312

• gcp.compute data stream:

  • migrate metricset [GCP] Add GCP Compute Metricset #2301
  • add data stream tests [GCP] add compute datastream tests #2849

• gcp.loadbalancer data stream:

  • migrate metricset: [GCP] Add GCP Load Balancer Metricset #2308
  • add data stream tests: todo

• gcp.pubsub data stream:

  • migrate metricset: [GCP] Add GCP PubSub metricset #2309
  • add data stream tests: todo

• gcp.storage data stream:

  • migrate metricset: [GCP] Add GCP Storage Metricset #2310
  • add data stream tests: todo

• gcp.metrics data stream:

  • migrate metricset: [GCP] Add GCP Custom Metricset #2314
  • add data stream tests: todo

• gcp.firestore data stream:

  • [GCP] Add firestore data_stream #2704
  • test integrated in the PR above

• ensure dashboards query the Agent stream and not the Metricbeat ones

🛑 Blockers

• gcp.billing require auxiliary files to bootstrap test data
  • copy all files for terraform system tests elastic-package#603
    • Support auxiliary files in terraform deploy spec package-spec#272
      • Use Terraform Dependecy Lock in system tests elastic-package#700
• elastic-package Terraform deployer should support GCP Cloud SDK blocks billing tests
  • Add GCP Cloud SDK to Terraform deployer image elastic-package#638
    • Use GCP ADC for auth in Terraform Deployer elastic-package#662
      • Add GCP test package elastic-package#701
• [GCP] Configuration breaking changes due to metrics data streams addition #2987
  • we decided to move forward with a new package, gcp_metrics to avoid the need for configuration breaking change. we reverted the decision and decided to wait on Kibana team to fix the upgrade issue and move forward with a single gcp package.

🏗 Implementation work

• [gcp] Migrate some gcp beat metricset to data streams #2707

❓ Open questions

• Should the new data streams be added to gcp package? As of now it contains only security related log collection.

🚨 Risks

• elastic/integrations CI does not support creating GCP infrastructure to be used during tests, but it supports the use case; support should be added.
  • copy all files for terraform system tests elastic-package#603
    • Support auxiliary files in terraform deploy spec package-spec#272
  • Add GCP Cloud SDK to Terraform deployer image elastic-package#638
    • Use GCP ADC for auth in Terraform Deployer elastic-package#662
      • Add GCP test package elastic-package#701
        With this solved we still need to migrate the CI configuration from elastic/elastic-package to elastic/integrations. The risk lies in the different CI and Stack versions in use