[GCP] Add billing additional fields

[ghost]

Overview

This change introduces new billing fields and fixes the aggregation functions in the billing dashboard lenses.

Previously, the max aggregation was used, since the granularity of the billing data was low. Due to increased granularity which produces multiple documents per day, the sum aggregation is appropriate for accurately displaying total costs.

Fields added:

• sku_id
• sku_description
• service_id
• service_description
• effective_price
• tags (nested)
  • key
  • value

With these additional fields, we can add new lenses to improve the insight into billing data. This should be handled in a separate enhancement PR.

Following the merge of elastic/beats#36902, we can proceed with this PR.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-11-22T12:46:10.438+0000

• Duration: 20 min 12 sec

Test stats 🧪

Test	Results
Failed	0
Passed	69
Skipped	0
Total	69

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (6/6)	💚
Files	100.0% (6/6)	💚
Classes	100.0% (6/6)	💚
Methods	87.826% (101/115)	👎 -8.559
Lines	95.124% (1502/1579)	👎 -4.876
Conditionals	100.0% (0/0)	💚

[ishleenk17]

Lets add the sample data here

[ishleenk17]

Lets add the sample data here

@gpop63 : Please do the needful

[ishleenk17]

Looks good!

[ghost]

/test

[zmoog]

The change looks good; I only have one question and one nit.

Question

The PR on Beats introduced a detailed usage cost table alongside the existing standard table. IIRC, users must use the detailed table pattern to get the extra fields.

How do integration users get standard vs. detailed usage cost data? Should they use the detailed table pattern as well?

If users get standard vs. detailed usage cost data by setting the table pattern, we should describe this behavior in the docs and give a hint in the setting description in the UI.

Nit

We should include tag values in the sample document.

[zmoog]

nit: It would be helpful to include tag values in the sample event.

[zmoog]

Thanks for the docs update!

[elasticmachine]

Package gcp - 2.32.0 containing this change is available at https://epr.elastic.co/search?package=gcp