[gcp_vpcflow_otel] Content pack of EDOT Cloud Forwarder for GCP - VPC Flow Logs

[mykola-elastic]

Proposed commit message

See title.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Screenshots

• I didn't use Axis names for IP addresses and Bytes as it seems to be very obvious (also would be double-stating as the axis names are already present in the panel title). Feel free to share your point if it feels otherwise
• I deliberately didn't filter out the (null) value from countries bar chart (as it will drop a big portion of logs for which the country couldn't be deduced)

[alaudazzi]

Left a minor suggestion.

[agithomas]

Does the input control drop-down allow selection of more than one value of Reporter? If yes, you may want to consider breaking down the values in the charts by the value of Reporter

[agithomas]

Please follow the best practices:

Avoid using “count” or “number” unless it leads to ambiguity
When giving a name to a widget or to the y-axis (or x-axis) of a time series graph, avoid adding words that can be assumed.

Break down complex widget names with hyphen
Example, instead of “Model invocation latency by model”, rename to “Model invocation latency - by model”.

[mykola-elastic]

Does the input control drop-down allow selection of more than one value of Reporter? If yes, you may want to consider breaking down the values in the charts by the value of Reporter

As for the reporter: it can be SRC, DEST, SRC_GATEWAY or DEST_GATEWAY. https://docs.cloud.google.com/vpc/docs/about-flow-logs-records

The side that reported the flow.
For VMs and serverless endpoints, the reporter can be SRC or DEST.
For gateways such as VLAN attachments for Cloud Interconnect and Cloud VPN tunnels, the reporter can be SRC_GATEWAY or DEST_GATEWAY.

Intuitively I decided that it would be better as a control. What do you think about that?

[mykola-elastic]

Avoid using “count” or “number” unless it leads to ambiguity
When giving a name to a widget or to the y-axis (or x-axis) of a time series graph, avoid adding words that can be assumed.

Should I remove all axis names? (The remaining are: @timestamp, Country ISO codes, and bytes sent) All the other ones are already removed as they are obvious from the look and the title of the panel.

Break down complex widget names with hyphen
Example, instead of “Model invocation latency by model”, rename to “Model invocation latency - by model”.

Thanks, I'll do that

[agithomas]

As for the reporter: it can be SRC, DEST, SRC_GATEWAY or DEST_GATEWAY. https://docs.cloud.google.com/vpc/docs/about-flow-logs-records

As i mentioned, if the multi-select of the value is possible in the input control, it would be best to have it split by the "reporter" value type so that comparison is possible - provided that it has an observability relevance.

Should I remove all axis names? (The remaining are: @timestamp, Country ISO codes, and bytes sent) All the other ones are already removed as they are obvious from the look and the title of the panel.

You could consider renaming "Traffic volume over time" as "Total bytes sent" and in the y-axis you can put the y-axis label "Bytes"

[ShourieG]

LGTM from my end

[ishleenk17]

@mykola-elastic : This is our current dashboard from Elastic (data is not there) and from the comparison I did we do have most of fields are present via the logging extension.
Can we see if we need to add more panels. Pls chek if we have missed on something important in the OTEL dashboard.

[constanca-m]

I don't have much knowledge on content packs, but it looks good to me

[mykola-elastic]

@agithomas @ishleenk17 Updated the dashboard according to your suggestions:

• Removed the control
• Renamed traffic to total bytes sent
• Added the panels bytes per reporter, per source VPC, per destination VPC

Before:

After:

[joecompute]

I also don't have much knowledge on content packs, but looks like a good start for Tech Preview, thank you!

Thank you for trying to match the ECS equivalent current dashboard.

• Are there any things you would like to call out that are missing? Do you feel they are important?
  • Are there any data gaps we need to bridge causing missing things in dashboards?

For example, I realize we are missing panels vs the existing dashboard (thank you @ishleenk17 for pointing that out, I think it's important to compare existing vs OTEL):

• the Sankey diagrams
• bytes source vs destination
• bytes per direction

I think those would be super useful and eventually necessary. But, I don't think we need to put them in this PR; we can interate. Can we get these tracked somehow?

I also think it would be good to take these assets a step further and eventually

• Have diagram for week-over-week or month-over-month changes in src-dest traffic to try and catch anomalies/things to investigate.
  • Could be a Sankey diagram

Maybe we can pull in a Security UX person to say what are "must-haves" and help us with a user journey here. CC @mlunadia

[ishleenk17]

Maybe we can pull in a Security UX person to say what are "must-haves" and help us with a user journey here

We already have security folks involved in the PR review.

@mykola-elastic : Some of the Y axis has bytes instead of Bytes.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 5283820

History

• 💚 Build #35693 succeeded 20eb731
• 💚 Build #35689 succeeded daeb3e0
• 💚 Build #35374 succeeded 3e4505a
• 💚 Build #35319 succeeded 18462df

cc @mykola-elastic

[elastic-vault-github-plugin-prod]

Package gcp_vpcflow_otel - 0.1.0 containing this change is available at https://epr.elastic.co/package/gcp_vpcflow_otel/0.1.0/