feat(parse_aws_vpc_flow_log): Add support for v6-v8 fields

[andrewkroh]

Proposed commit message

The parse_aws_vpc_flow_log processor has been updated to parse new
fields introduced in AWS VPC Flow Logs versions 6, 7, and 8.

Version 6 adds Transit Gateway fields which don't map well to ECS.
Version 7 adds fields related to Amazon ECS tasks, which are now mapped
to ECS orchestrator.*, container.*, and service.name fields.
Version 8 adds the reject_reason field, which is mapped to
event.reason.

This change also includes updated documentation and adds new test cases
to validate the processing of v6 and v7 flow logs.

References

https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-vpc-flow-logs-for-elastic-container-services/
https://docs.aws.amazon.com/vpc/latest/userguide/flow-log-records.html#flow-logs-fields
https://docs.aws.amazon.com/vpc/latest/tgw/tgw-flow-logs.html#flow-log-records

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[github-actions]

🔍 Preview links for changed docs

• docs/reference/filebeat/processor-parse-aws-vpc-flow-log.md

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[theletterf]

@andrewkroh To which versions do these changes apply? We might have to add applies_to comment where relevant, as per new docs strategy. +CC @colleenmcginnis

https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/syntax/applies

[andrewkroh]

The enhancements will apply to >=9.2.0. Given how there are multiple different areas of the doc that only apply to the 9.2.0+ release I will probably need to restructure this to avoid having many tags. Any suggestions how to organize the information avoid tagging each piece of new content?

Do I need to do anything special with the asciidoc file that's present?

[theletterf]

@andrewkroh No need to edit the Asciidoc file, we do cumulative docs only in the new Markdown format.

For table rows, you could add an inline annotation, like this:

| ecs_cluster_name | orchestrator.cluster.name {applies_to}`stack: ga 9.2` |

@colleenmcginnis Could you confirm?

[colleenmcginnis]

This is what I would suggest. There are a lot of tags, but they aren't too noisy on the rendered page since there's already a lot of whitespace in the tables.

Given how there are multiple different areas of the doc that only apply to the 9.2.0+ release I will probably need to restructure this to avoid having many tags.

However, if you wanted to reduce the number of tags you could split the rows that were added in 9.2.0 into a separate table with a heading like Fields added in 9.2.0. In my opinion, that would make for a worse reader experience, but you could try it out.

[andrewkroh]

Thanks for the edits. I have applied all of the suggested edits.