[AWS] Drop header log line in CloudFront events

[0ccupi3R]

By default the CloudFront stores these two lines as a header in each log file. Before this change it fails the pipeline and add message to error.message field.

Added the sample log line
#5016 6c95e22

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-01-25T09:04:12.768+0000

• Duration: 52 min 50 sec

Test stats 🧪

Test	Results
Failed	0
Passed	179
Skipped	3
Total	182

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[zmoog]

Hey @0ccupi3R, thank you for taking the time to work on this contribution to the AWS integration!

You also created #5016 with a sample file for the tests. Could you please move that sample file to this PR? Having them all together in the same PR allows us to work and merge the change as a single unit of work.

[0ccupi3R]

Hello @zmoog

Thanks for picking it. Even though it was just a header line, I have added it to the sample log line within the same PR.

[zmoog]

For the records, I see the Elastic Agent integration uses a 'local' processor to filter out the logs comments before sending:

integrations/packages/aws/data_stream/cloudfront_logs/agent/stream/aws-s3.yml.hbs

Lines 48 to 52 in ef38d55

	processors:
	- drop_event:
	when:
	regexp:
	message: "^#.*"

So when the Agent runs with the DEBUG log level we can see the following message:

[elastic_agent.filebeat
][debug
] Pipeline client receives callback 'onFilteredOut' for event: {Timestamp: 2023-01-20 03: 52: 03.775055087 +0000 UTC Meta: {
        "_id": "cd1aa7d915-000000000000",
        "input_id": "aws-s3-cloudfront-7efa1443-a33f-4852-bd84-b325aac7b311",
        "raw_index": "logs-aws.cloudfront_logs-default"
    } Fields: {
        "aws": {
            "s3": {
                "bucket": {
                    "arn": "arn:aws:s3:::zmoog-dev-cloudfront-standard-logs",
                    "name": "zmoog-dev-cloudfront-standard-logs"
                },
                "object": {
                    "key": "logs/EA060IVQIIUAL.2023-01-20-03.335883cd.gz"
                }
            }
        },
        "cloud": {
            "provider": "",
            "region": "eu-west-1"
        },
        "input": {
            "type": "aws-s3"
        },
        "log": {
            "file": {
                "path": "https://zmoog-dev-cloudfront-standard-logs.s3.eu-west-1.amazonaws.com/logs/EA060IVQIIUAL.2023-01-20-03.335883cd.gz"
            },
            "offset": 0
        },
        "message": "#Version: 1.0",
        "tags": [
            "preserve_original_event",
            "forwarded",
            "aws-cloudfront"
        ]
    } Private: 0x4000c96600 TimeSeries: false
}

I can see it contains "message":"#Version: 1.0".

@0ccupi3R, are you using the Elastic Agent to ingest the CloudFront standard logs?

[zmoog]

It's worth dropping invalid loglines to avoid errors.

[zmoog]

/test

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (14/14)	💚
Files	93.333% (14/15)	👎 -6.667
Classes	93.333% (14/15)	👎 -6.667
Methods	85.271% (220/258)	👎 -10.596
Lines	96.024% (5917/6162)	👍 0.256
Conditionals	100.0% (0/0)	💚

[zmoog]

/test

[zmoog]

/test

[zmoog]

/test

[zmoog]

/test

[elasticmachine]

Package aws - 1.29.1 containing this change is available at https://epr.elastic.co/search?package=aws