[AWS] Add support for Inspector datastream and Remove duplicate ECS fields

[vinit-chauhan]

What does this PR do?

• Added a data stream.
• Added data collection logic to the data stream.
• Added the ingest pipeline for the data stream.
• Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files
• Added dashboards and visualizations.
• Added test for pipeline for the data stream.
• Added system test cases for the data stream.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^8.4.0).

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/aws/data_stream directory.
• Run the following command to run tests.

elastic-package test

Related issues

• Related sei: multiple packages with duplicated field definitions #4398
• Closes AWS Inspector #4687

Screenshots

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-11-18T07:17:44.208+0000

• Duration: 38 min 35 sec

Test stats 🧪

Test	Results
Failed	0
Passed	174
Skipped	3
Total	177

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (14/14)	💚
Files	93.333% (14/15)	👎 -4.133
Classes	93.333% (14/15)	👎 -4.133
Methods	85.098% (217/255)	👎 -5.768
Lines	95.949% (5897/6146)	👍 4.242
Conditionals	100.0% (0/0)	💚

[kaiyan-sheng]

@vinit-elastic Could we actually remove the duplicate fields from agent.yml and keep them in the ecs.yml instead? Thanks!! I just did this for the ec2 integration: #4567

[vinit-chauhan]

@kaiyan-sheng the reason why I removed fields from ecs.yml is because the agent.yml is a static file and is the same across all the connectors. The same has been discussed with @P1llus earlier, and we came to a conclusion not to change it unless there's a type change in the newer ecs version. Therefore we didn't change it.

[P1llus]

@vinit-elastic I agree, if we want to change agent.yml we should simply do it in one go for all packages instead @kaiyan-sheng ?

[drewdaemon]

Hi @vinit-elastic! I'm going to request that we use the new Lens metric vis instead of the TSVB metric on the dashboard since it's a good fit and we're trying to get away from TSVB. Here's a screenshot of how you can configure it in the Lens editor:

And a couple optional requests:

1. Could you post a screenshot of this dashboard with data in it?

2. This is a big dashboard (16 visualizations). The concern with these is that they can load really slowly if the customer has a lot of data. All visualizations (including tables) (re)render every time the user navigates to the dashboard/adds a filter/changes the time range, regardless of whether or not they are above the fold. Is there a logical way it could make sense to break this up into smaller dashboards connected by hyperlinks?

[drewdaemon]

Thanks for making the visualization type change!

[kaiyan-sheng]

@vinit-elastic I agree, if we want to change agent.yml we should simply do it in one go for all packages instead @kaiyan-sheng ?

@P1llus For all packages, do you mean all integrations in AWS? I'm thinking to do that since I just did it for the ec2 integration. I think we should use ECS fields (with ecs.yml) when they are already defined in ECS so we don't have to duplicate the definition in agent.yml.

[kaiyan-sheng]

@vinit-elastic How about lets keep the duplicated ECS fields for now, and make a separate PR to remove all the duplicate fields from agent.yml instead?

[vinit-chauhan]

@vinit-elastic How about lets keep the duplicated ECS fields for now, and make a separate PR to remove all the duplicate fields from agent.yml instead?

Hey @kaiyan-sheng, if we keep the duplicate fields as they are, then the elastic-package check command gives an error. because both files have different types of descriptions and there is a type change in the newer Stack version as well as the ECS version. The same has been discussed with @P1llus earlier and we came to a conclusion that keep agent.yml as it is and remove duplicate fields in the ECS file.

[kaiyan-sheng]

@vinit-elastic OK! Thanks for the explanation. I will make a separate PR later after this one gets in to fix the agent.yml and ecs.yml then!

[kaiyan-sheng]

I made the change for removing duplicate fields from agent.yml and use ecs.yml instead in PR: #4657

[vinit-chauhan]

Hey @kaiyan-sheng - I saw your PR, appreciate your effort.
However, we are doing the opposite in all the connectors. That is, we keep the agent.yml file same and remove the duplicates from ecs.yml. so that agent.yml is the same across all the integrations. So could you please do the same? Thanks!

[kaiyan-sheng]

@vinit-elastic @P1llus Sorry thats the part I don't quite get. Why we are redefining ECS fields in agent.yml files? For example cloud.availability_zone. Unless we are using cloud.availability_zone in a different way than what ECS field defines, we should keep it in ecs.yml to leverage ECS as much as we can.

[P1llus]

@kaiyan-sheng Its mostly because all the integrations have been setup like this as far as I know, the fields are simply copied from the other integrations.

We could change it, but I rather keep it out of this PR and do it over all integrations if you want. There is plenty of ECS fields in agent.yml from before.
I am unsure what the difference is, its still an ECS field, we are not redefining it, its simply in another file.

[kaiyan-sheng]

@P1llus and I had a conversation offline and decides to merge my PR for moving all fields into ecs.yml first to resolve this issue. Sorry @vinit-elastic you have to rebase your PR and sorry for blocking it for a while... The part for adding support for inspector data stream looks good to me. Thank you both!!