Add GCP package

[marc-gr]

What does this PR do?

Adds GCP package

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all datasets collect metrics or logs.

Screenshots

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #459 updated

• Start Time: 2021-03-05T12:14:54.615+0000

• Duration: 12 min 33 sec

• Commit: 81927bb

Test stats 🧪

Test	Results
Failed	0
Passed	23
Skipped	0
Total	23

Trends 🧪

[andrewkroh]

I like that you converted as much as possible to Ingest Node.

The Google pub/sub service has an emulator. Perhaps the emulator can be loaded with our json logs and then the e2e tests pull from the emulator's topics.

The reason the logfile input exists in these GCP data streams is to allow for testing. By including it in the UI I think we make the usage less clear. We might be able to remove it since there is an alternative means of testing via the emulator. However some different services (like s3+sqs) don't have a good emulation option so logfile is still needed IMO. It might be useful to have a way for an input to be available only for testing.

[marc-gr]

I like that you converted as much as possible to Ingest Node.

The Google pub/sub service has an emulator. Perhaps the emulator can be loaded with our json logs and then the e2e tests pull from the emulator's topics.

The reason the logfile input exists in these GCP data streams is to allow for testing. By including it in the UI I think we make the usage less clear. We might be able to remove it since there is an alternative means of testing via the emulator. However some different services (like s3+sqs) don't have a good emulation option so logfile is still needed IMO. It might be useful to have a way for an input to be available only for testing.

I will give it a try, and I opened an issue elastic/package-spec#97 for what you are mentioning.

[marc-gr]

We have to wait for the changes in elastic/beats#23215 to be able to use the emulator here

[andrewkroh]

jenkins, run tests

Now that alternative_host has been merged to Filebeat let's see how this goes.

[andrewkroh]

@marc-gr This needs an update to change config.yml to test-pubsub-config.yml. ref: https://github.com/elastic/package-spec/blob/6435a2f6b3e0b92c11ffb4531a0ad3f575093d3f/versions/1/data_stream/_dev/test/system/spec.yml#L6

[adriansr]

For the logfile input, you may want to set enabled: false in the manifest.yml, so it's not enabled by default in the configuration.

Edit: nevermind, I see that you've dropped the input altogether.

[mtojek]

@marc-gr I see that you pushed a lot of Go code here, not sure if this is the expected way to go. In general it would be better to depend on DSLs/manifests/configs.

If you need to create some resources on GCP, you can try to use the Terraform service deployer: https://github.com/elastic/elastic-package/blob/master/docs/howto/system_testing.md#terraform-service-deployer

Here is the appliance for AWS: https://github.com/elastic/integrations/tree/master/packages/aws/data_stream/ec2_metrics/_dev/deploy/tf

[marc-gr]

@marc-gr I see that you pushed a lot of Go code here, not sure if this is the expected way to go. In general it would be better to depend on DSLs/manifests/configs.

If you need to create some resources on GCP, you can try to use the Terraform service deployer: https://github.com/elastic/elastic-package/blob/master/docs/howto/system_testing.md#terraform-service-deployer

Here is the appliance for AWS: https://github.com/elastic/integrations/tree/master/packages/aws/data_stream/ec2_metrics/_dev/deploy/tf

Finally we enhanced akroh/stream to support GCP pubsub as an output, so we moved the go code there and just using docker here 👍

[mtojek]

Finally we enhanced akroh/stream to support GCP pubsub as an output, so we moved the go code there and just using docker here 👍

Thanks for the information. Could you please share details behind this decision? I'm wondering if it's because of the fact that TF service deployer is not feasible for your use cases.

Another thought: do you plan to move akroh/stream under elastic org? I know it's Andrew's repository, but I believe in general we should depend on company's ownership.

[andrewkroh]

Thanks for the information. Could you please share details behind this decision? I'm wondering if it's because of the fact that TF service deployer is not feasible for your use cases.

I think it would be valuable to have a test like this too. It would serve as an integration test against the real service and could catch different types of issues than the emulator based testing with static logs. If the TF setup configures the actual services to write logs that's even better since it would let us catch when log formats changes. And lastly it could serve as a pseudo reference for describing the configuration steps necessary to setup GCP to use the package.

Another thought: do you plan to move akroh/stream under elastic org? I know it's Andrew's repository, but I believe in general we should depend on company's ownership

I agree. I wasn't sure whether it was going to be useful at first so I opted for a personal repo where I could more independently setup CI. I'll work on getting this moved over and have Jenkins setup.

[andrewkroh]

LGTM. Nice work with all the testing improvements to allow the emulator to be used.

[andrewkroh]

fyi: You can also pass any option via env vars:

environment:
  - STREAM_GCPPUBSUB_CLEAR=true
  - STREAM_GCPPUBSUB_PROJECT=audit