Rename config to policy and dataset to datastream

[mtojek]

What does this PR do?

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all datasets collect metrics or logs.

Author's Checklist

• Rename config template to policy template
• Rename dataset to data_stream
• Bump up release versions of integrations
• Make sure there are no "hacked" dependencies.

How to test this PR locally

mage clean
mage build
cd testing/environments
docker-compose -f snapshot.yml build package-registry
docker-compose -f snapshot.yml up --force-recreate package-registry

package-registry should be exposed under http://localhost:8080 .

Related issues

Screenshots

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #280 updated]

• Start Time: 2020-10-01T15:40:57.797+0000

• Duration: 4 min 34 sec

[jen-huang]

Hi @mtojek, thanks for helping me get my registry running with this branch. I found that the package info endpoint doesn't contain policy_templates or data_streams properties for any of the packages I checked (doesn't contain config_templates or datasets either). Example: http://localhost:8080/package/system/0.7.0/

Are you seeing the same?

[ycombinator]

@jen-huang I might be off here but is the registry you are running built with this PR's changes in it: elastic/package-registry#641?

[mtojek]

@jen-huang , confirmed (I see the same results).

It looks that you need to checkout the PR @ycombinator posted above (elastic/package-registry#641), then:

docker build . -t docker.elastic.co/package-registry/package-registry:wip

In testing/environments, you need to update the reference to the Docker image:

FROM docker.elastic.co/package-registry/package-registry:wip

COPY testing/environments/package-registry.config.yml /package-registry/config.yml
COPY build/integrations /packages/integrations

and finally:

docker-compose -f snapshot.yml build package-registry
docker-compose -f snapshot.yml up --force-recreate package-registry

[jen-huang]

Thanks both!! I am seeing the correct package info now. Will continue with my testing.

[jen-huang]

Another issue found: I think elastic/package-registry#641 is missing datasets.name->data_streams.datasets property renaming from the package spec changes.

Example from system package info:

  "data_streams": [
    {
      "type": "logs",
-      "name": "system.auth",
+      "dataset": "system.auth",
      "title": "System auth logs",
      "release": "experimental",
      "ingest_pipeline": "default",
      ...
    },
    ...
  ]

[jen-huang]

I was able to monkey patch my local package-registry to return data_stream.dataset property. Things are looking promising.

[jen-huang]

Didn't review actual code but tested this with elastic/kibana#78699 and it works well 👍

[jonathan-buttner]

I looked over the changes and they look good 👍