[linux] Avoid using dynamic templates for flattened objects

[mrodm]

Proposed commit message

Update flattened field definitions to not create dynamic templates.

As they were previously, this dynamic template is created

  "dynamic_templates": [
    {
      "linux.memory.vmstat.*": {
        "path_match": "linux.memory.vmstat.*",
        "match_mapping_type": "object",
        "mapping": {
          "type": "flattened"
        }
      }
    },
  ],

but this definition does not match the fields under linux.memory.vmstat.

With these changes:

        "linux": {
          "properties": {
            "memory": {
              "properties": {
                "vmstat": {
                  "type": "flattened"
                }

Example of errors found (buildkite link):

test case failed: one or more errors found in mappings in metrics-linux.memory index template:
[0] field "linux.memory.vmstat.allocstall_device" is undefined: field definition not found
[1] field "linux.memory.vmstat.allocstall_dma" is undefined: field definition not found
[2] field "linux.memory.vmstat.allocstall_dma32" is undefined: field definition not found
[3] field "linux.memory.vmstat.allocstall_movable" is undefined: field definition not found
[4] field "linux.memory.vmstat.allocstall_normal" is undefined: field definition not found
...

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Test with elastic-package enabled mappings: https://buildkite.com/elastic/integrations/builds/21927
• Remove debug changes .buildkite folder and go.mod/go.sum files.

How to test this PR locally

Run elastic-package with mappings validation enabled and run system tests:

cd packages/linux
elastic-package stack up -v -d --version <kibana_constraint>
export ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=mappings
# Add defer-cleanup to check the data ingested in the corresponding indices/data streams
elastic-package test system -v --data-streams memory --defer-cleanup 900s

elastic-package stack down -v

Related issues

• Relates [system tests] Validate fields are documented with dynamic templates if needed elastic-package#2207
• Relates [system tests] Validate fields in transforms are documented based on mappings elastic-package#2341

[mrodm]

Documents ingested would be like:

    "linux": {
      "memory": {
        "vmstat": {
          "workingset_refault_file": 2138589,
          "direct_map_level3_splits": 44,
          "direct_map_level2_splits": 1491,
          "thp_swpout_fallback": 0,
          "htlb_buddy_alloc_success": 0,
          ...
        }
      }
    }

And the content of that field is directly a JSON (as expected for flattened fields):

{
  "workingset_refault_file": 2138589,
  "direct_map_level3_splits": 44,
  "direct_map_level2_splits": 1491,
  "thp_swpout_fallback": 0,
  "htlb_buddy_alloc_success": 0,
  ...
}

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[leehinman]

I'm not sure about the buildkite changes but the rest of it looks fine to me.

[mrodm]

I'm not sure about the buildkite changes but the rest of it looks fine to me.

Thanks!
Buildkite changes are going to be reverted before merging

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: d010e9f

History

• 💚 Build #21927 succeeded ba7e4d9
• 💚 Build #21863 succeeded 163c8ff
• 💔 Build #21803 failed 91c19775bc66a0f5c216bbb3be773344c6631ee8
• 💔 Build #21797 failed fa4ee98cdb3120955c73ccf9cea4d6f778ede314
• 💚 Build #21789 succeeded 09360c0dfa145fab5b5c4db98946c9086ecb4826
• 💚 Build #21674 succeeded 2a82ca4598ff9e70c945974d1071bd9d22d9eb32

cc @mrodm

[elastic-vault-github-plugin-prod]

Package linux - 0.7.1 containing this change is available at https://epr.elastic.co/package/linux/0.7.1/