[GitLab] Update params mapping for nested object arrays

[tehbooom]

Proposed commit message

For the GitLab API datastream the params field is an array of objects that are dynamic. The fields were broken into key and value fields but broke when the value was another object.

This PR breaks each key value into their own field and adds field mappings for those fields.

For example this

"params": [
    {
        "key": "private_token",
        "value": "[FILTERED]"
    }
]

is now this with correct field mappings.

"params": [
    {
        "private_token": "[FILTERED]"
    }
]

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes [GitLab]: gitlab.api logs with nested key-value pairs fail to index #11941

Screenshots

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[chemamartinez]

I have noticed that the production data stream already handles this issue with a different approach. In that case, value is renamed to value_json and mapped as an object in case it is not a string.

I do prefer the solution given in this PR, but it makes me wonder a couple of things:

• Shall we unify the criteria here? that would make the integration more consistent.
• If we follow this approach, we could cause a breaking change in the other data stream by stopping using the value_json field.

[tehbooom]

I have noticed that the production data stream already handles this issue with a different approach. In that case, value is renamed to value_json and mapped as an object in case it is not a string.

I do prefer the solution given in this PR, but it makes me wonder a couple of things:

• Shall we unify the criteria here? that would make the integration more consistent.
• If we follow this approach, we could cause a breaking change in the other data stream by stopping using the value_json field.

I do prefer the solution in this PR as I think it presents the fields in a cleaner way and easier to understand. However, that does require an adjustment to the other datastream to ensure they line up correctly.

[chemamartinez]

I have noticed that the production data stream already handles this issue with a different approach. In that case, value is renamed to value_json and mapped as an object in case it is not a string.
I do prefer the solution given in this PR, but it makes me wonder a couple of things:

• Shall we unify the criteria here? that would make the integration more consistent.
• If we follow this approach, we could cause a breaking change in the other data stream by stopping using the value_json field.

I do prefer the solution in this PR as I think it presents the fields in a cleaner way and easier to understand. However, that does require an adjustment to the other datastream to ensure they line up correctly.

Yes, me too. I have checked that these value_json params are not being used in any dashboards, and in this particular case I think it is unlikely that anyone is using these fields to create custom assets. I'd go with the change for both data streams.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 5bb6f3bb2d39e98aedde6f41ec0de79d7d3c932f

History

• 💚 Build #20359 succeeded bd54f962675ebb64ed4ec4dd76f5e44360321eef
• 💔 Build #20357 failed 84acd66fba5e6a3e7413c7ac67f8e0a78a685102
• 💚 Build #20231 succeeded ace0fb45ab4d0687d2dbe66a5d0f1fa85fff3cfa

[chemamartinez]

LGTM!

[elastic-vault-github-plugin-prod]

Package gitlab - 2.0.0 containing this change is available at https://epr.elastic.co/package/gitlab/2.0.0/