[azure] Ensure event.kind is correctly set for pipeline errors

[MakoWish]

What does this PR do?

Modify azure integration to correctly set event.kind for pipeline errors and ensure error.message is an array.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

•  

How to test this PR locally

Related issues

• For sei: ensure packages set error.kind to "pipeline_error" in on_failure processor #6582

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-08-10T21:53:35.966+0000

• Duration: 14 min 41 sec

Test stats 🧪

Test	Results
Failed	0
Passed	83
Skipped	0
Total	83

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[zmoog]

Hey @MakoWish, thank you for taking the time to work on this PR, #7043, and #7039!

After a quick look, I can see setting event.kind with the pipeline_error value as a net improvement with the proper semantics since this an allowed value in ECS.

I am unsure about setting error.message as an array. Many integrations (for example, the generic Event Hub integration) set this field as a keyword, and changing it into an array could be a breaking change.

Please let me know what you think.

[MakoWish]

Hi @zmoog,

I had worked with @efd6 on #6582, and that is what he had set for all the pipelines he worked on. I just followed suit, but I do lean more toward having an array, because it is possible to have multiple pipeline errors if multiple pipelines are in use. In addition to potential pipeline errors being written to error.message, the entire idea behind #6582 was to allow putting error messages from the actual event into that field as is noted in the ECS Field Reference and as was discussed in #6258. If we are not allowing an array, it would overwrite an error message that may have been put there from the event itself.

Eric

[zmoog]

@MakoWish got it. I'm not questioning the usefulness of having an array of errors vs. a single one.

I think we are good if there are no backward compatibility issues in switching from keyword to array of strings for existing data streams. @efd6 I guess we can switch from keyword to array of string without problems, right?

[MakoWish]

As the CI currently functions, events cannot set a value to error.message. Any attempt to do so will cause the CI to fail (the reason for #6582 and all these PR's). Since events cannot currently utilize error.message, I don't believe this would be a breaking change, but yeah... let's see what @efd6 says.

[efd6]

@zmoog SEI is making all error.message fields arrays. It's not a breaking change since arrays of scalar fields are treated the same as scalar fields for search. So long as the pipeline is handling the field consistently, there should be no problem.

[zmoog]

Thank you, @efd6!

@MakoWish I asked the other cloud monitoring team members to provide their feedback before moving forward.

[efd6]

/test

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (10/10)	💚
Files	86.364% (19/22)	❕
Classes	86.364% (19/22)	❕
Methods	83.333% (155/186)	❕
Lines	85.003% (2817/3314)	❕
Conditionals	100.0% (0/0)	💚

[zmoog]

@MakoWish, my apologies for this delay 🙇

After discussing this with the other cloud monitoring team members, we think this is nice addition to the integration.

✅ ing them!

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!