Fix service destination template file name

[gregkalapos]

@carsonip found this issue - the file name of the metrics-service_destination.[xyz]@template templates did not have the .otel postfix.

This PR fixes the file names - the index pattern itself was correct, so that's untouched and I expect those continue to work.

I bumped the priority of the renamed files by 1 to avoid issues in clusters where the old templates are used.

[lahsivjar]

The resources.yml version also needs to be bumped from 10 to 11

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @gregkalapos, I've created a changelog YAML for you.

[dakrone]

I'm not sure we should just rename these files in a minor version and remove the old ones entirely.

What if a user had made modifications to the installed metrics-service_destination.60m@template template? When they upgraded to 9.2.0, the new metrics-service_destination.60m.otel@template would have a higher priority, and they could end up in a situation where they lose their customizations when the data stream rolls over.

I know we tell users to customize things in the …@custom variants, but that hasn't stopped users from changing these as they desire in the past.

Are these data streams ones where we know the user won't be customizing these template files? Are users going to complain because they tested something in 9.1.x which no longer works in a separate 9.2.0 cluster because the metrics-service_destination.60m@template template no longer exists?

[gregkalapos]

I'm not sure we should just rename these files in a minor version and remove the old ones entirely.

What if a user had made modifications to the installed metrics-service_destination.60m@template template? When they upgraded to 9.2.0, the new metrics-service_destination.60m.otel@template would have a higher priority, and they could end up in a situation where they lose their customizations when the data stream rolls over.

I know we tell users to customize things in the …@custom variants, but that hasn't stopped users from changing these as they desire in the past.

Are these data streams ones where we know the user won't be customizing these template files? Are users going to complain because they tested something in 9.1.x which no longer works in a separate 9.2.0 cluster because the metrics-service_destination.60m@template template no longer exists?

I agree this is a risk of doing it this way. We had a slack thread with @felixbarny and @carsonip and we were leaning towards accepting this risk. What we store in these data streams are derived metrics created by the elasticapm connector which is used by the APM UI. I'd say there isn't anything to customize here for users. I think if anything, they could break things by changing these templates :)

Having said that, indeed, we'll break customization. I'd accept this, because 1) this is a type of data stream that users should not touch, we also set hidden to true on these and 2) they still have then the option to move customization to @custom.

Maybe list other options on how to proceed:

• Keep as is, and leave these template without the .otel postfix (I don't think this is nice)
• Wait with this until the next major
• Keep the template and add another one. But here, we'd still have the issue with priority, right? If we want to fix it, we'd need a template with a higher priority than the current one, but the index pattern is the same, so we'd still kill customizations. Or would there be a solution here?

I think overall the best option is to just do the change now. What do others think?

[simitt]

My 2cents (from the perspective of MOTel): I am ok with this change, if it is tested that it doesn't break anything on update if there is no customization applied. MOTel just went GA and the risk of any customization is still very low.

[dakrone]

Alright, thanks for the discussion and background. I appreciate the history, and I think it's okay to go with the change now, given the info.