[ML] Preserve lost thread context in node inference action

[davidkyle]

If APM tracing is enabled the lost context meant the trace was never stopped and the span never released. After many calls to the action this small memory leak can amount to a substantial amount of memory potentially causing out of memory errors.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @davidkyle, I've created a changelog YAML for you.

[davidkyle]

The thread context was meant to be preserved by this code

elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/job/process/ProcessWorkerExecutorService.java

Line 49 in f1b48de

boolean added = queue.offer(contextHolder.preserveContext(command));

However there are 2 problems with that:

1. If the request times out it the listener is called on a different thread

   elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/inference/deployment/AbstractPyTorchAction.java

   Line 60 in 9ec61d5

   this.timeoutHandler = threadPool.schedule(

2. The Inference action is always completed on the result processor thread

   elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/inference/deployment/InferencePyTorchAction.java

   Line 149 in 9ec61d5

   ActionListener.wrap(

Anomaly detection uses the same ProcessWorkerExecutorService class but the thread context is preserved in that case because the same thread is always used.

[jonathan-buttner]

I tested this locally and it is working. Dave and I chatted about how to create a end to end test for this. We'll follow up on that later.

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.18
✅	9.0
✅	9.1
✅	8.19