Improve scalability of get-license action

[DaveCTurner]

Today this action runs on the transport worker thread and forwards the
request on to the master by default. It turns out that Elastic Agent
uses this API as a readiness check whenever opening a new connection, so
a thundering herd of 1000s of agents can prevent the transport worker
threads from doing more useful work for far too long, leading to high
latency and timeouts.

This commit changes the default behaviour to run the action on the local
node rather than forwarding to the master (although the option remains
to specify ?local=false) and dispatches the work off of the transport
worker early.

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[elasticsearchmachine]

Hi @DaveCTurner, I've created a changelog YAML for you.

[ywangd]

LGTM

I think it means a default GetLicense request now forks twice: First in its REST handler and second time in TransportMasterNodeAction. I think it's fine. Just wanted to call it out explicitly.

[DaveCTurner]

a default GetLicense request now forks twice

Yes you're right; I'd forgotten about that second dispatch in TransportMasterNodeAction which happens however it's invoked.

It concerns me slightly that any non-local TransportMasterNodeAction will do all the transport-layer security work on a transport worker thread on the master - this is itself surprisingly expensive even if the eventual action runs elsewhere. Hopefully there aren't too many other thundering-herds trying to call such actions, and/or we can eliminate most of this work from ever even reaching the master as described in #101805.