Disable logging in ClusterFormationFailureHelper on shutdown.

[JeremyDahlgren]

Modifies Coordinator to enable logging in ClusterFormationFailureHelper when started and disables logging in ClusterFormationFailureHelper when stopped. The warning scheduler handling and invariant check in the Coordinator are left as is, with the logging boolean set independently, eliminating the need to hold the mutex in doStop() when Coordinator.stop() is called when the Node is shutdown.

Closes #105559.

[JeremyDahlgren]

Initially it seemed that a

    synchronized(mutex) {
        clusterFormationFailureHelper.stop();
    }

block in Coordinator::doStop() would be sufficient, but besides needing to hold the lock when the Node is being shut down, it seems a race condition could exist where the failure helper is stopped on shutdown but another thread immediately calls becomeCandidate() and starts it back up?

Also if calls to ClusterFormationFailureHelper start() and stop() were not synchronized it looks like there is the potential for a NullPointerException in start() if the warningScheduler is set to null between the assignment of a new WarningScheduler instance and the next line that calls scheduleNextWarning (Intellij warns about this):

    public void start() {
        assert warningScheduler == null;
        warningScheduler = new WarningScheduler();
        warningScheduler.scheduleNextWarning();
    }

    public void stop() {
        warningScheduler = null;
    }

[elasticsearchmachine]

Hi @JeremyDahlgren, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

Neat solution, LGTM