Retry logic causing duplicate label #4370

We use the action with PAT of the automation user, so it can trigger further actions. Seems that since #4321 was addressed in #4356 the labels are applied twice, exactly a second apart:

This in turn causes actions, dependent on labeling to fire twice.

Would it be possible to make the behavior configurable?