fix: gracefully handle JIT config failures and terminate unconfigured instance

[Brend-Smits]

This pull request enhances the robustness and reliability of the GitHub Actions runner scaling logic by improving error handling and retry mechanisms for GitHub API calls. It introduces the @octokit/plugin-retry plugin to automatically retry failed API requests, adds detailed logging for retry attempts, and ensures that failures in creating JIT configs for individual runner instances do not halt the entire scaling process. Additionally, new tests are added to verify handling of various API failure scenarios.

GitHub API client improvements:

• Added @octokit/plugin-retry to dependencies (package.json) and integrated it into the Octokit client initialization to enable automatic retries for failed GitHub API requests. [1] [2] [3]
• Configured the retry plugin to log detailed warnings on each retry attempt, including the HTTP method, URL, error message, and status code.

Error handling and resilience in JIT config creation:

• Updated createJitConfig in scale-up.ts to catch and log errors for individual runner instances when creating JIT configs, allowing the process to continue for remaining instances and logging a summary of failed attempts at the end. [1] [2]
• Instances that failed to generate a configuration, will now be terminated to avoid generating waste.

Testing improvements:

• Added comprehensive tests to scale-up.test.ts to verify correct behavior when GitHub API calls fail for some instances, including retryable errors (e.g., 5xx), non-retryable errors (e.g., 4xx), and partial failures, ensuring only successful JIT configs are stored.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details
npm/@octokit/plugin-retry	8.0.3	🟢 6.9	Details Check Score Reason Security-Policy 🟢 9 security policy file detected Code-Review 🟢 10 all changesets reviewed Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Maintained 🟢 5 6 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 5 Binary-Artifacts 🟢 10 no binaries found in the repo Pinned-Dependencies 🟢 5 dependency not pinned by hash detected -- score normalized to 5 Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected License 🟢 10 license file detected Fuzzing ⚠️ 0 project is not fuzzed Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ -1 internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md Packaging 🟢 10 packaging workflow detected SAST 🟢 10 SAST tool is run on all commits
npm/@octokit/plugin-retry	8.0.3	🟢 6.9	Details Check Score Reason Security-Policy 🟢 9 security policy file detected Code-Review 🟢 10 all changesets reviewed Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Maintained 🟢 5 6 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 5 Binary-Artifacts 🟢 10 no binaries found in the repo Pinned-Dependencies 🟢 5 dependency not pinned by hash detected -- score normalized to 5 Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected License 🟢 10 license file detected Fuzzing ⚠️ 0 project is not fuzzed Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ -1 internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md Packaging 🟢 10 packaging workflow detected SAST 🟢 10 SAST tool is run on all commits

Scanned Files

• lambdas/functions/control-plane/package.json
• lambdas/yarn.lock

[npalm]

@Brend-Smits Thx