Add PR label management to test verification skill - #33739
Conversation
Enhances the verification process by automatically adding or removing PR labels (`s/ai-reproduction-confirmed` and `s/ai-reproduction-failed`) based on test outcomes. Updates documentation to describe label behavior and integrates label updates into both quick and full verification modes.
📋 PR Finalization ReviewReview 1: Initial finalization check
|
There was a problem hiding this comment.
Pull request overview
This PR enhances the test verification skill by adding automated PR label management based on verification outcomes. The skill now adds s/ai-reproduction-confirmed when tests correctly fail without a fix (proving they catch the bug) or s/ai-reproduction-failed when tests pass without a fix (indicating they don't catch the bug).
Changes:
- Implemented label management function that updates PR labels based on test verification results
- Integrated label updates into both quick verification mode (test-only) and full verification mode (test + fix)
- Added comprehensive documentation explaining label behavior, colors, and when each label is applied
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 |
Added Update-VerificationLabels function and integrated it at all verification result points (both success and failure paths in quick and full modes) |
.github/skills/verify-tests-fail-without-fix/SKILL.md |
Added PR Labels section documenting the two labels, their colors, when they're applied, and their behavior during re-verification |
| if ($LASTEXITCODE -eq 0) { | ||
| Write-Host "✅ Labels updated successfully" -ForegroundColor Green | ||
| } else { | ||
| Write-Host "⚠️ Failed to update labels (may not have permission)" -ForegroundColor Yellow | ||
| } |
There was a problem hiding this comment.
The $LASTEXITCODE check on line 164 only reflects the exit code of the last gh pr edit command (line 162 - adding the label), not the gh pr edit command on line 157 (removing the label). If removing the label fails but adding succeeds, the function will report success even though the removal may have failed.
Consider checking $LASTEXITCODE after the remove operation as well, or tracking both operations separately to provide accurate feedback.
kubaflo
added
the
area-ai-agents
Address Copilot review feedback - track success of both DELETE and POST operations separately. If the DELETE fails but POST succeeds, report partial failure instead of full success to avoid misleading logs and potential conflicting label state.
Adds automatic PR label management to the verify-tests-fail-without-fix skill. After test verification completes, the skill now updates labels on the PR to indicate verification status. **New Labels:** - s/ai-reproduction-confirmed - Tests correctly FAIL without fix - s/ai-reproduction-failed - Tests PASS without fix **Files Changed:** - SKILL.md - Added PR Labels documentation section - verify-tests-fail.ps1 - Added Update-VerificationLabels function --------- Co-authored-by: Shane Neuville <shneuvil@microsoft.com>
Adds automatic PR label management to the verify-tests-fail-without-fix skill. After test verification completes, the skill now updates labels on the PR to indicate verification status. **New Labels:** - s/ai-reproduction-confirmed - Tests correctly FAIL without fix - s/ai-reproduction-failed - Tests PASS without fix **Files Changed:** - SKILL.md - Added PR Labels documentation section - verify-tests-fail.ps1 - Added Update-VerificationLabels function --------- Co-authored-by: Shane Neuville <shneuvil@microsoft.com>
Adds automatic PR label management to the verify-tests-fail-without-fix skill. After test verification completes, the skill now updates labels on the PR to indicate verification status.
New Labels:
Files Changed: