Feat 514: add comment value score

[priyanshuhaldar007]

What?

Closes #514

Adds a Value Score column to the comment moderation list table. Comments are now analyzed for how relevant and valuable they are to the article they're posted on, in addition to the existing toxicity and sentiment signals.

Why?

Toxicity and sentiment alone don't tell the full story of a comment's quality. A comment can be perfectly polite but still be spam, a generic "+1", or completely off-topic. This PR introduces a value score (0–1) so moderators can quickly identify substantive, on-topic contributions versus low-effort noise — making triage faster and more informed.

How?

The implementation follows the exact same pattern as the existing toxicity_score field end-to-end:

AI / Prompt layer

• Updated system-instruction.php to instruct the model to return a third field, value_score, scored 0–1 with clear band definitions (low/medium/high). The prompt now also passes post context (excerpt → AI summary → trimmed content fallback) so the model can actually assess relevance against the article.
• Added get_post_context() to Comment_Analysis.php to fetch and prepare that context, with a graceful 650-char truncation fallback on raw content.
• analyze_comment() now accepts and passes $post_id through to the prompt builder.

Schema & storage

• output_schema() and response_schema() both declare value_score as a nullable float (0–1). It's nullable for cases where the post content is unavailable or too sparse to judge relevance.
• sanitize_analysis_result() clamps the value to [0, 1] and preserves null.
• Results are stored in a new comment meta key META_VALUE_SCORE (_wpai_value_score) and returned in the ability response payload.

Comment Moderation UI (PHP)

• Added VALUE_SCORE_LOW / MEDIUM / HIGH constants and get_value_score_config() with the same range-bucket shape used by toxicity, so the frontend JS can resolve badges identically.
• New wpai_value_score column registered in add_columns() and add_sortable_columns().
• render_value_score_column() and render_value_score_badge() added, mirroring the toxicity equivalents.
• Filter dropdown added for value score levels.
• handle_sorting_and_filtering() extended to support wpai_value_score ordering via a meta query, same pattern as toxicity sorting.
• enqueue_assets() now passes value_score label config into window.aiCommentModerationData.labels.
• CSS: reused existing green/yellow/red badge classes by appending ai-badge--high-value, ai-badge--medium-value, and ai-badge--low-value to the existing selectors — no new colour definitions needed.

Frontend JS / TSX

• AnalysisResult type extended with value_score: number.
• Window declaration extended with the value_score labels shape.
• PendingComment type extended with valueScoreBadge: HTMLElement.
• getValueScoreDisplay() helper added alongside getToxicityDisplay(), using the same range-bucket lookup.
• updateBadges(), findPendingComments(), analyzeComment() all updated to handle the third badge — detection, processing state, result rendering, and failure state.

Use of AI Tools

AI assistance: Yes
Tool(s): Claude
Model(s): Claude Sonnet 4.6
Used for: Drafting this PR description from the git diff. All code was written and reviewed by me, with some contributions from the copilot for updating and generating doc blocks

Testing Instructions

1. Install and activate the plugin with the AI experiment enabled.
2. Create a post with meaningful content and leave a few comments — mix of on-topic, generic ("great post!"), and spam.
3. Navigate to Comments in wp-admin.
4. Confirm a Value column appears alongside Sentiment and Toxicity.
5. Trigger analysis (or wait for lazy analysis on page load). Verify badges appear with appropriate levels — high-value comments should show 🌟 High, spam/generic should show ✓ Low.
6. Test the Value Score filter dropdown — filtering by Low, Medium, or High should correctly narrow the list.
7. Test sorting by clicking the Value column header — ascending and descending should order comments by their score.
8. Confirm the dashboard comment excerpt pills also show the value score badge.

Screenshots or screencast

Before	After

Changelog Entry

Added - Value Score column to the comment moderation table, providing a relevance signal (0–1) for each comment based on how substantive and on-topic it is relative to the post it was left on.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @priyanshuhaldar007.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Unlinked contributors: priyanshuhaldar007.

Co-authored-by: dkotter <dkotter@git.wordpress.org>
Co-authored-by: jeffpaul <jeffpaul@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[dkotter]

Left a number of comments and also seeing quite a few lint failures that need fixed here. In addition, we should look to add both unit tests and update our E2E tests to ensure these changes are covered

[dkotter]

Any benefit to using the excerpt or summary over just using the full post content?

[dkotter]

Not sure we need to trim this. I understand it will save tokens but likely gives us worse results

[priyanshuhaldar007]

I have implemented changes based on the reviews. For the value score generation fallback, I'd like to know your preference on the method, whether to keep this implementation or simply pass the entire post content and not the excerpt or summary.

[jeffpaul]

@priyanshuhaldar007 FYI a merge conflict to resolve to help keep this moving along

[dkotter]

@priyanshuhaldar007

For the value score generation fallback, I'd like to know your preference on the method, whether to keep this implementation or simply pass the entire post content and not the excerpt or summary.

I would suggest testing both approaches and seeing what results you get. If you get more or less the same results by using the excerpt/summary, I'd say we're fine to keep that as-is to save tokens. But if you get more accurate / better results by passing in the full content, we should switch to that

[codecov]

Codecov Report

❌ Patch coverage is 69.59459% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.38%. Comparing base (6dcd0ab) to head (a10fcd5).

Files with missing lines	Patch %	Lines
...eriments/Comment_Moderation/Comment_Moderation.php	59.45%	45 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #681      +/-   ##
=============================================
- Coverage      78.44%   78.38%   -0.06%     
- Complexity      2454     2469      +15     
=============================================
  Files            104      104              
  Lines           9925    10041     +116     
=============================================
+ Hits            7786     7871      +85     
- Misses          2139     2170      +31     

Flag	Coverage Δ
unit	78.38% <69.59%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[priyanshuhaldar007]

I set up a testing framework to measure whether using a post's excerpt/summary or its full content actually changes the quality of the generated value scores.

Test setup

Created three sample posts covering common scenarios:

• A post with a strong excerpt and detailed content (baseline case)
• A post with a generic/minimal excerpt but a detailed body (to highlight missing context)
• A post with no excerpt (to test fallback behavior)

Added 18 comments across these posts to reflect realistic user interactions:

• 6 thoughtful, relevant comments (expected to score high)
• 6 short, generic comments (expected to score low)
• 4 spam or off-topic comments (expected to score low)
• 2 edge-case comments designed to reveal differences between excerpt and full-content analysis

Testing framework

Built a browser-based test harness that:

• Scores every comment twice—once using the excerpt and once using the full post content
• Compares the resulting value scores from both approaches
• Checks how closely each score matches the expected High/Medium/Low rating
• Calculates overall accuracy for each mode
• Recommends the better approach based on the final results

Handling API limitations

To avoid quota and rate-limit issues with OpenAI and Gemini Pro, the test runner:

• Waits 500ms between requests
• Processes comments one at a time instead of in parallel
• Handles failed API calls gracefully, making retries straightforward
• Uses a 300-second timeout for slower responses
• Continues running even if individual comment evaluations fail

Decision criteria

• If the accuracy difference is under 10%, we'll keep using excerpts to save tokens.
• If full-content analysis improves accuracy by 15% or more, we'll switch to passing the entire post.
• The results provide a clear data-backed trade-off between token usage and scoring accuracy.

This approach replaces assumptions with measurable results, making it easier to decide which context provides the best balance of accuracy and efficiency.

Results

Excerpt-based generation performed almost as well as full-content generation, with accuracy ranging from 80–100% compared to 80–87.5% for full content. This suggests we can reduce token usage without a noticeable drop in generation quality.

A significant number of test cases returned null responses (8–11 per test cycle). However, these were not due to poor model performance. They were caused by external API issues from AI providers, such as rate limits and temporary server errors (e.g., HTTP 502), which prevented responses from being generated. While these failures should be accounted for when evaluating overall system reliability, they do not reflect the quality of the underlying prompts or generation logic.

Excerpt mode offers better token efficiency, while full-content mode occasionally produces more consistent results. However, these findings are based on a relatively small number of tests (around 8–21 comment generations per cycle), so additional testing is needed before making a final recommendation.

I'm continuing to expand the test dataset, refine the prompts, and evaluate both excerpt and full-content generation across a wider range of scenarios. I'm also investigating retry and fallback strategies to handle transient API failures (such as rate limits and 502 errors) so the comparison reflects model performance rather than external provider issues. This should lead to a more reliable evaluation and a more stable content generation pipeline.

[priyanshuhaldar007]

@dkotter, shall I upload the browser-end test script and all the necessary details(post content and comments) here so that you can have a look at them on your end as well?

[dkotter]

@priyanshuhaldar007 Thanks for the detailed testing, that's super helpful! Based on the results you're seeing, I think we're fine sticking with the excerpt only for now.

I know there's some failing tests here that need fixed up but is there anything else you're tracking that needs done before this is ready for another review?