feat(giskard-checks): add Sentiment check using textblob#2386
feat(giskard-checks): add Sentiment check using textblob#2386Koushik-Salammagari wants to merge 20 commits intoGiskard-AI:mainfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new Sentiment check to the giskard-checks library, leveraging the textblob library for local sentiment analysis without requiring LLM API calls. The implementation includes a new nlp_metrics.py module, integration into the package's built-in checks, and a comprehensive suite of unit tests. The nlp optional dependency group was also added to pyproject.toml. Review feedback focuses on improving type safety by using defined aliases, removing redundant type-ignore comments, and enhancing the clarity of success messages when score boundaries are partially specified.
| ) | ||
|
|
||
|
|
||
| def _polarity_to_label(polarity: float) -> str: |
| default="trace.last.outputs", | ||
| description="JSONPath expression to extract the text from the trace.", | ||
| ) | ||
| expected: _SENTIMENT_LABELS | None = Field( # type: ignore[valid-type] |
There was a problem hiding this comment.
| if self.min_score is not None or self.max_score is not None: | ||
| success_parts.append( | ||
| f"Score {polarity:.4f} is within the required range " | ||
| f"[{self.min_score}, {self.max_score}]." | ||
| ) |
There was a problem hiding this comment.
When either min_score or max_score is None, the success message currently displays None in the range (e.g., [0.1, None]). It would be clearer to use the actual polarity bounds (-1.0 and 1.0) as defaults in the message for better readability.
| if self.min_score is not None or self.max_score is not None: | |
| success_parts.append( | |
| f"Score {polarity:.4f} is within the required range " | |
| f"[{self.min_score}, {self.max_score}]." | |
| ) | |
| if self.min_score is not None or self.max_score is not None: | |
| low = self.min_score if self.min_score is not None else -1.0 | |
| high = self.max_score if self.max_score is not None else 1.0 | |
| success_parts.append( | |
| f"Score {polarity:.4f} is within the required range [{low}, {high}]." | |
| ) |
2c704c1 to
fa5610f
Compare
fa5610f to
65318c9
Compare
|
Hi team! This PR is ready for review — could a maintainer add the |
|
Hi! Just checking in on #2386 — the Sentiment check PR. All the Gemini bot suggestions have been addressed and the code is ready for review. Could a maintainer add the |
…2364) Adds a new built-in `Sentiment` check that computes the polarity of model output text locally using textblob — no LLM API call required. - Polarity score computed in [-1.0, 1.0] via TextBlob - Label derived from polarity: positive (>0.05) / negative (<-0.05) / neutral - `expected` parameter asserts the sentiment label - `min_score` / `max_score` assert numeric polarity thresholds - `text` / `text_key` support direct values or JSONPath extraction from trace (defaults to `trace.last.outputs`) - Polarity score attached as a `Metric` on every result - Returns CheckStatus.ERROR with a helpful pip-install message when textblob is not installed - Registered as `"sentiment"` in the discriminated union registry; full Pydantic round-trip serialisation support - `textblob` added as `[nlp]` optional dependency in pyproject.toml Closes Giskard-AI#2364
- Use _SENTIMENT_LABELS return type on _polarity_to_label() for tighter type safety - Remove redundant # type: ignore[valid-type] on `expected` field — _SENTIMENT_LABELS is a valid Literal type alias - Replace None with -1.0/1.0 bounds in success message when only one of min_score/max_score is set, avoiding "[0.1, None]" output - Clean up unused TYPE_CHECKING import block
65318c9 to
418aebe
Compare
Description
Adds a new built-in
Sentimentcheck that validates the sentiment polarity of model output text using thetextbloblibrary — no LLM API call required.What's included
Sentimentcheck (builtin/nlp_metrics.py) — registered as"sentiment"in the discriminated union registrytests/builtin/test_sentiment.py) — covering all acceptance criteria from the issuetextblobadded under[project.optional-dependencies] nlpinpyproject.tomlbuiltin/__init__.pyand the top-levelgiskard.checksnamespaceuv.lockupdatedFeatures
[-1.0, 1.0]viaTextBlob(runs locally, no network call)"positive"(>0.05) /"negative"(<-0.05) /"neutral"expected: Literal["positive", "negative", "neutral"] | None— asserts the sentiment labelmin_score/max_score— asserts numeric polarity thresholds (both optional, each independently)text/text_key— supports direct values or JSONPath extraction from trace (default:trace.last.outputs)Metricon every resultCheckStatus.ERRORwith a helpfulpip install 'giskard-checks[nlp]'message whentextblobis not installedExample usage
Install the optional dependency
pip install 'giskard-checks[nlp]'Related Issue
Closes #2364
Type of Change
Checklist
uv.lockrunninguv lock