Skip to content

feat(giskard-checks): add Sentiment check using textblob#2386

Open
Koushik-Salammagari wants to merge 20 commits intoGiskard-AI:mainfrom
Koushik-Salammagari:feat/sentiment-check
Open

feat(giskard-checks): add Sentiment check using textblob#2386
Koushik-Salammagari wants to merge 20 commits intoGiskard-AI:mainfrom
Koushik-Salammagari:feat/sentiment-check

Conversation

@Koushik-Salammagari
Copy link
Copy Markdown
Contributor

Description

Adds a new built-in Sentiment check that validates the sentiment polarity of model output text using the textblob library — no LLM API call required.

What's included

  • Sentiment check (builtin/nlp_metrics.py) — registered as "sentiment" in the discriminated union registry
  • 19 unit tests (tests/builtin/test_sentiment.py) — covering all acceptance criteria from the issue
  • Optional dependencytextblob added under [project.optional-dependencies] nlp in pyproject.toml
  • Exports — added to builtin/__init__.py and the top-level giskard.checks namespace
  • uv.lock updated

Features

  • Polarity score computed in [-1.0, 1.0] via TextBlob (runs locally, no network call)
  • Sentiment label derived from polarity: "positive" (>0.05) / "negative" (<-0.05) / "neutral"
  • expected: Literal["positive", "negative", "neutral"] | None — asserts the sentiment label
  • min_score / max_score — asserts numeric polarity thresholds (both optional, each independently)
  • text / text_key — supports direct values or JSONPath extraction from trace (default: trace.last.outputs)
  • Polarity score attached as a Metric on every result
  • Returns CheckStatus.ERROR with a helpful pip install 'giskard-checks[nlp]' message when textblob is not installed
  • Full Pydantic round-trip serialisation/deserialisation support

Example usage

from giskard.checks import Sentiment, Scenario

# Assert the response is positive
scenario = (
    Scenario(name="positive_response")
    .interact(inputs="How is our service?", outputs="Your service is excellent!")
    .check(Sentiment(expected="positive"))
)

# Assert polarity is within a numeric range
check = Sentiment(
    text="This product is fantastic!",
    expected="positive",
    min_score=0.1,
)

# Just measure, no assertion (always passes — useful for monitoring)
check = Sentiment()

Install the optional dependency

pip install 'giskard-checks[nlp]'

Related Issue

Closes #2364

Type of Change

  • 🚀 New feature (non-breaking change which adds functionality)

Checklist

  • I've read the CODE_OF_CONDUCT.md document.
  • I've read the CONTRIBUTING.md guide.
  • I've written tests for all new methods and classes that I created.
  • I've written the docstring in NumPy format for all the methods and classes that I created or modified.
  • I've updated the uv.lock running uv lock
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new Sentiment check to the giskard-checks library, leveraging the textblob library for local sentiment analysis without requiring LLM API calls. The implementation includes a new nlp_metrics.py module, integration into the package's built-in checks, and a comprehensive suite of unit tests. The nlp optional dependency group was also added to pyproject.toml. Review feedback focuses on improving type safety by using defined aliases, removing redundant type-ignore comments, and enhancing the clarity of success messages when score boundaries are partially specified.

)


def _polarity_to_label(polarity: float) -> str:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The return type of _polarity_to_label can be more specific by using the _SENTIMENT_LABELS type alias instead of a generic str.

Suggested change
def _polarity_to_label(polarity: float) -> str:
def _polarity_to_label(polarity: float) -> _SENTIMENT_LABELS:
default="trace.last.outputs",
description="JSONPath expression to extract the text from the trace.",
)
expected: _SENTIMENT_LABELS | None = Field( # type: ignore[valid-type]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The type: ignore[valid-type] comment appears to be unnecessary here, as _SENTIMENT_LABELS is a valid Literal type alias. Removing it helps maintain a cleaner codebase.

Suggested change
expected: _SENTIMENT_LABELS | None = Field( # type: ignore[valid-type]
expected: _SENTIMENT_LABELS | None = Field(
Comment on lines +239 to +243
if self.min_score is not None or self.max_score is not None:
success_parts.append(
f"Score {polarity:.4f} is within the required range "
f"[{self.min_score}, {self.max_score}]."
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

When either min_score or max_score is None, the success message currently displays None in the range (e.g., [0.1, None]). It would be clearer to use the actual polarity bounds (-1.0 and 1.0) as defaults in the message for better readability.

Suggested change
if self.min_score is not None or self.max_score is not None:
success_parts.append(
f"Score {polarity:.4f} is within the required range "
f"[{self.min_score}, {self.max_score}]."
)
if self.min_score is not None or self.max_score is not None:
low = self.min_score if self.min_score is not None else -1.0
high = self.max_score if self.max_score is not None else 1.0
success_parts.append(
f"Score {polarity:.4f} is within the required range [{low}, {high}]."
)
@Koushik-Salammagari
Copy link
Copy Markdown
Contributor Author

Hi team! This PR is ready for review — could a maintainer add the safe for build label so CI can run? Happy to address any feedback. Thanks!

@Koushik-Salammagari
Copy link
Copy Markdown
Contributor Author

Hi! Just checking in on #2386 — the Sentiment check PR. All the Gemini bot suggestions have been addressed and the code is ready for review. Could a maintainer add the safe for build label so CI can run? Happy to address any feedback. Thanks!

BotReleaser and others added 20 commits April 14, 2026 09:43
…2364)

Adds a new built-in `Sentiment` check that computes the polarity of
model output text locally using textblob — no LLM API call required.

- Polarity score computed in [-1.0, 1.0] via TextBlob
- Label derived from polarity: positive (>0.05) / negative (<-0.05) / neutral
- `expected` parameter asserts the sentiment label
- `min_score` / `max_score` assert numeric polarity thresholds
- `text` / `text_key` support direct values or JSONPath extraction
  from trace (defaults to `trace.last.outputs`)
- Polarity score attached as a `Metric` on every result
- Returns CheckStatus.ERROR with a helpful pip-install message when
  textblob is not installed
- Registered as `"sentiment"` in the discriminated union registry;
  full Pydantic round-trip serialisation support
- `textblob` added as `[nlp]` optional dependency in pyproject.toml

Closes Giskard-AI#2364
- Use _SENTIMENT_LABELS return type on _polarity_to_label() for
  tighter type safety
- Remove redundant # type: ignore[valid-type] on `expected` field —
  _SENTIMENT_LABELS is a valid Literal type alias
- Replace None with -1.0/1.0 bounds in success message when only
  one of min_score/max_score is set, avoiding "[0.1, None]" output
- Clean up unused TYPE_CHECKING import block
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment