Summary

Add a built-in check that validates whether the output is well-formed JSON, with optional JSON Schema validation. This is critical for testing LLM structured output.

Motivation

Structured output from LLMs is increasingly common (OpenAI structured outputs, tool calls, function calling). Validating JSON correctness is a fundamental evaluation need. DeepEval ships JsonCorrectnessMetric and promptfoo ships is-json — this is table stakes.

Implementation Guide

Pattern to follow

Reference: libs/giskard-checks/src/giskard/checks/builtin/fn.py (for simple validation pattern)

Steps

1. Create libs/giskard-checks/src/giskard/checks/builtin/json_valid.py
2. Register with @Check.register("json_valid")
3. Implement async run(self, trace: Trace) -> CheckResult
4. Support:
   • key: JSONPathStr — JSONPath to extract the value to validate (default: trace.last.outputs)
   • schema: dict | None = None — optional JSON Schema to validate against
   • Parse the output as JSON; if it fails, return CheckResult.failure()
   • If schema is provided, validate against it (use jsonschema library or stdlib)
5. Add to __init__.py exports
6. Add tests in tests/builtin/test_json_valid.py

Dependencies

• Consider adding jsonschema as an optional dependency for schema validation
• Basic JSON parsing uses stdlib json module (no extra dependency)

Example usage

from giskard.checks import JsonValid, Scenario

# Basic JSON validation
scenario = (
    Scenario(name="structured_output")
    .interact(inputs="Return a JSON object", outputs='{"name": "Alice", "age": 30}')
    .check(JsonValid())
)

# With schema validation
schema = {
    "type": "object",
    "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
    "required": ["name", "age"]
}
scenario = (
    Scenario(name="schema_validation")
    .interact(inputs="Return user data", outputs='{"name": "Alice", "age": 30}')
    .check(JsonValid(schema=schema))
)

Acceptance Criteria

• Passes for valid JSON strings
• Fails for invalid JSON (with helpful error message showing parse error)
• Optional JSON Schema validation works
• JSONPath extraction works for nested trace values
• Check is registered and serializable
• Tests cover: valid JSON, invalid JSON, nested objects, arrays, schema pass/fail, JSONPath