Add CLI for running and validating checks

[raghavtries]

Description

Add a giskard CLI for running and validating serialized scenarios and suites.

This PR adds:

• giskard run <file> for YAML/JSON scenario and suite definitions
• giskard validate <file> to validate definitions without execution
• giskard list checks to expose built-in check kinds
• output formats for rich terminal output, JSON, and JUnit XML

It also preserves support for serialized trace_type values and ensures python:... target resolution prefers the definition file directory over the current working directory when module names collide.

Related Issue

Closes #2376

Type of Change

• 📚 Examples / docs / tutorials / dependencies update
• 🔧 Bug fix (non-breaking change which fixes an issue)
• 🥂 Improvement (non-breaking change which improves an existing feature)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 🔐 Security fix

Checklist

• I've read the CODE_OF_CONDUCT.md document.
• I've read the CONTRIBUTING.md guide.
• I've written tests for all new methods and classes that I created.
• I've written the docstring in NumPy format for all the methods and classes that I created or modified.
• I've updated the uv.lock running uv lock (only applicable when pyproject.toml has been modified)

[gemini-code-assist]

Code Review

This pull request introduces a command-line interface for Giskard, enabling users to run, validate, and list checks using YAML or JSON definition files. The implementation includes a new giskard entry point, logic for dynamic Python target resolution, and support for multiple output formats like JSON and JUnit. Review feedback suggests simplifying the attribute resolution logic, ensuring parent directories are created before writing output files, and avoiding reliance on private internal registries for check discovery.

[gemini-code-assist]

The path.write_text call will fail if the parent directory of the output file does not exist. For a better user experience, consider ensuring the parent directory exists before writing.

Suggested change

	path.write_text(output, encoding="utf-8")
	path.parent.mkdir(parents=True, exist_ok=True)
	path.write_text(output, encoding="utf-8")

[kevinmessiaen]

@raghavtries Thanks for your submission. This look good but I feel like we could simplify a bit by leveraging Pydantic capabilities

[raghavtries]

Thanks for the details in the review, I addressed the feedback by moving scenario target, suite target, trace_type loading into Pydantic validators, along with simplifying CLI loading through TypeAdapter, keeping the serialized format canonical with checks/interacts. Also, adding public registered-kind/type accessors on Discriminated.

Also made YAML support optional and moved reusable CLI definitions into fixture files.

I also fixed output directory creation for --output.

[kevinmessiaen]

@gemini-code-assist Can you review the updated code?

[gemini-code-assist]

Code Review

This pull request introduces a command-line interface for Giskard checks, enabling the execution, validation, and listing of scenarios and suites from JSON or YAML definitions. Key additions include a robust Python reference resolution system, Pydantic-based validation for dynamic targets, and support for various output formats like JUnit and JSON. Review feedback highlighted the need to prevent over-aggressive clearing of the module cache during reference resolution and to ensure parent directories are created when writing JUnit reports.

[gemini-code-assist]

The logic for clearing sys.modules is too aggressive. It currently removes any cached module that is not found in the provided search paths (e.g., standard library modules like os or json if they are referenced in a python:... string). This forces unnecessary re-imports and can lead to unexpected side effects in long-running processes. The cache should only be cleared if a collision is detected with a module in the preferred search path (the definition file's directory).

Suggested change

	if _module_exists_in_search_path(module_name, preferred_search_path):
	if not _module_belongs_to_search_path(
	cached_module, preferred_search_path
	):
	sys.modules.pop(module_name, None)
	elif not _module_belongs_to_search_paths(cached_module, search_paths):
	sys.modules.pop(module_name, None)
	if _module_exists_in_search_path(module_name, preferred_search_path):
	if not _module_belongs_to_search_path(
	cached_module, preferred_search_path
	):
	sys.modules.pop(module_name, None)

[gemini-code-assist]

When using the junit format with an --output path, the parent directories are not automatically created. This can cause the command to fail if the specified output path includes directories that do not yet exist. For consistency with the json format (which uses _write_text_output), ensure the parent directory is created before calling to_junit_xml.

Suggested change

	elif args.format == "junit":
	xml = _build_suite_result(execution).to_junit_xml(path=output_path)
	if output_path is None:
	print(xml)
	elif args.format == "junit":
	if output_path:
	output_path.parent.mkdir(parents=True, exist_ok=True)
	xml = _build_suite_result(execution).to_junit_xml(path=output_path)
	if output_path is None:
	print(xml)

[raghavtries]

Hi there! I've made the needed edits from the Gemini review, please let me know if there's anything else I should change!