feat(cli): add 'add-all' command and auto_delete_added_files config

[sebastianbraun25]

Note

This PR was created in collaboration between a human and AI: implementation, tests, and
PR text were created by an AI assistant under the guidance and review of the human author.

Problem

The openkb add command supports adding directories recursively, but users who frequently download documents to /raw must manually invoke openkb add raw/ and then manually delete processed files. There is no batch processing command for the /raw staging directory, and no configuration option to auto-delete successfully ingested files and duplicates.

Solution / Changes

New CLI command:

• openkb add-all processes all files in the KB's raw/ directory recursively
• Ingests all supported file types (PDF, Markdown, DOCX, PPTX, XLSX, XLS, HTML, TXT, CSV)
• Returns summary of operation: Added, Skipped, Failed, Deleted counts

New configuration parameter:

• auto_delete_added_files (boolean, default: false)
• When enabled, both add and add-all automatically delete files after processing
• Deletes on both "added" (successful ingestion) and "skipped" (duplicate/already in KB)
• Preserves "failed" files to allow user retries
• Applies to all ingest methods: direct files, directories, and URLs

Implementation:

• Added _delete_if_auto_cleanup_enabled() helper function to handle cleanup logic
• Updated both add and add-all commands to use the helper function
• Updated docstrings to document cleanup behavior on both success and skip cases

Issues

• Resolves feat(cli): add-all command and auto_delete_added_files config #231

[sebastianbraun25]

Rewrote this branch's history to remove unrelated commits.

The branch had accidentally picked up commits from the (now-withdrawn, see
#229/#230) retry-timeout work as ancestors — likely from being built on top
of a local integration branch at the time rather than directly off
main. That made this PR's diff show unrelated changes to
openkb/agent/compiler.py and a new tests/test_compiler_retry.py
that have nothing to do with add-all/auto-delete.

Rebuilt cleanly from current main with only the two commits that
actually belong to this feature (add-all command + the
auto-delete-on-skipped follow-up fix). Diff now only touches
openkb/cli.py and openkb/config.py as expected. No functional
changes to the add-all feature itself — force-pushed the same branch name so
this PR stays open with its history/comments intact.