Skip to content

feat(config): pass through global LiteLLM settings via a litellm: config block - #138

Merged
KylinMountain merged 1 commit into
mainfrom
feat/litellm-config-passthrough
Jun 25, 2026
Merged

feat(config): pass through global LiteLLM settings via a litellm: config block#138
KylinMountain merged 1 commit into
mainfrom
feat/litellm-config-passthrough

Conversation

@KylinMountain

Copy link
Copy Markdown
Collaborator

What

Adds an optional litellm: mapping to .openkb/config.yaml whose keys are passed through verbatim onto the litellm module (litellm.<key>):

litellm:
  drop_params: true     # let LiteLLM drop params a provider rejects
  num_retries: 3

Why

Closes #137. A user on a local Ollama backend hit litellm.UnsupportedParamsError because OpenKB sends parallel_tool_calls, which Ollama doesn't support. The fix LiteLLM suggests is litellm.drop_params=True, but OpenKB only forwarded model (plus the existing extra_headers/timeout whitelist) — there was no way to set it from config, only the LITELLM_DROP_PARAMS env var. This generalizes the existing per-key whitelist to a verbatim passthrough so any LiteLLM module global can be set from config.

How

  • config.resolve_litellm_settings() — validates the value is a mapping, drops non-string keys, forwards values unchanged (no validation/coercion; the user owns them).
  • cli._apply_litellm_settings()setattrs each key onto litellm inside _setup_llm_key, so it covers every command and every call path routed through LiteLLM (these are module-level globals). Two guards:
    • an unknown key (typo / litellm-version mismatch) is skipped with a warning instead of silently creating a dead attribute;
    • a key whose current value is callable (e.g. litellm.completion) is refused — overwriting a function with a config scalar would brick later calls.
  • Warnings go through the logger (stderr, like the sibling resolvers), not click.echo, so a typo can't corrupt piped stdout (e.g. openkb query > out.txt).
  • Process-wide and sticky on purpose: settings are applied, never reset (documented in the docstring, README, and config.yaml.example).

Notes

  • The currently-pinned PageIndex hardcodes litellm.drop_params=True per call during long-PDF indexing, so a user's drop_params: false won't make indexing strict — noted in the docs. (A separate upstream PageIndex change stops it from mutating the shared global.)

Tests

tests/test_config.py (resolver: mapping/non-mapping/non-string-key + caplog on the warning paths) and a new tests/test_llm_config_passthrough.py (verbatim apply, unknown-key + callable guards, sticky no-reset, env-isolated end-to-end). Full suite green.

@KylinMountain
KylinMountain force-pushed the feat/litellm-config-passthrough branch 3 times, most recently from ade8500 to 29af484 Compare June 25, 2026 10:48
…lock

Add an optional `litellm:` mapping in .openkb/config.yaml as the single place to
tune LiteLLM. Keys are forwarded to LiteLLM: `timeout` and `extra_headers` apply
per request (the existing per-call mechanism, covering both the compiler and the
agents-SDK paths), and the rest are set as litellm module globals (drop_params,
num_retries, ssl_verify, ...). Resolves the Ollama UnsupportedParamsError in
#137: `litellm: {drop_params: true}` lets LiteLLM drop params a provider rejects
(e.g. parallel_tool_calls on Ollama).

- config.resolve_litellm_settings(): validate the value is a mapping, drop
  non-string keys, pass values through verbatim (the user owns them).
- cli._setup_llm_key(): the `litellm:` block is canonical — when it specifies
  `timeout` / `extra_headers` they route to the per-call stashes and replace the
  legacy top-level keys (an empty `extra_headers: {}` clears, not reverts); the
  rest go to cli._apply_litellm_settings(), which setattrs each onto litellm.
  Guards: skip+warn an unknown key, and refuse to overwrite a litellm function.
  Globals are applied process-wide and not reset.
- Back-compat: the legacy top-level `timeout:` / `extra_headers:` keys still work
  (now undocumented; the `litellm:` block is the documented surface).
- warnings go through the logger (stderr), not click.echo, so a typo can't
  corrupt piped stdout (e.g. `openkb query > out.txt`).
- docs: config.yaml.example consolidated under `litellm:`; README drops the
  now-redundant top-level extra_headers section.

Closes #137
@KylinMountain
KylinMountain force-pushed the feat/litellm-config-passthrough branch from 29af484 to 63c0e1b Compare June 25, 2026 10:53
@KylinMountain
KylinMountain merged commit b204ddf into main Jun 25, 2026
1 check passed
@KylinMountain
KylinMountain deleted the feat/litellm-config-passthrough branch June 25, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant