Skip to content

NL-to-ESQL task calling unavailable tools #231830

@viduni94

Description

@viduni94

Kibana version: Observed in 8.18 onwards

Original install method (e.g. download page, yum, from source, etc.):

Describe the bug:
The NL-to-ESQL tool tries to call unavailable tools sometimes with certain user prompts.

Steps to reproduce:

  1. Run the evaluation framework for the Observability AI Assistant - specifically noticed in the Find the top 5 slowest Nginx requests for the 'api-gateway' service in the last two hours. ES|QL scenario
  2. It's mostly observed with Claude and Gemini models
  3. This might have to be run multiple times to notice the error (Trace):
ERROR ChatCompletionError: Tool "get_field_info" called but was not available
          at Object.next (throw_serialized_chat_completion_errors.ts:29:17)
          at /Users/viduni/Workspace/Elastic/kibana/node_modules/rxjs/src/internal/operators/tap.ts:189:31
          ....
          at processTicksAndRejections (node:internal/process/task_queues:105:5) {
        code: 'internalError',
        meta: { name: 'get_field_info' }
      }

OR

ERROR ChatCompletionError: Tool "get_dataset_info" called but was not available
          at Object.next (throw_serialized_chat_completion_errors.ts:29:17)
          at /Users/viduni/Workspace/Elastic/kibana/node_modules/rxjs/src/internal/operators/tap.ts:189:31
          ....
          at processTicksAndRejections (node:internal/process/task_queues:105:5) {
        code: 'internalError',
        meta: { name: 'get_dataset_info' }
      }

The NL-to-ESQL has a separate system prompt. We don't send the o11y/search AI Assistant system prompt to this task.

Expected behavior: The LLM should not call unavailable tools.

Any additional context: Recently, there was an incident related to an error like this where the o11y/search AI Assistant was trying to call tools which are only exposed to the NL-to-ESQL task, outside of this task. This was resolved by collapsing these tool calls for the main conversation. Related PR - #226078

Metadata

Metadata

Assignees

Labels

:mlFeature: AI InfraRelating to the AI Assistant flow and any work impacting/involving the AI/Inference ConnectorTeam:AI InfraPlatform AppEx AI Infrastructure Team t//bugFixes for quality problems that affect the customer experiencev9.2.0

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions