-
Notifications
You must be signed in to change notification settings - Fork 8.5k
Labels
:mlFeature: AI InfraRelating to the AI Assistant flow and any work impacting/involving the AI/Inference ConnectorRelating to the AI Assistant flow and any work impacting/involving the AI/Inference ConnectorTeam:AI InfraPlatform AppEx AI Infrastructure Team t//Platform AppEx AI Infrastructure Team t//bugFixes for quality problems that affect the customer experienceFixes for quality problems that affect the customer experiencev9.2.0
Description
Kibana version: Observed in 8.18 onwards
Original install method (e.g. download page, yum, from source, etc.):
Describe the bug:
The NL-to-ESQL tool tries to call unavailable tools sometimes with certain user prompts.
Steps to reproduce:
- Run the evaluation framework for the Observability AI Assistant - specifically noticed in the
Find the top 5 slowest Nginx requests for the 'api-gateway' service in the last two hours.ES|QL scenario - It's mostly observed with Claude and Gemini models
- This might have to be run multiple times to notice the error (Trace):
ERROR ChatCompletionError: Tool "get_field_info" called but was not available
at Object.next (throw_serialized_chat_completion_errors.ts:29:17)
at /Users/viduni/Workspace/Elastic/kibana/node_modules/rxjs/src/internal/operators/tap.ts:189:31
....
at processTicksAndRejections (node:internal/process/task_queues:105:5) {
code: 'internalError',
meta: { name: 'get_field_info' }
}
OR
ERROR ChatCompletionError: Tool "get_dataset_info" called but was not available
at Object.next (throw_serialized_chat_completion_errors.ts:29:17)
at /Users/viduni/Workspace/Elastic/kibana/node_modules/rxjs/src/internal/operators/tap.ts:189:31
....
at processTicksAndRejections (node:internal/process/task_queues:105:5) {
code: 'internalError',
meta: { name: 'get_dataset_info' }
}
The NL-to-ESQL has a separate system prompt. We don't send the o11y/search AI Assistant system prompt to this task.
Expected behavior: The LLM should not call unavailable tools.
Any additional context: Recently, there was an incident related to an error like this where the o11y/search AI Assistant was trying to call tools which are only exposed to the NL-to-ESQL task, outside of this task. This was resolved by collapsing these tool calls for the main conversation. Related PR - #226078
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
:mlFeature: AI InfraRelating to the AI Assistant flow and any work impacting/involving the AI/Inference ConnectorRelating to the AI Assistant flow and any work impacting/involving the AI/Inference ConnectorTeam:AI InfraPlatform AppEx AI Infrastructure Team t//Platform AppEx AI Infrastructure Team t//bugFixes for quality problems that affect the customer experienceFixes for quality problems that affect the customer experiencev9.2.0