Skip to content

Patterns: /loki/api/v1/patterns limit #18213

Open
@gtk-grafana

Description

@gtk-grafana

Describe the bug
When the patterns endpoint returns more results then the users configured max_query_series, we get a maximum of series (20000) reached for a single query 400 response.

Can we add a limit parameter so queries that hit the max_query_series can still return partial pattern results?

Since the patterns endpoint doesn't take filters on parsed fields, this error is especially frustrating/confusing to users as every other query in Logs Drilldown can be returning valid results.

To Reproduce
Steps to reproduce the behavior:
Find a Loki instance with high cardinality for a label selector:

Expected behavior

  • API can set limit parameter and still return something when labels contain more patterns then max_query_series

Environment:

  • Infrastructure: N/A
  • Deployment tool: N/A

Screenshots, Promtail config, or terminal output
Image

Notes:
Something weird I've noticed is that if you hit the patterns API endpoint directly in the browser you get a 200 and the endpoint returns valid results, but when calling the same URL in the web application we're getting a 400, even though they are both GET requests and there don't seem to be any HTTP headers getting set that should cause this behavior.

Additionally, I'm seeing some of those responses that throw this error in the web applications return fewer results then the max_query_series error that is thrown, e.g. in ops I get the maximum of series (20000) reached for a single query", but when I hit the patterns API directly in the browser I only get 300 patterns back. I'm guessing this is because error getting thrown is because the query passed into the patterns endpoint is hitting the max_query_series, and it's not complaining about the number of patterns being returned.

Related:
grafana/logs-drilldown#1355

CC: @trevorwhitney

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bugSomehing is not working as expected

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions