Description
Describe the bug
When the patterns endpoint returns more results then the users configured max_query_series
, we get a maximum of series (20000) reached for a single query
400 response.
Can we add a limit parameter so queries that hit the max_query_series
can still return partial pattern results?
Since the patterns endpoint doesn't take filters on parsed fields, this error is especially frustrating/confusing to users as every other query in Logs Drilldown can be returning valid results.
To Reproduce
Steps to reproduce the behavior:
Find a Loki instance with high cardinality for a label selector:
- Ops link
- Notice the error getting thrown on every page view when navigating around.
Expected behavior
- API can set
limit
parameter and still return something when labels contain more patterns thenmax_query_series
Environment:
- Infrastructure: N/A
- Deployment tool: N/A
Screenshots, Promtail config, or terminal output
Notes:
Something weird I've noticed is that if you hit the patterns API endpoint directly in the browser you get a 200 and the endpoint returns valid results, but when calling the same URL in the web application we're getting a 400, even though they are both GET requests and there don't seem to be any HTTP headers getting set that should cause this behavior.
Additionally, I'm seeing some of those responses that throw this error in the web applications return fewer results then the max_query_series
error that is thrown, e.g. in ops I get the maximum of series (20000) reached for a single query"
, but when I hit the patterns API directly in the browser I only get 300 patterns back. I'm guessing this is because error getting thrown is because the query passed into the patterns endpoint is hitting the max_query_series
, and it's not complaining about the number of patterns being returned.
Related:
grafana/logs-drilldown#1355
CC: @trevorwhitney