Skip to content

panic: expected aggregation operator but got approx_topk . with multilple storage configs and schemas #17731

New issue
New issue
Open
Open
panic: expected aggregation operator but got approx_topk . with multilple storage configs and schemas#17731
Labels
type/bugSomehing is not working as expected
@pavlozt

Description

@pavlozt
pavlozt
opened on May 18, 2025

Panic in operator parser approx_topk

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki image grafana/loki:3.4.3 as single binary with docker compose configuration.
  2. Query: approx_topk(100, (sum without (level) ( count_over_time({domain="$domain",job="http"} |= "$line_filter" | logfmt uid | __error__ = "" | uid !="" [$__range]) ) ) )

Some other simpler queries using Aprox_topk seem to work.
approx_topk function works on old data, v12 scheme.

Server config


common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

ingester:
  chunk_target_size: 4194304
  chunk_encoding: snappy
  max_chunk_age: 4h
  chunk_idle_period: 2h


limits_config:
  retention_period: 180d
  retention_stream:
    - selector: '{env="test"}'
      priority: 1
      period: 1d # 
  split_queries_by_interval: 24h
  max_query_parallelism: 100
  max_query_series: 1000000 # really need it without apporx_topk. 
  query_timeout: 5m
  ingestion_rate_mb: 32
  allow_structured_metadata: false 
  shard_aggregations:
    - approx_topk

query_range:
  shard_aggregations: "approx_topk"

query_scheduler:
  max_outstanding_requests_per_tenant: 4096
frontend:
  max_outstanding_per_tenant: 4096
  encoding: "protobuf" 


schema_config:
  configs:
    - from: 2023-01-01
      index:
        period: 24h
        prefix: index_
      object_store: filesystem
      schema: v12
      store: tsdb

    - from: 2025-05-04
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v12  
      store: tsdb
    - from: 2025-05-18
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v13 
      store: tsdb

Panic log

loki-1  | goroutine 9993 [running]:
loki-1  | github.com/grafana/loki/v3/pkg/util/server.onPanic({0x38f1f80, 0xc007d40a98})
loki-1  |       /src/loki/pkg/util/server/recovery.go:57 +0x4d
loki-1  | github.com/grafana/loki/v3/pkg/util/server.init.func2.1.1()
loki-1  |       /src/loki/pkg/util/server/recovery.go:46 +0x2f
loki-1  | panic({0x38f1f80?, 0xc007d40a98?})
loki-1  |       /usr/local/go/src/runtime/panic.go:791 +0x132
loki-1  | github.com/grafana/loki/v3/pkg/logql.(*VectorAggEvaluator).Next(0xc0067f53c0)
loki-1  |       /src/loki/pkg/logql/evaluator.go:570 +0x2173
loki-1  | github.com/grafana/loki/v3/pkg/logql.(*query).evalSample(0xc0099db040, {0x4788848, 0xc0007f8150}, {0x47ae918, 0xc0099dabe0})
loki-1  |       /src/loki/pkg/logql/engine.go:378 +0x397
loki-1  | github.com/grafana/loki/v3/pkg/logql.(*query).Eval(0xc0099db040, {0x4788848, 0xc0007f8150})
loki-1  |       /src/loki/pkg/logql/engine.go:309 +0x632
loki-1  | github.com/grafana/loki/v3/pkg/logql.(*query).Exec(_, {_, _})
loki-1  |       /src/loki/pkg/logql/engine.go:275 +0xcb5
loki-1  | github.com/grafana/loki/v3/pkg/querier.(*QuerierAPI).InstantQueryHandler(_, {_, _}, _)
loki-1  |       /src/loki/pkg/querier/http.go:102 +0x16c
loki-1  | github.com/grafana/loki/v3/pkg/querier.(*Handler).Do(0xc0006aa3e0, {0x47887a0, 0xc0099f9e90}, {0x47b0fb0, 0xc0099baa00})
loki-1  |       /src/loki/pkg/querier/handler.go:40 +0x2a7
loki-1  | github.com/grafana/loki/v3/pkg/loki.(*Loki).initQuerier.CacheGenNumberContextSetterMiddleware.func33.1({0x47887a0, 0xc0099f9e90}, {0x47b0fb0, 0xc0099baa00})
loki-1  |       /src/loki/pkg/querier/queryrange/queryrangebase/middleware.go:45 +0xb9
loki-1  | github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase.HandlerFunc.Do(0x47887a0?, {0x47887a0?, 0xc0099f9e90?}, {0x47b0fb0?, 0xc0099baa00?})
loki-1  |       /src/loki/pkg/querier/queryrange/queryrangebase/roundtrip.go:80 +0x37
loki-1  | github.com/grafana/loki/v3/pkg/querier/queryrange.(*Tracer).Wrap.Tracer.Wrap.func1({0x47887a0, 0xc0099f9e60}, {0x47b0fb0, 0xc0099baa00})
loki-1  |       /src/loki/pkg/querier/queryrange/instrument.go:68 +0xfb
loki-1  | github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase.HandlerFunc.Do(0x3fb6738?, {0x47887a0?, 0xc0099f9e60?}, {0x47b0fb0?, 0xc0099baa00?})
loki-1  |       /src/loki/pkg/querier/queryrange/queryrangebase/roundtrip.go:80 +0x37
loki-1  | github.com/grafana/loki/v3/pkg/querier/queryrange.Instrument.Wrap.func1({0x47887a0, 0xc0099f9e60}, {0x47b0fb0, 0xc0099baa00})
loki-1  |       /src/loki/pkg/querier/queryrange/instrument.go:38 +0x1b3
loki-1  | github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase.HandlerFunc.Do(0xc0099f9e30?, {0x47887a0?, 0xc0099f9e60?}, {0x47b0fb0?, 0xc0099baa00?})
loki-1  |       /src/loki/pkg/querier/queryrange/queryrangebase/roundtrip.go:80 +0x37
loki-1  | github.com/grafana/loki/v3/pkg/util/server.init.func2.1({0x47887a0?, 0xc0099f9e60?}, {0x47b0fb0?, 0xc0099baa00?})
loki-1  |       /src/loki/pkg/util/server/recovery.go:49 +0x8a
loki-1  | github.com/grafana/loki/v3/pkg/querier/queryrange/queryrangebase.HandlerFunc.Do(0xc003b7ed60?, {0x47887a0?, 0xc0099f9e60?}, {0x47b0fb0?, 0xc0099baa00?})
loki-1  |       /src/loki/pkg/querier/queryrange/queryrangebase/roundtrip.go:80 +0x37
loki-1  | github.com/grafana/loki/v3/pkg/querier/worker.handleQueryRequest({0x47887a0?, 0xc0099f9e30?}, 0xc002d4cf70?, {0x7ff312329ea8, 0xc0023864c8}, {0x7ff312329e80?, 0x6c811a0?})
loki-1  |       /src/loki/pkg/querier/worker/util.go:133 +0x151
loki-1  | github.com/grafana/loki/v3/pkg/querier/worker.(*schedulerProcessor).runQueryRequest(0xc001cb6000, {0x47887a0?, 0xc0099f9e30?}, {0x47439c0, 0xc0099daf00}, 0x26916360e1f4bd7a, {0xc0099cd170, 0xf}, 0x0?, 0xc00a11f950)
loki-1  |       /src/loki/pkg/querier/worker/scheduler_processor.go:187 +0x10d
loki-1  | github.com/grafana/loki/v3/pkg/querier/worker.(*schedulerProcessor).querierLoop.func1()
loki-1  |       /src/loki/pkg/querier/worker/scheduler_processor.go:166 +0x326
loki-1  | created by github.com/grafana/loki/v3/pkg/querier/worker.(*schedulerProcessor).querierLoop in goroutine 1759
loki-1  |       /src/loki/pkg/querier/worker/scheduler_processor.go:144 +0x16f
loki-1  |
loki-1  | goroutine 1 [select]:
loki-1  | github.com/grafana/dskit/services.(*Manager).AwaitStopped(0xc001fbde60, {0x47885e0, 0x6c811a0})
loki-1  |       /src/loki/vendor/github.com/grafana/dskit/services/manager.go:153 +0x67
loki-1  | github.com/grafana/loki/v3/pkg/loki.(*Loki).Run(0xc0009c4000, {0x0?, {0x4?, 0x2?, 0x6c597a0?}})
loki-1  |       /src/loki/pkg/loki/loki.go:593 +0xdf0
loki-1  | main.main()
loki-1  |       /src/loki/cmd/loki/main.go:129 +0x1305
loki-1  |
loki-1  | goroutine 18 [select]:
loki-1  | github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
loki-1  |       /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xa5
loki-1  | created by github.com/baidubce/bce-sdk-go/util/log.NewLogger in goroutine 1
loki-1  |       /src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x116
loki-1  |
loki-1  | goroutine 5 [select]:
loki-1  | go.opencensus.io/stats/view.(*worker).start(0xc0006d8000)
loki-1  |       /src/loki/vendor/go.opencensus.io/stats/view/worker.go:292 +0x9f
loki-1  | created by go.opencensus.io/stats/view.init.0 in goroutine 1
loki-1  |       /src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x8d
loki-1  |
loki-1  | goroutine 789 [select]:
loki-1  | github.com/grafana/loki/v3/pkg/distributor.(*Distributor).pushIngesterWorker(0xc000d71208, {0x47887d8, 0xc001b5f2c0})
loki-1  |       /src/loki/pkg/distributor/distributor.go:1027 +0x16b
loki-1  | created by github.com/grafana/loki/v3/pkg/distributor.(*Distributor).running in goroutine 776
loki-1  |       /src/loki/pkg/distributor/distributor.go:385 +0xcb
loki-1  |
loki-1  | goroutine 101 [chan receive]:
loki-1  | github.com/grafana/loki/v3/pkg/util/log.newPrometheusLogger.WithFlushPeriod.func2.1()
loki-1  |       /src/loki/vendor/github.com/grafana/dskit/log/buffered.go:76 +0x94
loki-1  | created by github.com/grafana/loki/v3/pkg/util/log.newPrometheusLogger.WithFlushPeriod.func2 in goroutine 1
loki-1  |       /src/loki/vendor/github.com/grafana/dskit/log/buffered.go:72 +0x65
loki-1  |
loki-1  | goroutine 115 [select]:
loki-1  | github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollControllerWithTicker(0xc00092add0, 0xc000996150)
loki-1  |       /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:153 +0x85
loki-1  | github.com/uber/jaeger-client-go.(*RemotelyControlledSampler).pollController(0xc00092add0)
loki-1  |       /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:148 +0x58
loki-1  | created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler in goroutine 1
loki-1  |       /src/loki/vendor/github.com/uber/jaeger-client-go/sampler_remote.go:87 +0x156
loki-1  |
loki-1  | goroutine 117 [select]:
loki-1  | github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000582c40, 0x0?)
loki-1  |       /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:70 +0xa5
loki-1  | created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn in goroutine 1
loki-1  |       /src/loki/vendor/github.com/uber/jaeger-client-go/utils/reconnecting_udp_conn.go:60 +0x1df
loki-1  |
loki-1  | goroutine 118 [select]:
loki-1  | github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000f35440)
loki-1  |       /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:296 +0xc8
loki-1  | created by github.com/uber/jaeger-client-go.NewRemoteReporter in goroutine 1
loki-1  |       /src/loki/vendor/github.com/uber/jaeger-client-go/reporter.go:237 +0x23f
loki-1  |
loki-1  | goroutine 144 [select]:
loki-1  | google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000835890, {0x47887d8, 0xc000ed5cc0})
loki-1  |       /src/loki/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0x10c
loki-1  | created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
loki-1  |       /src/loki/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a
loki-1  |
loki-1  | goroutine 145 [select]:
loki-1  | google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000835980, {0x47887d8, 0xc000ed5d10})
loki-1  |       /src/loki/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0x10c
loki-1  | created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
loki-1  |       /src/loki/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a
loki-1  |
loki-1  | goroutine 146 [select]:
loki-1  | google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0008359d0, {0x47887d8, 0xc000ed5d60})
loki-1  |       /src/loki/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0x10c
loki-1  | created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
loki-1  |       /src/loki/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a
loki-1  |
loki-1  | goroutine 786 [select]:
loki-1  | google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001fbe900, 0x1)
loki-1  |       /src/loki/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:412 +0x108
loki-1  | google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001f85000)
loki-1  |       /src/loki/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:575 +0x7b
loki-1  | google.golang.org/grpc/interlevel=error ts=2025-05-18T08:35:24.994870506Z caller=retry.go:107 org_id=fake traceID=0363f025926662c3 msg="error processing request" try=4 type=queryrange.LokiInstantRequest query="approx_topk(100, \r\n  (sum without (level) \r\n   ( count_over_time({domain=\"xxx.domain.com\",job=\"http\"}  |= \"\" | logfmt  uid,ip,cf_country | __error__ = \"\" | uid !=\"\" [43200s]) ) \r\n  )\r\n)" query_hash=1827531652 start=2025-05-18T08:35:04.899Z end=2025-05-18T08:35:04.899Z start_delta=20.09586381s end_delta=20.095864431s length=0s retry_in=4.124462433s code=Code(500) err="rpc error: code = Code(500) desc = error while processing request: expected aggregation operator but got \"approx_topk\""

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bugSomehing is not working as expected

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions