Skip to content

Remove shared_store and shared_store_key_prefix from shipper and compactor #10840

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Oct 30, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixup! Merge branch 'main' into ashwanth/remove-shared-store
  • Loading branch information
ashwanthgoli committed Oct 19, 2023
commit fd8c32217f37389c30897259892abfb242ae6ddc
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
* [10709](https://github.com/grafana/loki/pull/10709) **chaudum**/**salvacorts** Remove `ingester.max-transfer-retries` configuration option in favor of using the WAL.
* [10736](https://github.com/grafana/loki/pull/10736) **ashwanthgoli** Deprecate write dedupe cache as this is not required by the newer single store indexes (tsdb and boltdb-shipper).
* [10693](https://github.com/grafana/loki/pull/10693) **ashwanthgoli** Embedded cache: Updates the metric prefix from `querier_cache_` to `loki_embeddedcache_` and removes duplicate metrics.
* [10840](https://github.com/grafana/loki/pull/10840) **ashwanthgoli** Removes `shared_store` and `shared_store_key_prefix` from tsdb and boltdb shipper configs and their corresponding CLI flags.
* [10840](https://github.com/grafana/loki/pull/10840) **ashwanthgoli** Removes `shared_store` and `shared_store_key_prefix` from tsdb, boltdb shipper and compactor configs and their corresponding CLI flags.
* [10793](https://github.com/grafana/loki/pull/10793) **ashwanthgoli** Config: Better configuration defaults to provide a better experience for users out of the box.
* [10785](https://github.com/grafana/loki/pull/10785) **ashwanthgoli** Config: Removes `querier.worker-parallelism` and updates default value of `querier.max-concurrent` to 4.
* [10733](https://github.com/grafana/loki/pull/10733) **shantanualsi** Add support for case-insensitive logql funtions
Expand Down
2 changes: 1 addition & 1 deletion pkg/loki/modules.go
Original file line number Diff line number Diff line change
Expand Up @@ -1194,7 +1194,7 @@ func (t *Loki) initCompactor() (services.Service, error) {
return nil, fmt.Errorf("failed to create delete request store object client: %w", err)
}
} else {
return nil, fmt.Errorf("-compactor.delete-request-store should be configured when retention is enabled.")
return nil, fmt.Errorf("compactor.delete-request-store should be configured when retention is enabled.")
}
}

Expand Down
13 changes: 7 additions & 6 deletions tools/tsdb/bloom-tester/readlib.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ import (
"github.com/grafana/loki/pkg/logql/log"
"github.com/grafana/loki/pkg/storage/bloom/v1/filter"
"github.com/grafana/loki/pkg/storage/chunk"
"github.com/grafana/loki/pkg/storage/config"
tsdbindex "github.com/grafana/loki/pkg/storage/stores/shipper/indexshipper/tsdb/index"

"math"
Expand Down Expand Up @@ -62,30 +61,32 @@ func executeRead() {

flag.Parse()

objectClient, err := storage.NewObjectClient(conf.StorageConfig.TSDBShipperConfig.SharedStoreType, conf.StorageConfig, clientMetrics)
periodCfg, tableRange, tableName, err := helpers.GetPeriodConfigForTableNumber(bucket, conf.SchemaConfig.Configs)
helpers.ExitErr("find period config for bucket", err)

objectClient, err := storage.NewObjectClient(periodCfg.ObjectType, conf.StorageConfig, clientMetrics)
helpers.ExitErr("creating object client", err)

chunkClient := client.NewClient(objectClient, nil, conf.SchemaConfig)

tableRanges := helpers.GetIndexStoreTableRanges(config.TSDBType, conf.SchemaConfig.Configs)

openFn := func(p string) (shipperindex.Index, error) {
return tsdb.OpenShippableTSDB(p, tsdb.IndexOpts{})
}

indexShipper, err := indexshipper.NewIndexShipper(
periodCfg.IndexTables.PathPrefix,
conf.StorageConfig.TSDBShipperConfig.Config,
objectClient,
overrides,
nil,
openFn,
tableRanges[len(tableRanges)-1],
tableRange,
prometheus.WrapRegistererWithPrefix("loki_tsdb_shipper_", prometheus.DefaultRegisterer),
util_log.Logger,
)
helpers.ExitErr("creating index shipper", err)

tenants, tableName, err := helpers.ResolveTenants(objectClient, bucket, tableRanges)
tenants, err := helpers.ResolveTenants(objectClient, periodCfg.IndexTables.PathPrefix, tableName)
level.Info(util_log.Logger).Log("tenants", strings.Join(tenants, ","), "table", tableName)
helpers.ExitErr("resolving tenants", err)

Expand Down