Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
minor corrections
  • Loading branch information
ashwanthgoli committed Oct 30, 2023
commit 6f2f476163fe4056b7c5687dd4cec70ca4de6d12
2 changes: 1 addition & 1 deletion docs/sources/operations/storage/logs-deletion.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Log entry deletion relies on configuration of the custom logs retention workflow
## Configuration

Enable log entry deletion by setting `retention_enabled` to true in the compactor's configuration and setting and `deletion_mode` to `filter-only` or `filter-and-delete` in the runtime config.
`delete_request_store` also needs to be configured when retention is enabled to process delete requests, this decides the storage bucket that stores the delete requests.
`delete_request_store` also needs to be configured when retention is enabled to process delete requests, this determines the storage bucket that stores the delete requests.

> **Warning:** Be very careful when enabling retention. It is strongly recommended that you also enable versioning on your objects in object storage to allow you to recover from accidental misconfiguration of a retention setting. If you want to enable deletion but not not want to enforce retention, configure the `retention_period` setting with a value of `0s`.

Expand Down
4 changes: 2 additions & 2 deletions docs/sources/operations/storage/retention.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ compactor:
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
delete_requests_store: gcs
delete_request_store: gcs
schema_config:
configs:
- from: "2020-07-31"
Expand Down Expand Up @@ -256,6 +256,6 @@ limits_config:

compactor:
working_directory: /data/retention
delete_requests_store: gcs
delete_request_store: gcs
retention_enabled: true
```
4 changes: 2 additions & 2 deletions docs/sources/setup/upgrade/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,11 @@ different from `object_store` in `period_config`.

- If you have not configured `-boltdb.shipper.shared-store`,`-tsdb.shipper.shared-store` or their corresponding YAML setting before, no changes are required as part of the upgrade.
- If you have configured `-boltdb.shipper.shared-store` or its YAML setting:
- If it matches the value of `object_store` for all the periods that use `boltdb-shipper` as index type, no changes are required.
- If it matches the value of `object_store` for all the periods that use `boltdb-shipper` as index type, no additional changes are required besides removing the usage of the deleted configuration option.
- If there is a mismatch, you lose access to the index for periods where `-boltdb.shipper.shared-store` does not match `object_store`.
- To make these indexes queryable, index tables need to moved or copied to the store configured in `object_store`.
- If you have configured `-tsdb.shipper.shared-store` or its YAML setting:
- If it matches the value of `object_store` for all the periods that use `tsdb` as index type, no changes are required.
- If it matches the value of `object_store` for all the periods that use `tsdb` as index type, no additional changes are required besides removing the usage of the deleted configuration option.
- If there is a mismatch, you lose access to the index for periods where `-tsdb.shipper.shared-store` does not match `object_store`.
- To make these indexes queryable, index tables need to moved or copied to the store configured in `object_store`.

Expand Down
17 changes: 14 additions & 3 deletions pkg/compactor/compactor.go
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,19 @@ func (cfg *Config) Validate() error {
if cfg.MaxCompactionParallelism < 1 {
return errors.New("max compaction parallelism must be >= 1")
}
if cfg.RetentionEnabled && cfg.ApplyRetentionInterval != 0 && cfg.ApplyRetentionInterval%cfg.CompactionInterval != 0 {
return errors.New("interval for applying retention should either be set to a 0 or a multiple of compaction interval")

if cfg.RetentionEnabled {
if cfg.DeleteRequestStore == "" {
return fmt.Errorf("compactor.delete-request-store should be configured when retention is enabled")
}

if cfg.ApplyRetentionInterval != 0 && cfg.ApplyRetentionInterval%cfg.CompactionInterval != 0 {
return fmt.Errorf("interval for applying retention should either be set to a 0 or a multiple of compaction interval")
}

if err := config.ValidatePathPrefix(cfg.DeleteRequestStoreKeyPrefix); err != nil {
return fmt.Errorf("validate delete store path prefix: %w", err)
}
}

if cfg.DeletionMode != "" {
Expand Down Expand Up @@ -579,7 +590,7 @@ func (c *Compactor) CompactTable(ctx context.Context, tableName string, applyRet

sc, ok := c.storeContainers[schemaCfg.From]
if !ok {
return fmt.Errorf("index store client not found for period starting at %s", &schemaCfg.From)
return fmt.Errorf("index store client not found for period starting at %s", schemaCfg.From.String())
}

table, err := newTable(ctx, filepath.Join(c.cfg.WorkingDirectory, tableName), sc.indexStorageClient, indexCompactor,
Expand Down
2 changes: 0 additions & 2 deletions production/ksonnet/loki/shipper.libsonnet
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@
using_tsdb_shipper: false,
using_shipper_store: $._config.using_boltdb_shipper || $._config.using_tsdb_shipper,

// run ingesters and queriers as statefulsets when using boltdb-shipper to avoid using node disk for storing the index.
stateful_ingesters: if self.using_shipper_store then true else super.stateful_ingesters,
stateful_queriers: if self.using_shipper_store && !self.use_index_gateway then true else super.stateful_queriers,

compactor_pvc_size: '10Gi',
Expand Down
2 changes: 1 addition & 1 deletion production/nomad/loki-distributed/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ storage_config:

compactor:
working_directory: {{ env "NOMAD_ALLOC_DIR" }}/compactor
delete_requests_store: s3
delete_request_store: s3
compaction_interval: 24h
retention_enabled: true

Expand Down
2 changes: 1 addition & 1 deletion production/nomad/loki-simple/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ limits_config:

compactor:
working_directory: {{ env "NOMAD_ALLOC_DIR" }}/compactor
delete_requests_store: s3
delete_request_store: s3
compaction_interval: 5m
retention_enabled: true

Expand Down
2 changes: 1 addition & 1 deletion production/nomad/loki/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ limits_config:
compactor:
working_directory: {{ env "NOMAD_ALLOC_DIR" }}/compactor
compaction_interval: 5m
delete_requests_store: s3
delete_request_store: s3
retention_enabled: true

ruler:
Expand Down