You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* storage: Refactor block compaction to allow shard-splitting
* Rename to CompactWithSplitting for consistency
* Introduce back compaction series testing.
* Add sharding compaction level
* Add some tests for CompactWithSplitting
* Fix symbols split-compaction (#2371)
* Fixes a race that was actually surfacing another real issue.
* Add a tests for meta min/max time
* Fixes meta min/max after split
* Add mimir compactor code
* Fixing more code
* Introduce a meta fetcher that synchronize filtered metadata of all block locally
* Fixes all tests
* make go/mod
* Adds a tenant scanner
* Working on the blocks cleaner
* Adds bucketindex package
* Fixing and adapting more code
* Add missing minCompactor config
* Fixes as much as possible
* block: Adds Download and MarkForNoCompact
* improve documentation
* wip
* Adds a way to validate local blocks
* fixes wrong merge
* more integration fixes
* fixes metrics namespace
* more fixes
* fixes block_cleaner_test
* remove upload block api
* Fixes split_merge_compactor tests
* Fixes split_merge_compactor tests
* Fixes more tests
* fixing more tests
* Fixes more tests
* Add limits for the compactor
* Removes unused limits and code
* Switches to Compactor interface for mocking
* Fixes blocks_cleaner_test.go
* Removes tombstones removal
* Fixes the compactor shard label not being saved
* Fixes most compactor tests.
* Implement e2e tests
* lint
* Fixes a race detected when generating ULID
* Add the compactor compenent
* Add the compactor helm component
* Fixes tests after the range change
* change: store-gateway uses the bucketindex
Changes the store-gateway to use the bucketindex instead of iterating
through all blocks.
This also tweak block query config.
* Fixes helm to not use a disk
* Use stable time format for logs
* Fixes time format issue
* Changes the default compactor dir
* Fixes panic while rewriting symbols.
* Fixes mark global path and add 8h compaction range
* Fixes store-gateway local cache.
* Fixes store-gateway stale meta
* Add compaction metrics
* Fixes panic in compaction metrics
* Discard __hostname__ labels from block for compaction planning
* Improves parquet profile file reader
* Fixes more broken tests
* Makes configurable the split by compaction
* lint
* Correct compaction range for the current default
* review feedback first round
* make helm/check
* ULID are now based on minTime
* Use model.Time for parent block meta
* Fixes broken tests from previous commit
* Validates that the first compaction range is divisible by the block duration
* Add filesystem tests for delete block
* Fixes tests
* Convert profiles rows when compacting to ensure backward compatibility
---------
Co-authored-by: Anton Kolesnikov <anton.e.kolesnikov@gmail.com>
Blocks with minimum time within this duration are ignored, and not loaded by store-gateway. Useful when used together with -querier.query-store-after to prevent loading young blocks, because there are usually many of them (depending on number of ingesters) and they are not yet compacted. Negative values or 0 disable the filter. (default 2h0m0s)
Duration after which the blocks marked for deletion will be filtered out while fetching blocks. The idea of ignore-deletion-marks-delay is to ignore blocks that are marked for deletion with some delay. This ensures store can still serve blocks that are meant to be deleted but do not have a replacement yet. (default 1h0m0s)
10
+
-blocks-storage.bucket-store.meta-sync-concurrency int
11
+
Number of Go routines to use when syncing block meta files from object storage per tenant. (default 20)
8
12
-blocks-storage.bucket-store.sync-dir string
9
13
Directory to store synchronized pyroscope block headers. This directory is not required to be persisted between restarts, but it's highly recommended in order to improve the store-gateway startup time. (default "./data/pyroscope-sync/")
How frequently to scan the bucket, or to refresh the bucket index (if enabled), in order to look for changes (new blocks shipped by ingesters and blocks deleted by retention or compaction). (default 15m0s)
12
16
-blocks-storage.bucket-store.tenant-sync-concurrency int
13
17
Maximum number of concurrent tenants synching blocks. (default 10)
18
+
-compactor.block-ranges value
19
+
List of compaction time ranges. (default 3h0m0s,6h0m0s,12h0m0s)
20
+
-compactor.block-sync-concurrency int
21
+
Number of Go routines to use when downloading blocks for compaction and uploading resulting blocks. (default 8)
22
+
-compactor.blocks-retention-period duration
23
+
Delete blocks containing samples older than the specified retention period. 0 to disable.
24
+
-compactor.cleanup-concurrency int
25
+
Max number of tenants for which blocks cleanup and maintenance should run concurrently. (default 20)
26
+
-compactor.cleanup-interval duration
27
+
How frequently compactor should run blocks cleanup and maintenance, as well as update the bucket index. (default 15m0s)
28
+
-compactor.compaction-concurrency int
29
+
Max number of concurrent compactions running. (default 1)
30
+
-compactor.compaction-interval duration
31
+
The frequency at which the compaction runs (default 1h0m0s)
32
+
-compactor.compaction-jobs-order string
33
+
The sorting to use when deciding which compaction jobs should run first for a given tenant. Supported values are: smallest-range-oldest-blocks-first, newest-blocks-first. (default "smallest-range-oldest-blocks-first")
34
+
-compactor.compaction-retries int
35
+
How many times to retry a failed compaction within a single compaction run. (default 3)
36
+
-compactor.compaction-split-by string
37
+
Experimental: The strategy to use when splitting blocks during compaction. Supported values are: fingerprint, stacktracePartition. (default "fingerprint")
38
+
-compactor.compactor-tenant-shard-size int
39
+
Max number of compactors that can compact blocks for single tenant. 0 to disable the limit and use all compactors.
40
+
-compactor.data-dir string
41
+
Directory to temporarily store blocks during compaction. This directory is not required to be persisted between restarts. (default "./data-compactor")
42
+
-compactor.deletion-delay duration
43
+
Time before a block marked for deletion is deleted from bucket. If not 0, blocks will be marked for deletion and compactor component will permanently delete blocks marked for deletion from the bucket. If 0, blocks will be deleted straight away. Note that deleting blocks immediately can cause query failures. (default 12h0m0s)
Comma separated list of tenants that cannot be compacted by this compactor. If specified, and compactor would normally pick given tenant for compaction (via -compactor.enabled-tenants or sharding), it will be ignored instead.
Comma separated list of tenants that can be compacted. If specified, only these tenants will be compacted by compactor, otherwise all tenants can be compacted. Subject to sharding.
How long the compactor waits before compacting first-level blocks that are uploaded by the ingesters. This configuration option allows for the reduction of cases where the compactor begins to compact blocks before all ingesters have uploaded their blocks to the storage. (default 25m0s)
50
+
-compactor.max-compaction-time duration
51
+
Max time for starting compactions for a single tenant. After this time no new compactions for the tenant are started before next compaction cycle. This can help in multi-tenant environments to avoid single tenant using all compaction time, but also in single-tenant environments to force new discovery of blocks more often. 0 = disabled. (default 1h0m0s)
52
+
-compactor.max-opening-blocks-concurrency int
53
+
Number of goroutines opening blocks before compaction. (default 1)
54
+
-compactor.meta-sync-concurrency int
55
+
Number of Go routines to use when syncing block meta files from the long term storage. (default 20)
56
+
-compactor.no-blocks-file-cleanup-enabled
57
+
[experimental] If enabled, will delete the bucket-index, markers and debug files in the tenant bucket when there are no blocks left in the index.
58
+
-compactor.partial-block-deletion-delay duration
59
+
If a partial block (unfinished block without meta.json file) hasn't been modified for this time, it will be marked for deletion. The minimum accepted value is 4h0m0s: a lower value will be ignored and the feature disabled. 0 to disable. (default 1d)
60
+
-compactor.ring.consul.acl-token string
61
+
ACL Token used to interact with Consul.
62
+
-compactor.ring.consul.cas-retry-delay duration
63
+
Maximum duration to wait before retrying a Compare And Swap (CAS) operation. (default 1s)
64
+
-compactor.ring.consul.client-timeout duration
65
+
HTTP timeout when talking to Consul (default 20s)
66
+
-compactor.ring.consul.consistent-reads
67
+
Enable consistent reads to Consul.
68
+
-compactor.ring.consul.hostname string
69
+
Hostname and port of Consul. (default "localhost:8500")
70
+
-compactor.ring.consul.watch-burst-size int
71
+
Burst size used in rate limit. Values less than 1 are treated as 1. (default 1)
72
+
-compactor.ring.consul.watch-rate-limit float
73
+
Rate limit when watching key or prefix in Consul, in requests per second. 0 disables the rate limit. (default 1)
74
+
-compactor.ring.etcd.dial-timeout duration
75
+
The dial timeout for the etcd connection. (default 10s)
76
+
-compactor.ring.etcd.endpoints string
77
+
The etcd endpoints to connect to.
78
+
-compactor.ring.etcd.max-retries int
79
+
The maximum number of retries to do for failed ops. (default 10)
80
+
-compactor.ring.etcd.password string
81
+
Etcd password.
82
+
-compactor.ring.etcd.tls-ca-path string
83
+
Path to the CA certificates to validate server certificate against. If not set, the host's root CA certificates are used.
84
+
-compactor.ring.etcd.tls-cert-path string
85
+
Path to the client certificate, which will be used for authenticating with the server. Also requires the key path to be configured.
86
+
-compactor.ring.etcd.tls-cipher-suites string
87
+
Override the default cipher suite list (separated by commas).
88
+
-compactor.ring.etcd.tls-enabled
89
+
Enable TLS.
90
+
-compactor.ring.etcd.tls-insecure-skip-verify
91
+
Skip validating server certificate.
92
+
-compactor.ring.etcd.tls-key-path string
93
+
Path to the key for the client certificate. Also requires the client certificate to be configured.
Maximum time to wait for ring stability at startup. If the compactor ring keeps changing after this period of time, the compactor will start anyway. (default 5m0s)
Minimum time to wait for ring stability at startup. 0 to disable.
132
+
-compactor.split-and-merge-shards int
133
+
The number of shards to use when splitting blocks. 0 to disable splitting.
134
+
-compactor.split-groups int
135
+
Number of groups that blocks for splitting should be grouped into. Each group of blocks is then split separately. Number of output split shards is controlled by -compactor.split-and-merge-shards. (default 1)
14
136
-config.expand-env
15
137
Expands ${var} in config according to the values of the environment variables.
Copy file name to clipboardExpand all lines: cmd/pyroscope/help.txt.tmpl
+26Lines changed: 26 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -5,6 +5,32 @@ Usage of ./pyroscope:
5
5
When set to true, incoming HTTP requests must specify tenant ID in HTTP X-Scope-OrgId header. When set to false, tenant ID anonymous is used instead.
6
6
-blocks-storage.bucket-store.sync-dir string
7
7
Directory to store synchronized pyroscope block headers. This directory is not required to be persisted between restarts, but it's highly recommended in order to improve the store-gateway startup time. (default "./data/pyroscope-sync/")
8
+
-compactor.blocks-retention-period duration
9
+
Delete blocks containing samples older than the specified retention period. 0 to disable.
10
+
-compactor.compactor-tenant-shard-size int
11
+
Max number of compactors that can compact blocks for single tenant. 0 to disable the limit and use all compactors.
12
+
-compactor.data-dir string
13
+
Directory to temporarily store blocks during compaction. This directory is not required to be persisted between restarts. (default "./data-compactor")
How long the compactor waits before compacting first-level blocks that are uploaded by the ingesters. This configuration option allows for the reduction of cases where the compactor begins to compact blocks before all ingesters have uploaded their blocks to the storage. (default 25m0s)
16
+
-compactor.partial-block-deletion-delay duration
17
+
If a partial block (unfinished block without meta.json file) hasn't been modified for this time, it will be marked for deletion. The minimum accepted value is 4h0m0s: a lower value will be ignored and the feature disabled. 0 to disable. (default 1d)
18
+
-compactor.ring.consul.hostname string
19
+
Hostname and port of Consul. (default "localhost:8500")
20
+
-compactor.ring.etcd.endpoints string
21
+
The etcd endpoints to connect to.
22
+
-compactor.ring.etcd.password string
23
+
Etcd password.
24
+
-compactor.ring.etcd.username string
25
+
Etcd username.
26
+
-compactor.ring.instance-interface-names string
27
+
List of network interface names to look up when finding the instance IP address. (default [<private network interfaces>])
28
+
-compactor.ring.store string
29
+
Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist")
30
+
-compactor.split-and-merge-shards int
31
+
The number of shards to use when splitting blocks. 0 to disable splitting.
32
+
-compactor.split-groups int
33
+
Number of groups that blocks for splitting should be grouped into. Each group of blocks is then split separately. Number of output split shards is controlled by -compactor.split-and-merge-shards. (default 1)
8
34
-config.expand-env
9
35
Expands ${var} in config according to the values of the environment variables.
0 commit comments