Skip to content

Optimize cleaner run time #6815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jun 16, 2025
Merged

Optimize cleaner run time #6815

merged 3 commits into from
Jun 16, 2025

Conversation

alexqyle
Copy link
Contributor

What this PR does:

Two optimization being made for cleaner:

  • Use non-block way to push cleaner jobs to active user channel and delete user channel. So that long run time deletion job would not block cleaning job in the next cycle.
  • Introduced concurrent way to do DeletePrefix. So it is faster to delete directory with large number of files. Tested with 1k files in one directory. Sequential deletion took 17s and Concurrent deletion took 1.3s

Which issue(s) this PR fixes:

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
@alexqyle
optimize cleaner run time
b8a8d87 
Signed-off-by: Alex Le <leqiyue@amazon.com>
SungJin1212
SungJin1212 reviewed Jun 13, 2025
View reviewed changes
danielblando
danielblando reviewed Jun 13, 2025
View reviewed changes
@alexqyle
make old bucket.DeletePrefix concurrent
80d9119 
Signed-off-by: Alex Le <leqiyue@amazon.com>
@danielblando
Copy link
Contributor

We can add an ENHANCEMENT in the changelog. Good to know what is being deployed in a new rev
yeya24
yeya24 reviewed Jun 14, 2025
View reviewed changes
yeya24
yeya24 approved these changes Jun 14, 2025
View reviewed changes
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. We can update changelog as @danielblando mentioned
@alexqyle
added metric to record failure pushing to job channel. updated CHANGE…
9705f8b 
…LOG. put more logs with timing values

Signed-off-by: Alex Le <leqiyue@amazon.com>
@pull-request-size pull-request-size bot added size/L and removed size/M labels Jun 16, 2025
@alexqyle
Copy link
Contributor Author

We can add an ENHANCEMENT in the changelog. Good to know what is being deployed in a new rev

Updated CHANGELOG
@danielblando danielblando merged commit ffecf3b into cortexproject:master Jun 16, 2025
17 checks passed
@alexqyle alexqyle deleted the cleaner-optimization branch June 16, 2025 18:48
bogdan-at-adobe pushed a commit to bogdan-at-adobe/cortex that referenced this pull request Jun 30, 2025
@alexqyle @bogdan-at-adobe
Optimize cleaner run time (cortexproject#6815)
5cb75dc 
* optimize cleaner run time

Signed-off-by: Alex Le <leqiyue@amazon.com>

* make old bucket.DeletePrefix concurrent

Signed-off-by: Alex Le <leqiyue@amazon.com>

* added metric to record failure pushing to job channel. updated CHANGELOG. put more logs with timing values

Signed-off-by: Alex Le <leqiyue@amazon.com>

---------

Signed-off-by: Alex Le <leqiyue@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/compactor size/L type/performance
4 participants
@alexqyle @danielblando @SungJin1212 @yeya24