Skip to content

feat(blooms): disk-backed queue for the bloom-planner #14874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 13, 2024
Merged

feat(blooms): disk-backed queue for the bloom-planner #14874

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
015e123
feat(blooms): disk-backed queue for the bloom-planner
salvacorts Nov 12, 2024
d733889
CR
salvacorts Nov 12, 2024
51d4c57
Update pkg/bloombuild/planner/queue/queue.go
salvacorts Nov 13, 2024
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
CR
  • Loading branch information
@salvacorts
salvacorts committed Nov 12, 2024
commit d73388914add64b9f7e459bd1c6719f6089d3bef
2 changes: 1 addition & 1 deletion docs/sources/shared/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1229,7 +1229,7 @@ planner:
# CLI flag: -bloom-build.planner.queue.store-tasks-on-disk
[store_tasks_on_disk: <boolean> | default = false]

# Directory to store tasks on disk. All files will be removed on startup.
# Directory to store tasks on disk.
# CLI flag: -bloom-build.planner.queue.tasks-disk-directory
[tasks_disk_directory: <string> | default = "/tmp/bloom-planner-queue"]

Expand Down
2 changes: 1 addition & 1 deletion pkg/bloombuild/planner/queue/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ type Config struct {
func (cfg *Config) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet) {
f.IntVar(&cfg.MaxQueuedTasksPerTenant, prefix+".max-tasks-per-tenant", 30000, "Maximum number of tasks to queue per tenant.")
f.BoolVar(&cfg.StoreTasksOnDisk, prefix+".store-tasks-on-disk", false, "Whether to store tasks on disk.")
f.StringVar(&cfg.TasksDiskDirectory, prefix+".tasks-disk-directory", "/tmp/bloom-planner-queue", "Directory to store tasks on disk. All files will be removed on startup.")
f.StringVar(&cfg.TasksDiskDirectory, prefix+".tasks-disk-directory", "/tmp/bloom-planner-queue", "Directory to store tasks on disk.")
f.BoolVar(&cfg.CleanTasksDirectory, prefix+".clean-tasks-directory", false, "Whether to clean the tasks directory on startup.")
}

Expand Down
2 changes: 1 addition & 1 deletion pkg/bloombuild/planner/queue/queue.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ func (q *Queue) TotalPending() (total int) {

func getTaskPath(task *protos.ProtoTask) string {
table := protos.FromProtoDayTableToDayTable(task.Table)
taskFile := task.Id + ".task"
taskFile := task.Id + ".protobuf"
return filepath.Join("tasks", task.Tenant, table.String(), taskFile)
}

Expand Down
3 changes: 2 additions & 1 deletion pkg/bloombuild/planner/queue/queue_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,8 @@ func TestQueue(t *testing.T) {
useDisk bool
}{
{
name: "in-memory",
name: "in-memory",
useDisk: false,
},
{
name: "on-disk",
Expand Down
Loading