Skip to content

Added support for flushing blocks #2794

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Jun 30, 2020
Merged

Added support for flushing blocks #2794

merged 13 commits into from
Jun 30, 2020

Conversation

pstibrany
Copy link
Contributor

@pstibrany pstibrany commented Jun 25, 2020

What this PR does: This PR adds support for flushing blocks.

Flushing can be triggered by:

  • using -experimental.tsdb.flush-blocks-on-shutdown option, honored when ingester shuts down
  • using /flush endpoint (works for chunks too) -- this triggers flush and shipping (if enabled), but endpoint returns immediately. There are log messages written after flushing and shipping (if enabled) has finished
  • using /shutdown endpoint (works for chunks too) -- this shuts down ingester with flush flag enabled. Endpoint waits until flushing and shipping has finished.

This follows the proposal from PR #2717.

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
@pstibrany pstibrany requested a review from pracucci June 25, 2020 15:25
Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing job! Very clean and well tested, thanks! I left few nits, but already good.

pstibrany added 12 commits June 30, 2020 17:29
Flushing can be triggered by:
- using -experimental.tsdb.flush-blocks-on-shutdown option
- using /flush endpoint (works for chunks too) -- this triggers flush and shipping (if enabled), but endpoint returns immediately
- using /shutdown endpoint (works for chunks too) -- enables flush flag, and shuts down ingester. Endpoint waits until flushing and shipping has finished.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Both shipping and compaction now send completion notification back.
Flushing reacts on ingester stopping, and writes proper message.

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
@pstibrany
Copy link
Contributor Author

Thanks for review Marco. I did some changes in compactBlocks method after rebase, so would you please take a look again?

@pstibrany pstibrany requested a review from pracucci June 30, 2020 15:54
Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing my feedback! LGTM (modulo a nit)

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>
@pstibrany pstibrany merged commit 5bc5654 into cortexproject:master Jun 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2 participants