Add retries to copying files in GCS - #2111
Conversation
|
Comparing this to #2108: This is the safer option and support adding this to the codebase. I'm also good on #2108 but it would require more detailed consideration than this change. i.e. how does #2108 interact with hedging? or impact query speeds? Ideally a full "fix" would take into account the different needs of the compactor and the querier. The compactor works best with less aggressive and more consistent settings. On the other hand, we'd prefer the querier to very aggressively hit the backend and fail fast if there was an issue so the job could be retried. Also, ideally we should fix the issue where the compaction is considered successful before the meta rename. |
|
I agree we should take this one over #2108, and that we would want to probably make hedging mutually exclusive with retries, and so some other work is needed there. |
* Add retries to copying files in GCS * Retry idempotent operations in GCS backend * Add retries only to copy * Fix changelog
3 participants
What this PR does:
Adds retries to idempotent write operations (non-hedged). Also better propagates errors failing to mark blocks as compacted.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]