-
Notifications
You must be signed in to change notification settings - Fork 628
[Enhancement] put actual size for writing to backend #5413
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement] put actual size for writing to backend #5413
Conversation
7bc473b to
2d882fa
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a comment in the minio PutObject about multipart uploads:
For size smaller than 16MiB PutObject automatically does a single atomic PUT operation.
For size larger than 16MiB PutObject automatically does a multipart upload operation.
For size input as -1 PutObject does a multipart Put operation until input stream reaches EOF. Maximum object size that can be uploaded through this operation will be 5TiB.
WARNING: Passing down ‘-1’ will use memory and these cannot be reused for best outcomes for PutObject(), pass the size always.
https://pkg.go.dev/github.com/minio/minio-go/v7@v7.0.94#Client.PutObject
Maybe the checksum problem is related to multipart uploads?
| // corrupted. This means that we don't need to extend the interface of the | ||
| // backend in order to delete a corrupted seed file. | ||
| err = objectClient.Write(context.Background(), backend.ClusterSeedFileName, []string{}, bytes.NewReader([]byte("{")), -1, nil) | ||
| err = objectClient.Write(context.Background(), backend.ClusterSeedFileName, []string{}, bytes.NewReader([]byte("{")), 1, nil) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the difference here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The line imitates write with partial data. I thought if I changed the file write function, it would be nice to be close to it. I don't think it's a big difference, it is just a test anyway, both functions will eventually write the same data
It's a bit complicated. There is decision tree what function to use for Put. In some function, it sets checksum only if TrailingHeaders is enabled: trailingHeaderSupport is In some cases (for example, in case size is -1), it does not have such checks. The issue was addressed here: https://github.com/minio/minio-go/pull/2130/files#diff-d63a6c6b6013f9e4432b05d4f81f4a7b9037fc91453bfb9fc658d6cfa412e926R448 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks straightforward and more accurate!
What this PR does:
It passes actual size for all writes to backend.
From minio-go library:
As we know object's size before Write, there are no obstacles to passing the parameter.
Bonus. If I'm not mistaken, write with negative size is the only case when minio-go library passes default (CRC32C) checksum which should solve the problem #5392 for object storages that don't support CRC32C.
Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]