Skip to content

Conversation

@ruslan-mikhailov
Copy link
Contributor

@ruslan-mikhailov ruslan-mikhailov commented Jul 17, 2025

What this PR does:
It passes actual size for all writes to backend.

From minio-go library:

// PutObject creates an object in a bucket.
// <...>
//   - For size input as -1 PutObject does a multipart Put operation
//     until input stream reaches EOF. Maximum object size that can
//     be uploaded through this operation will be 5TiB.
//
//     WARNING: Passing down '-1' will use memory and these cannot
//     be reused for best outcomes for PutObject(), pass the size always.
//

As we know object's size before Write, there are no obstacles to passing the parameter.

Bonus. If I'm not mistaken, write with negative size is the only case when minio-go library passes default (CRC32C) checksum which should solve the problem #5392 for object storages that don't support CRC32C.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
Copy link
Contributor

@javiermolinar javiermolinar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a comment in the minio PutObject about multipart uploads:

For size smaller than 16MiB PutObject automatically does a single atomic PUT operation.
For size larger than 16MiB PutObject automatically does a multipart upload operation.
For size input as -1 PutObject does a multipart Put operation until input stream reaches EOF. Maximum object size that can be uploaded through this operation will be 5TiB.
WARNING: Passing down ‘-1’ will use memory and these cannot be reused for best outcomes for PutObject(), pass the size always.

https://pkg.go.dev/github.com/minio/minio-go/v7@v7.0.94#Client.PutObject

Maybe the checksum problem is related to multipart uploads?

// corrupted. This means that we don't need to extend the interface of the
// backend in order to delete a corrupted seed file.
err = objectClient.Write(context.Background(), backend.ClusterSeedFileName, []string{}, bytes.NewReader([]byte("{")), -1, nil)
err = objectClient.Write(context.Background(), backend.ClusterSeedFileName, []string{}, bytes.NewReader([]byte("{")), 1, nil)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the difference here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The line imitates write with partial data. I thought if I changed the file write function, it would be nice to be close to it. I don't think it's a big difference, it is just a test anyway, both functions will eventually write the same data

@ruslan-mikhailov
Copy link
Contributor Author

Maybe the checksum problem is related to multipart uploads?

It's a bit complicated. There is decision tree what function to use for Put. In some function, it sets checksum only if TrailingHeaders is enabled:

	if c.trailingHeaderSupport {
		opts.AutoChecksum.SetDefault(ChecksumCRC32C)
		addAutoChecksumHeaders(&opts)
	}

trailingHeaderSupport is opts.TrailingHeaders && clnt.overrideSignerType.IsV4(), TrailingHeaders is disabled by default. That makes it skip checksum.

In some cases (for example, in case size is -1), it does not have such checks.

The issue was addressed here: https://github.com/minio/minio-go/pull/2130/files#diff-d63a6c6b6013f9e4432b05d4f81f4a7b9037fc91453bfb9fc658d6cfa412e926R448

Copy link
Contributor

@mattdurham mattdurham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks straightforward and more accurate!

@ruslan-mikhailov ruslan-mikhailov merged commit 936c35e into grafana:main Jul 18, 2025
22 checks passed
knylander-grafana pushed a commit to knylander-grafana/tempo-doc-work that referenced this pull request Jul 18, 2025
ruslan-mikhailov added a commit to ruslan-mikhailov/tempo that referenced this pull request Aug 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

4 participants