Skip to content

feat(storage): Azure backend using thanos.io/objstore - #11315

Merged
ashwanthgoli merged 20 commits into
grafana:mainfrom
JoaoBraveCoding:log-4550-azure
Oct 25, 2024
Merged

feat(storage): Azure backend using thanos.io/objstore#11315
ashwanthgoli merged 20 commits into
grafana:mainfrom
JoaoBraveCoding:log-4550-azure

Conversation

@JoaoBraveCoding

@JoaoBraveCoding JoaoBraveCoding commented Nov 24, 2023

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it*:

This PR adds support to use the thanos.io/objstore backend for the Azure provider for all components excluding the Ruler since it's similar to the work done in #11132.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

  • We tried to the best of our efforts to preserve feature parity so the new backend still uses the ObjectClient interface
  • Fix how some CLI variables we being passed (some HTTP CLI variables were under .http while others weren't)
  • Thanos.io/objstore doesn't support all the CLI flags Loki supports nowadays (see snippet below)
  • This PR was inspired by the way Mimir uses thanos.io/objstore with the goal of making this part of the codebase as similar as possible

CLI Table

azure.account-name
azure.account-key
azure.connection-string
azure.container-name
azure.endpoint-suffix
azure.user-assigned-id
azure.max-retries
azure.max-retry-delay

NOT SUPPORTED AT THE MOMENT
azure.environment
azure.use-managed-identity
azure.use-federated-token
azure.chunk-delimiter
azure.request-timeout
azure.download-buffer-size
azure.upload-buffer-size
azure.download-buffer-size
azure.min-retry-delay
azure.use-service-principal
azure.tenant-id
azure.client-id
azure.client-secret

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
    • If the change is worth mentioning in the release notes, add add-to-release-notes label
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR
@github-actions

Copy link
Copy Markdown
Contributor

Trivy scan found the following vulnerabilities:

@JoaoBraveCoding JoaoBraveCoding changed the title storage: Azure backend using thanos.io/objstore Mar 14, 2024
@github-actions github-actions Bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Mar 15, 2024
@JoaoBraveCoding
JoaoBraveCoding marked this pull request as ready for review March 18, 2024 15:59
@JoaoBraveCoding
JoaoBraveCoding requested a review from a team as a code owner March 18, 2024 15:59

@JoaoBraveCoding JoaoBraveCoding left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, the adapter was a great idea!

@JoaoBraveCoding JoaoBraveCoding left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On pkg/storage/bucket/client.go we should add to func (cfg *StorageBackendConfig) Validate() error {

	if err := cfg.Azure.Validate(); err != nil {
		return errors.Wrap(err, "invalid Azure Storage config")
	}

@JoaoBraveCoding JoaoBraveCoding left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@JoaoBraveCoding

Copy link
Copy Markdown
Collaborator Author

Tested and it works 👍

"github.com/go-kit/log"
"github.com/go-kit/log/level"
"github.com/pkg/errors"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder why the linter did not catch this empty newline. Please run make format

Comment on lines +39 to +43
func WithRetryableErrFunc(f func(err error) bool) func(*ObjectClientAdapter) {
return func(o *ObjectClientAdapter) {
o.isRetryableErr = f
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Personally I find this is quite an unusual API.
You are supposed to call bucket.WithRetryableErrFunc(func(err error) bool {})(client), right?

return o
}

type options func(*ObjectClientAdapter)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
type options func(*ObjectClientAdapter)
type ClientOptions func(*ObjectClientAdapter)
@ashwanthgoli
ashwanthgoli merged commit 5824e3d into grafana:main Oct 25, 2024
@JoaoBraveCoding
JoaoBraveCoding deleted the log-4550-azure branch November 8, 2024 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories

3 participants