Blob

Blobs are a common abstraction for storing unstructured data on Cloud storage services and accessing them via HTTP. This guide shows how to work with blobs in the Go CDK.

The blob package supports operations like reading and writing blobs (using standard io package interfaces), deleting blobs, and listing blobs in a bucket.

Subpackages contain driver implementations of blob for various services, including Cloud and on-prem solutions. You can develop your application locally using fileblob, then deploy it to multiple Cloud providers with minimal initialization reconfiguration.

Opening a Bucket🔗

The first step in interacting with unstructured storage is to instantiate a portable *blob.Bucket for your storage service.

The easiest way to do so is to use blob.OpenBucket and a service-specific URL pointing to the bucket, making sure you “blank import” the driver package to link it in.

import (
	"gocloud.dev/blob"
	_ "gocloud.dev/blob/<driver>"
)
...
bucket, err := blob.OpenBucket(context.Background(), "<driver-url>")
if err != nil {
    return fmt.Errorf("could not open bucket: %v", err)
}
defer bucket.Close()
// bucket is a *blob.Bucket; see usage below
...

See Concepts: URLs for general background and the guide below for URL usage for each supported service.

Alternatively, if you need fine-grained control over the connection settings, you can call the constructor function in the driver package directly.

import "gocloud.dev/blob/<driver>"
...
bucket, err := <driver>.OpenBucket(...)
...

You may find the wire package useful for managing your initialization code when switching between different backing services.

See the guide below for constructor usage for each supported service.

Prefixed Buckets🔗

You can wrap a *blob.Bucket to always operate on a subfolder of the bucket using blob.PrefixedBucket:

import "gocloud.dev/blob"

// Wrap the bucket using blob.PrefixedBucket.
// The prefix should end with "/", so that the resulting bucket operates
// in a subfolder.
bucket = blob.PrefixedBucket(bucket, "a/subfolder/")

// The original bucket is no longer usable; it has been closed.
// The wrapped bucket should be closed when done.
defer bucket.Close()

// Bucket operations on <key> will be translated to "a/subfolder/<key>".

Alternatively, you can configure the prefix directly in the blob.OpenBucket URL:

import (
	"context"

	"gocloud.dev/blob"
)

// Connect to a bucket using a URL, using the "prefix" query parameter to
// target a subfolder in the bucket.
// The prefix should end with "/", so that the resulting bucket operates
// in a subfolder.
b, err := blob.OpenBucket(ctx, "mem://?prefix=a/subfolder/")
if err != nil {
	return err
}
defer b.Close()

// Bucket operations on <key> will be translated to "a/subfolder/<key>".

Single Key Buckets🔗

You can wrap a *blob.Bucket to always operate on a single key using blob.SingleKeyBucket:

import "gocloud.dev/blob"

// Wrap the bucket using blob.SingleKeyBucket.
// The bucket always references the provided key.
bucket = blob.SingleKeyBucket(bucket, "foo.txt")

// The original bucket is no longer usable; it has been closed.
// The wrapped bucket should be closed when done.
defer bucket.Close()

// Bucket operations will ignore the passed-in key and always reference foo.txt.

Alternatively, you can configure the single key directly in the blob.OpenBucket URL:

import (
	"context"

	"gocloud.dev/blob"
)

// Connect to a bucket using a URL, using the "key" query parameter to
// make the bucket always reference that key.
b, err := blob.OpenBucket(ctx, "mem://?key=foo.txt")
if err != nil {
	return err
}
defer b.Close()

// Bucket operations will ignore the passed-in key and always reference foo.txt.

The resulting bucket will ignore the key parameter to its functions, and always refer to the single key. This can be useful to allow configuration of a specific “file” via a single URL.

List functions will not work on single key buckets.

Using a Bucket🔗

Once you have opened a bucket for the storage provider you want, you can store and access data from it using the standard Go I/O patterns described below. Other operations like listing and reading metadata are documented in the blob package documentation.

Writing Data to a Bucket🔗

To write data to a bucket, you create a writer, write data to it, and then close the writer. Closing the writer commits the write to the provider, flushing any buffers, and releases any resources used while writing, so you must always check the error of Close.

The writer implements io.Writer, so you can use any functions that take an io.Writer like io.Copy or fmt.Fprintln.

// Open the key "foo.txt" for writing with the default options.
w, err := bucket.NewWriter(ctx, "foo.txt", nil)
if err != nil {
	return err
}
_, writeErr := fmt.Fprintln(w, "Hello, World!")
// Always check the return value of Close when writing.
closeErr := w.Close()
if writeErr != nil {
	log.Fatal(writeErr)
}
if closeErr != nil {
	log.Fatal(closeErr)
}

In some cases, you may want to cancel an in-progress write to avoid the blob being created or overwritten. A typical reason for wanting to cancel a write is encountering an error in the stream your program is copying from. To abort a write, you cancel the Context you pass to the writer. Again, you must always Close the writer to release the resources, but in this case you can ignore the error because the write’s failure is expected.

// Create a cancelable context from the existing context.
writeCtx, cancelWrite := context.WithCancel(ctx)
defer cancelWrite()

// Open the key "foo.txt" for writing with the default options.
w, err := bucket.NewWriter(writeCtx, "foo.txt", nil)
if err != nil {
	return err
}

// Assume some writes happened and we encountered an error.
// Now we want to abort the write.

if err != nil {
	// First cancel the context.
	cancelWrite()
	// You must still close the writer to avoid leaking resources.
	w.Close()
}

Reading Data from a Bucket🔗

Once you have written data to a bucket, you can read it back by creating a reader. The reader implements io.Reader, so you can use any functions that take an io.Reader like io.Copy or io/ioutil.ReadAll. You must always close a reader after using it to avoid leaking resources.

// Open the key "foo.txt" for reading with the default options.
r, err := bucket.NewReader(ctx, "foo.txt", nil)
if err != nil {
	return err
}
defer r.Close()
// Readers also have a limited view of the blob's metadata.
fmt.Println("Content-Type:", r.ContentType())
fmt.Println()
// Copy from the reader to stdout.
if _, err := io.Copy(os.Stdout, r); err != nil {
	return err
}

Many storage providers provide efficient random-access to data in buckets. To start reading from an arbitrary offset in the blob, use NewRangeReader.

// Open the key "foo.txt" for reading at offset 1024 and read up to 4096 bytes.
r, err := bucket.NewRangeReader(ctx, "foo.txt", 1024, 4096, nil)
if err != nil {
	return err
}
defer r.Close()
// Copy from the read range to stdout.
if _, err := io.Copy(os.Stdout, r); err != nil {
	return err
}

Deleting a Bucket🔗

You can delete blobs using the Bucket.Delete method.

if err := bucket.Delete(ctx, "foo.txt"); err != nil {
	return err
}

Other Usage Samples🔗

Supported Storage Services🔗

Google Cloud Storage🔗

Google Cloud Storage (GCS) URLs in the Go CDK closely resemble the URLs you would see in the gsutil CLI.

blob.OpenBucket will use Application Default Credentials; if you have authenticated via gcloud auth application-default login, it will use those credentials. See Application Default Credentials to learn about authentication alternatives, including using environment variables.

import (
	"context"

	"gocloud.dev/blob"
	_ "gocloud.dev/blob/gcsblob"
)

// blob.OpenBucket creates a *blob.Bucket from a URL.
// This URL will open the bucket "my-bucket" using default credentials.
bucket, err := blob.OpenBucket(ctx, "gs://my-bucket")
if err != nil {
	return err
}
defer bucket.Close()

Full details about acceptable URLs can be found under the API reference for gcsblob.URLOpener.

GCS Constructor🔗

The gcsblob.OpenBucket constructor opens a GCS bucket. You must first create a *net/http.Client that sends requests authorized by Google Cloud Platform credentials. (You can reuse the same client for any other API that takes in a *gcp.HTTPClient.) You can find functions in the gocloud.dev/gcp package to set this up for you.

import (
	"context"

	"gocloud.dev/blob/gcsblob"
	"gocloud.dev/gcp"
)

// Your GCP credentials.
// See https://cloud.google.com/docs/authentication/production
// for more info on alternatives.
creds, err := gcp.DefaultCredentials(ctx)
if err != nil {
	return err
}

// Create an HTTP client.
// This example uses the default HTTP transport and the credentials
// created above.
client, err := gcp.NewHTTPClient(
	gcp.DefaultTransport(),
	gcp.CredentialsTokenSource(creds))
if err != nil {
	return err
}

// Create a *blob.Bucket.
bucket, err := gcsblob.OpenBucket(ctx, client, "my-bucket", nil)
if err != nil {
	return err
}
defer bucket.Close()

S3🔗

S3 URLs in the Go CDK closely resemble the URLs you would see in the AWS CLI. You should specify the region query parameter to ensure your application connects to the correct region.

If you set the “awssdk=v1” query parameter, blob.OpenBucket will create a default AWS Session with the SharedConfigEnable option enabled; if you have authenticated with the AWS CLI, it will use those credentials. See AWS Session to learn about authentication alternatives, including using environment variables.

If you set the “awssdk=v2” query parameter, it will instead create an AWS Config based on the AWS SDK V2; see AWS V2 Config to learn more.

If no “awssdk” query parameter is set, Go CDK will use a default (currently V1).

Full details about acceptable URLs can be found under the API reference for s3blob.URLOpener.

import (
	"context"

	"gocloud.dev/blob"
	_ "gocloud.dev/blob/s3blob"
)

// blob.OpenBucket creates a *blob.Bucket from a URL.
bucket, err := blob.OpenBucket(ctx, "s3://my-bucket?region=us-west-1")
if err != nil {
	return err
}
defer bucket.Close()

// Forcing AWS SDK V2.
bucket, err = blob.OpenBucket(ctx, "s3://my-bucket?region=us-west-1&awssdk=v2")
if err != nil {
	return err
}
defer bucket.Close()

S3 Constructor🔗

The s3blob.OpenBucket constructor opens an S3 bucket. You must first create an AWS session with the same region as your bucket:

import (
	"context"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"gocloud.dev/blob/s3blob"
)

// Establish an AWS session.
// See https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ for more info.
// The region must match the region for "my-bucket".
sess, err := session.NewSession(&aws.Config{
	Region: aws.String("us-west-1"),
})
if err != nil {
	return err
}

// Create a *blob.Bucket.
bucket, err := s3blob.OpenBucket(ctx, sess, "my-bucket", nil)
if err != nil {
	return err
}
defer bucket.Close()

s3blob.OpenBucketV2 is similar but uses the AWS SDK V2.

import (
	"context"

	awsv2cfg "github.com/aws/aws-sdk-go-v2/config"
	s3v2 "github.com/aws/aws-sdk-go-v2/service/s3"
	"gocloud.dev/blob/s3blob"
)

// Establish a AWS V2 Config.
// See https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/ for more info.
cfg, err := awsv2cfg.LoadDefaultConfig(ctx)
if err != nil {
	return err
}

// Create a *blob.Bucket.
clientV2 := s3v2.NewFromConfig(cfg)
bucket, err := s3blob.OpenBucketV2(ctx, clientV2, "my-bucket", nil)
if err != nil {
	return err
}
defer bucket.Close()

S3-Compatible Servers🔗

The Go CDK can also interact with S3-compatible storage servers that recognize the same REST HTTP endpoints as S3, like Minio, Ceph, or SeaweedFS. You can change the endpoint by changing the Endpoint field on the *aws.Config you pass to s3blob.OpenBucket. If you are using blob.OpenBucket, you can switch endpoints by using the S3 URL using query parameters like so:

bucket, err := blob.OpenBucket("s3://mybucket?" +
    "endpoint=my.minio.local:8080&" +
    "disableSSL=true&" +
    "s3ForcePathStyle=true")

See aws.ConfigFromURLParams for more details on supported URL options for S3.

Azure Blob Storage🔗

Azure Blob Storage URLs in the Go CDK allow you to identify Azure Blob Storage containers when opening a bucket with blob.OpenBucket. Go CDK uses the environment variables AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_KEY, and AZURE_STORAGE_SAS_TOKEN, among others, to configure the credentials.

import (
	"context"

	"gocloud.dev/blob"
	_ "gocloud.dev/blob/azureblob"
)

// blob.OpenBucket creates a *blob.Bucket from a URL.
// This URL will open the container "my-container" using default
// credentials found in environment variables as documented in
// the package.
// Assuming AZURE_STORAGE_ACCOUNT is set to "myaccount",
// and other options aren't set, the service URL will look like:
// "https://myaccount.blob.core.windows.net/my-container".
bucket, err := blob.OpenBucket(ctx, "azblob://my-container")
if err != nil {
	return err
}
defer bucket.Close()

// Another example, against a local emulator.
// Assuming AZURE_STORAGE_ACCOUNT is set to "myaccount",
// the service URL will look like:
// "http://localhost:10001/myaccount/my-container".
localbucket, err := blob.OpenBucket(ctx, "azblob://my-container?protocol=http&domain=localhost:10001")
if err != nil {
	return err
}
defer localbucket.Close()

Full details about acceptable URLs can be found under the API reference for azureblob.URLOpener.

Azure Blob Constructor🔗

The azureblob.OpenBucket constructor opens an Azure Blob Storage container. azureblob operates on Azure Storage Block Blobs. You must first create an Azure Service Client before you can open a container.

import (
	"context"

	"gocloud.dev/blob/azureblob"
)

const (
	// The storage container to access.
	containerName = "my-container"
)

// Construct the service URL.
// There are many forms of service URLs, see ServiceURLOptions.
opts := azureblob.NewDefaultServiceURLOptions()
serviceURL, err := azureblob.NewServiceURL(opts)
if err != nil {
	return err
}

// There are many ways to authenticate to Azure.
// This approach uses environment variables as described in azureblob package
// documentation.
// For example, to use shared key authentication, you would set
// AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY.
// To use a SAS token, you would set AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_SAS_TOKEN.
// You can also construct a client using the azblob constructors directly, like
// azblob.NewServiceClientWithSharedKey.
client, err := azureblob.NewDefaultClient(serviceURL, containerName)
if err != nil {
	return err
}

// Create a *blob.Bucket.
b, err := azureblob.OpenBucket(ctx, client, nil)
if err != nil {
	return err
}
defer b.Close()

// Now we can use b to read or write files to the container.
data, err := b.ReadAll(ctx, "my-key")
if err != nil {
	return err
}
_ = data

Local Storage🔗

The Go CDK provides blob drivers for storing data in memory and on the local filesystem. These are primarily intended for testing and local development, but may be useful in production scenarios where an NFS mount is used.

Local storage URLs take the form of either mem:// or file:/// URLs. Memory URLs are always mem:// with no other information and always create a new bucket. File URLs convert slashes to the operating system’s native file separator, so on Windows, C:\foo\bar would be written as file:///C:/foo/bar.

import (
    "gocloud.dev/blob"
    _ "gocloud.dev/blob/fileblob"
    _ "gocloud.dev/blob/memblob"
)

// ...

bucket1, err := blob.OpenBucket(ctx, "mem://")
if err != nil {
    return err
}
defer bucket1.Close()

bucket2, err := blob.OpenBucket(ctx, "file:///path/to/dir")
if err != nil {
    return err
}
defer bucket2.Close()

Local Storage Constructors🔗

You can create an in-memory bucket with memblob.OpenBucket:

import (
	"context"
	"fmt"

	"gocloud.dev/blob/memblob"
)

// Create an in-memory bucket.
bucket := memblob.OpenBucket(nil)
defer bucket.Close()

// Now we can use bucket to read or write files to the bucket.
err := bucket.WriteAll(ctx, "my-key", []byte("hello world"), nil)
if err != nil {
	return err
}
data, err := bucket.ReadAll(ctx, "my-key")
if err != nil {
	return err
}
fmt.Println(string(data))

// Output:
// hello world

You can use a local filesystem directory with fileblob.OpenBucket:

import (
	"os"

	"gocloud.dev/blob/fileblob"
)

// The directory you pass to fileblob.OpenBucket must exist first.
const myDir = "path/to/local/directory"
if err := os.MkdirAll(myDir, 0777); err != nil {
	return err
}

// Create a file-based bucket.
bucket, err := fileblob.OpenBucket(myDir, nil)
if err != nil {
	return err
}
defer bucket.Close()