Uploading files to OpenStack Swift in Go with Gophercloud

Tim Koschützki

Co-founder · Berlin, Germany ·

Efficient file storage and management are crucial for modern applications. OpenStack Swift, an open-source object storage system, offers a scalable solution for storing and retrieving large amounts of data in the cloud. This DevTip explains how to upload files to OpenStack Swift using Go and the Gophercloud library, a robust Go SDK for interacting with OpenStack APIs.

Prerequisites

Before you begin, ensure you have:

Go installed (version 1.22 or higher)
Basic knowledge of Go programming
Access to an OpenStack Swift environment
OpenStack credentials (configurable via clouds.yaml)

Setting up the Go environment

First, create a new directory for your project and initialize a Go module:

mkdir swift-upload-example
cd swift-upload-example
go mod init example.com/swift-upload

Installing Gophercloud

Gophercloud is an open-source Go SDK for working with OpenStack APIs. Install the v2 version, plus the v2 line of the companion utils module and the rate limiter used further down:

go get github.com/gophercloud/gophercloud/v2
go get github.com/gophercloud/utils/v2/openstack/clientconfig
go get golang.org/x/time/rate

Both the utils and x/time modules keep their packages in subdirectories, so asking for the package paths above is what you want. The utils module also has its own major version: pulling the v1 path alongside Gophercloud v2 gives you a *gophercloud.ServiceClient from a different package, which will not type-check against anything below.

Authenticating with OpenStack Swift

Create a new file named main.go. Every Go fence below is a fragment of that one file, so the repeated package main lines and import blocks fold together; paste each new function in and keep one merged import block. Below are two methods of authentication: using direct credentials and clouds.yaml.

package main

import (
	"context"
	"fmt"

	"github.com/gophercloud/gophercloud/v2"
	"github.com/gophercloud/gophercloud/v2/openstack"
	"github.com/gophercloud/utils/v2/openstack/clientconfig"
)

func authenticateWithCredentials(ctx context.Context) (*gophercloud.ServiceClient, error) {
	opts := gophercloud.AuthOptions{
		IdentityEndpoint: "https://your-openstack-auth-url",
		Username:         "your-username",
		Password:         "your-password",
		TenantName:       "your-tenant-name",
		DomainName:       "your-domain-name",
	}

	provider, err := openstack.AuthenticatedClient(ctx, opts)
	if err != nil {
		return nil, fmt.Errorf("error creating OpenStack provider client: %w", err)
	}

	client, err := openstack.NewObjectStorageV1(provider, gophercloud.EndpointOpts{
		Region: "your-region",
	})
	if err != nil {
		return nil, fmt.Errorf("error creating Swift service client: %w", err)
	}

	return client, nil
}

func authenticateWithCloudsYAML(ctx context.Context) (*gophercloud.ServiceClient, error) {
	opts := &clientconfig.ClientOpts{
		Cloud: "openstack", // Name of the cloud in clouds.yaml
	}

	provider, err := clientconfig.AuthenticatedClient(ctx, opts)
	if err != nil {
		return nil, fmt.Errorf("error creating provider client: %w", err)
	}

	client, err := openstack.NewObjectStorageV1(provider, gophercloud.EndpointOpts{})
	if err != nil {
		return nil, fmt.Errorf("error creating Swift service client: %w", err)
	}

	return client, nil
}

Uploading files to Swift containers

Ensure container exists

Before uploading, verify that the target container exists. If it does not, create it with robust error handling. Note that in v2 every request function takes a context.Context as its first argument, and that Gophercloud ships ResponseCodeIs so you do not have to pattern-match on error strings:

package main

import (
	"context"
	"fmt"
	"net/http"

	"github.com/gophercloud/gophercloud/v2"
	"github.com/gophercloud/gophercloud/v2/openstack/objectstorage/v1/containers"
)

func ensureContainer(ctx context.Context, client *gophercloud.ServiceClient, containerName string) error {
	result := containers.Get(ctx, client, containerName, nil)
	if result.Err == nil {
		return nil // Container exists
	}
	if !gophercloud.ResponseCodeIs(result.Err, http.StatusNotFound) {
		return fmt.Errorf("error checking container %s: %w", containerName, result.Err)
	}

	// Create the container since it does not exist
	_, err := containers.Create(ctx, client, containerName, containers.CreateOpts{}).Extract()
	if err != nil {
		return fmt.Errorf("error creating container %s: %w", containerName, err)
	}

	return nil
}

Upload a file

Use the following function to upload a file with proper content type and error handling.

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/gophercloud/gophercloud/v2"
	"github.com/gophercloud/gophercloud/v2/openstack/objectstorage/v1/objects"
)

func uploadFile(ctx context.Context, client *gophercloud.ServiceClient, containerName, objectName, filePath string) error {
	file, err := os.Open(filePath)
	if err != nil {
		return fmt.Errorf("error opening file: %w", err)
	}
	defer file.Close()

	stat, err := file.Stat()
	if err != nil {
		return fmt.Errorf("error getting file info: %w", err)
	}

	createOpts := objects.CreateOpts{
		Content:       file,
		ContentLength: stat.Size(),
		ContentType:   "application/octet-stream",
	}

	result := objects.Create(ctx, client, containerName, objectName, createOpts)
	if err := result.Err; err != nil {
		return fmt.Errorf("error uploading file: %w", err)
	}

	return nil
}

Content is an io.Reader, but Gophercloud does not simply stream it. Unless you set NoETag or supply your own ETag, CreateOpts.ToObjectCreateParams computes an MD5 checksum of the body first. For an io.ReadSeeker such as the *os.File above it hashes and then seeks back, so the file is read twice; for any other reader it calls io.ReadAll and holds the whole body in memory. Passing a pipe or a network response here is therefore not memory-bounded. Set NoETag: true when you want a true single-pass stream, and accept that Swift then has no client-side integrity check.

Retry logic for uploads

Implement a retry mechanism to handle transient errors when uploading files.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/gophercloud/gophercloud/v2"
)

func uploadWithRetry(ctx context.Context, client *gophercloud.ServiceClient, containerName, objectName, filePath string) error {
	backoff := []time.Duration{time.Second, 2 * time.Second, 5 * time.Second}
	var err error

	for i, wait := range backoff {
		if err = uploadFile(ctx, client, containerName, objectName, filePath); err == nil {
			return nil
		}

		if i == len(backoff)-1 {
			break
		}

		// Waiting on the context means a cancelled upload stops retrying immediately.
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-time.After(wait):
		}
	}

	return fmt.Errorf("failed after %d attempts: %w", len(backoff), err)
}

uploadFile reopens the file on every attempt, which is what makes it safe to retry. The generic version below has to be more careful.

Upload an object with retry (using createopts)

For uploading segments or objects that do not originate from file paths, use this generalized retry function. It takes the payload as a byte slice and builds a fresh reader for every attempt: a CreateOpts whose Content reader was already drained by a failed attempt would silently upload zero bytes on the next one, and every error check would still pass.

package main

import (
	"bytes"
	"context"
	"fmt"
	"time"

	"github.com/gophercloud/gophercloud/v2"
	"github.com/gophercloud/gophercloud/v2/openstack/objectstorage/v1/objects"
)

func uploadObjectWithRetry(ctx context.Context, client *gophercloud.ServiceClient, containerName, objectName string, opts objects.CreateOpts, content []byte) error {
	backoff := []time.Duration{time.Second, 2 * time.Second, 5 * time.Second}
	var err error

	for i, wait := range backoff {
		attemptOpts := opts
		attemptOpts.Content = bytes.NewReader(content)
		attemptOpts.ContentLength = int64(len(content))

		if err = objects.Create(ctx, client, containerName, objectName, attemptOpts).Err; err == nil {
			return nil
		}
		if i == len(backoff)-1 {
			break
		}

		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-time.After(wait):
		}
	}

	return fmt.Errorf("failed after %d attempts: %w", len(backoff), err)
}

Handling large files with segmented uploads

For large files, divide the file into segments and upload each segment separately. Then, create a manifest object to assemble these segments. This builds a Dynamic Large Object: the manifest is a zero-byte object whose X-Object-Manifest header names a container and prefix, and Swift concatenates everything under that prefix on read. CreateOpts has a dedicated ObjectManifest field for it; putting the header in Metadata instead sends X-Object-Meta-X-Object-Manifest and produces a manifest that assembles nothing, without failing any error check.

Because Swift concatenates everything under that prefix, the prefix has to be unique per upload. A fixed <objectName>/ prefix breaks as soon as you upload the same object twice: replacing an 11 MB file with a 2 MB one overwrites segment 00000000 and leaves 00000001 and 00000002 behind, so the manifest then serves 2 MB of the new file followed by 6 MB of the old one. Overwriting segments in place is also visible to readers immediately, because the live manifest already points at them. Give every upload its own random version prefix instead.

package main

import (
	"context"
	"crypto/rand"
	"encoding/hex"
	"fmt"
	"io"
	"os"

	"github.com/gophercloud/gophercloud/v2"
	"github.com/gophercloud/gophercloud/v2/openstack/objectstorage/v1/objects"
)

// newSegmentPrefix returns a prefix that no other upload of the same object can collide with, so
// the segments a live manifest points at are never overwritten or extended by a later run.
func newSegmentPrefix(objectName string) (string, error) {
	version := make([]byte, 16)
	if _, err := rand.Read(version); err != nil {
		return "", fmt.Errorf("error generating segment prefix: %w", err)
	}

	return fmt.Sprintf("%s/%s/", objectName, hex.EncodeToString(version)), nil
}

func uploadLargeFile(ctx context.Context, client *gophercloud.ServiceClient, containerName, objectName, filePath string) error {
	file, err := os.Open(filePath)
	if err != nil {
		return fmt.Errorf("error opening file: %w", err)
	}
	defer file.Close()

	// Both containers have to exist before any of this is worth doing: a missing destination
	// container only fails at the manifest, once every segment has already been paid for.
	if err := ensureContainer(ctx, client, containerName); err != nil {
		return fmt.Errorf("error ensuring destination container: %w", err)
	}

	// Use a separate container for segments
	segmentsContainerName := containerName + "_segments"
	if err := ensureContainer(ctx, client, segmentsContainerName); err != nil {
		return fmt.Errorf("error ensuring segments container: %w", err)
	}

	// One prefix per upload, chosen before the first segment is written and used even when the file
	// turns out to be empty and no segment is written at all.
	segmentPrefix, err := newSegmentPrefix(objectName)
	if err != nil {
		return err
	}

	buffer := make([]byte, 5*1024*1024) // 5 MB segments
	segmentNum := 0

	for {
		// ReadFull keeps every segment except the last one exactly 5 MB, so the segment names stay
		// in size order. A bare Read can return short and produce a ragged set of segments.
		n, readErr := io.ReadFull(file, buffer)
		if n > 0 {
			segmentName := fmt.Sprintf("%s%08d", segmentPrefix, segmentNum)
			segmentOpts := objects.CreateOpts{ContentType: "application/octet-stream"}
			if err := uploadObjectWithRetry(ctx, client, segmentsContainerName, segmentName, segmentOpts, buffer[:n]); err != nil {
				return fmt.Errorf("error uploading segment %d: %w", segmentNum, err)
			}
			segmentNum++
		}

		if readErr == io.EOF || readErr == io.ErrUnexpectedEOF {
			break
		}
		if readErr != nil {
			return fmt.Errorf("error reading file: %w", readErr)
		}
	}

	// Publish last. A brand new object 404s until this lands, and a replacement keeps serving the
	// previous version in full, because the manifest it is replacing names the previous prefix.
	manifestOpts := objects.CreateOpts{
		ContentType:    "application/octet-stream",
		ObjectManifest: segmentsContainerName + "/" + segmentPrefix,
	}
	if err := uploadObjectWithRetry(ctx, client, containerName, objectName, manifestOpts, nil); err != nil {
		return fmt.Errorf("error creating manifest for large file: %w", err)
	}

	return nil
}

Two limits worth knowing before you ship this. A failed run leaves the segments it already uploaded behind, so you need a separate sweep of the _segments container. That sweep must not simply delete everything under <objectName>/: the version prefix the live manifest points at lives there too. Read the manifest's X-Object-Manifest header first (objects.Get(...).Extract() hands it back as GetHeader.ObjectManifest), keep that one prefix, and only delete version prefixes that are both unreferenced and older than your longest possible upload, so you never collect a run that is still in flight.

And deleting the manifest does not delete its segments. There is no single call that does both for a Dynamic Large Object: ?multipart-manifest=delete belongs to Static Large Objects, and Swift ignores it here, so the manifest disappears while you keep paying for every segment. Clean up in two steps instead: list <container>_segments with the exact prefix the manifest names, delete those objects (objects.BulkDelete takes a batch of names), and then delete the manifest. The swift CLI's swift delete <container> <objectName> does the same walk for you, reading the manifest header rather than guessing the prefix. See the Swift large object docs for how the two manifest types differ.

Handling errors and best practices

Rate limiting

Implement rate limiting to avoid overwhelming the Swift API. This example uses Go's rate limiter.

package main

import (
	"context"
	"fmt"

	"github.com/gophercloud/gophercloud/v2"
	"golang.org/x/time/rate"
)

type RateLimitedClient struct {
	client  *gophercloud.ServiceClient
	limiter *rate.Limiter
}

func NewRateLimitedClient(client *gophercloud.ServiceClient, rps float64) *RateLimitedClient {
	return &RateLimitedClient{
		client:  client,
		limiter: rate.NewLimiter(rate.Limit(rps), 1),
	}
}

func (r *RateLimitedClient) UploadFile(ctx context.Context, containerName, objectName, filePath string) error {
	if err := r.limiter.Wait(ctx); err != nil {
		return fmt.Errorf("rate limit wait error: %w", err)
	}

	return uploadWithRetry(ctx, r.client, containerName, objectName, filePath)
}

Conclusion

Uploading files to OpenStack Swift using Go and Gophercloud v2 provides a flexible and efficient way to manage cloud storage. By incorporating proper error handling, retry logic, and rate limiting, you can build reliable systems that gracefully handle various scenarios.

For more advanced workflows, Transloadit offers a suite of file handling services. Explore our file exporting service for additional options to streamline your file processing.

Happy coding!