Uploading files to OpenStack Swift in Go with Gophercloud
Use Gophercloud v2 to authenticate through Keystone, upload a local file to OpenStack Swift, and check the downloaded bytes against the original. The command below also supports Dynamic Large Objects (DLOs), using a fresh segment prefix for each upload so a shorter replacement cannot inherit old segments.
Prerequisites
You need:
- Go 1.26.9 or a later maintained patch of Go 1.26. This example was tested with Go 1.26.9 on Linux. Go 1.27 is also supported upstream, but is outside this example's tested range. The Go release history explains patch fixes and support.
- Bash for the shell commands below
- Basic knowledge of Go programming
- An existing Keystone v3 and Swift environment, including an
object-storeendpoint in your region's service catalog and permission to create containers and read, write, and delete objects - Your provider's
clouds.yaml, or username/password credentials and the user/project domain
The storage examples were tested against Swift 2.38.2 and Keystone 30.0.0 in a local single-node environment. They do not establish a hosted provider's quotas, replication, or listing consistency. Use a container and object name you own, with no concurrent readers or writers during replacement or deletion. Container names accept ASCII letters, digits, underscores, and hyphens; object names also accept periods. Local file paths are separate from object names and may contain spaces.
Setting up the Go environment
Create a fresh project. Each shell block runs in a subshell, so failures preserve your caller's
directory and options. GOWORK=off keeps an enclosing Go workspace from selecting another module.
An existing swift-upload-example directory is refused; choose another fresh directory and adjust
the later paths instead of deleting an existing project.
(
command -v go >/dev/null || { printf 'Install Go first.\n' >&2; exit 1; }
export GOWORK=off
mkdir swift-upload-example &&
cd swift-upload-example &&
go mod init example.com/swift-upload
)
Installing Gophercloud
Install Gophercloud v2.15.0,
the v2 utils module for credential loading, and the rate limiter used by limited mode. These are the exact
dependency versions used here:
(
export GOWORK=off
cd swift-upload-example &&
go get github.com/gophercloud/gophercloud/v2@v2.15.0 &&
go get github.com/gophercloud/utils/v2/openstack/clientconfig@v2.0.0-20260922174841-b4759ea0529b &&
go get golang.org/x/time/rate@v0.16.0
)
Keep both Gophercloud and utils on their v2 import paths. If installation fails, rerun this block
from the project's parent directory; do not rerun directory creation over it. Keep its go.mod and
go.sum together so subsequent builds use the recorded resolution.
Authenticating with OpenStack Swift
Save each Go block below in the named file inside swift-upload-example. They are separate source
files in one main package; keep their import blocks as printed.
Save your provider's credential file as swift-upload-example/clouds.yaml. For password-based
Keystone v3 authentication, its entry has this shape; replace every placeholder with the values
from your provider:
clouds:
openstack:
auth_type: v3password
auth:
auth_url: https://identity.example.com/v3
username: YOUR_USERNAME
password: YOUR_PASSWORD
project_name: YOUR_PROJECT
user_domain_name: YOUR_USER_DOMAIN
project_domain_name: YOUR_PROJECT_DOMAIN
Keep this file out of version control and readable only by you. The commands select its openstack
entry with OS_CLOUD and select the Swift catalog endpoint with OS_REGION_NAME; replace RegionOne
with your region. For a private CA, set SSL_CERT_FILE to its PEM bundle for the process. Keep TLS
certificate verification enabled.
Save this as auth.go. The default path uses clouds.yaml; the -auth env alternative reads
OS_AUTH_URL, OS_USERNAME, OS_PASSWORD, OS_PROJECT_NAME, and OS_DOMAIN_NAME from your
environment. That small alternative assumes the user and project share a domain; use YAML when
they differ.
package main
import (
"context"
"fmt"
"os"
"github.com/gophercloud/gophercloud/v2"
"github.com/gophercloud/gophercloud/v2/openstack"
"github.com/gophercloud/utils/v2/openstack/clientconfig"
)
func authenticateWithCredentials(ctx context.Context) (*gophercloud.ServiceClient, error) {
opts := gophercloud.AuthOptions{
IdentityEndpoint: os.Getenv("OS_AUTH_URL"),
Username: os.Getenv("OS_USERNAME"),
Password: os.Getenv("OS_PASSWORD"),
TenantName: os.Getenv("OS_PROJECT_NAME"),
DomainName: os.Getenv("OS_DOMAIN_NAME"),
}
provider, err := openstack.AuthenticatedClient(ctx, opts)
if err != nil {
return nil, fmt.Errorf("error creating OpenStack provider client: %w", err)
}
client, err := openstack.NewObjectStorageV1(provider, gophercloud.EndpointOpts{
Region: os.Getenv("OS_REGION_NAME"),
})
if err != nil {
return nil, fmt.Errorf("error creating Swift service client: %w", err)
}
return client, nil
}
func authenticateWithCloudsYAML(ctx context.Context) (*gophercloud.ServiceClient, error) {
opts := &clientconfig.ClientOpts{
Cloud: os.Getenv("OS_CLOUD"),
}
provider, err := clientconfig.AuthenticatedClient(ctx, opts)
if err != nil {
return nil, fmt.Errorf("error creating provider client: %w", err)
}
client, err := openstack.NewObjectStorageV1(provider, gophercloud.EndpointOpts{
Region: os.Getenv("OS_REGION_NAME"),
})
if err != nil {
return nil, fmt.Errorf("error creating Swift service client: %w", err)
}
return client, nil
}
Uploading files to Swift containers
Ensure container exists
Save this as container.go. Create a missing container only after a 404; a denied request is
returned to the caller. Gophercloud's ResponseCodeIs checks the HTTP status through wrapped errors:
package main
import (
"context"
"fmt"
"net/http"
"github.com/gophercloud/gophercloud/v2"
"github.com/gophercloud/gophercloud/v2/openstack/objectstorage/v1/containers"
)
func ensureContainer(ctx context.Context, client *gophercloud.ServiceClient, containerName string) error {
result := containers.Get(ctx, client, containerName, nil)
if result.Err == nil {
return nil
}
if !gophercloud.ResponseCodeIs(result.Err, http.StatusNotFound) {
return fmt.Errorf("error checking container %s: %w", containerName, result.Err)
}
_, err := containers.Create(ctx, client, containerName, containers.CreateOpts{}).Extract()
if err != nil {
return fmt.Errorf("error creating container %s: %w", containerName, err)
}
return nil
}
Upload a file
Save this as upload.go. The input must be a regular file and remain unchanged throughout upload
and verification. The example stores it as application/octet-stream.
package main
import (
"context"
"fmt"
"os"
"github.com/gophercloud/gophercloud/v2"
"github.com/gophercloud/gophercloud/v2/openstack/objectstorage/v1/objects"
)
func uploadFile(ctx context.Context, client *gophercloud.ServiceClient, containerName, objectName, filePath string) error {
file, err := os.Open(filePath)
if err != nil {
return fmt.Errorf("error opening file: %w", err)
}
defer file.Close()
stat, err := file.Stat()
if err != nil {
return fmt.Errorf("error getting file info: %w", err)
}
if !stat.Mode().IsRegular() {
return fmt.Errorf("input must be a regular file")
}
createOpts := objects.CreateOpts{
Content: file,
ContentLength: stat.Size(),
ContentType: "application/octet-stream",
}
result := objects.Create(ctx, client, containerName, objectName, createOpts)
if err := result.Err; err != nil {
return fmt.Errorf("error uploading file: %w", err)
}
return nil
}
By default, Gophercloud computes an MD5 ETag before uploading.
For a seekable *os.File, it hashes and rewinds the file, reading it twice without loading the
whole file into memory. A non-seekable reader is buffered instead. The later verification command
compares SHA-256 hashes of the complete local file and downloaded object, including empty files.
Retry logic for uploads
Save this as retry.go. Retry 429, 502, 503, and 504 responses up to three attempts, waiting 1 and
2 seconds between them. Other failures return immediately. Each attempt reopens the input file.
package main
import (
"context"
"fmt"
"time"
"github.com/gophercloud/gophercloud/v2"
)
func uploadWithRetry(ctx context.Context, client *gophercloud.ServiceClient, containerName, objectName, filePath string) error {
backoff := []time.Duration{time.Second, 2 * time.Second, 5 * time.Second}
var err error
for i, wait := range backoff {
if err = uploadFile(ctx, client, containerName, objectName, filePath); err == nil {
return nil
}
if !retryable(err) {
return err
}
if i == len(backoff)-1 {
break
}
// Waiting on the context means a cancelled upload stops retrying immediately.
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(wait):
}
}
return fmt.Errorf("failed after %d attempts: %w", len(backoff), err)
}
func retryable(err error) bool {
return gophercloud.ResponseCodeIs(err, 429) ||
gophercloud.ResponseCodeIs(err, 502) ||
gophercloud.ResponseCodeIs(err, 503) ||
gophercloud.ResponseCodeIs(err, 504)
}
Retrying uses the same object name and replaces any object already there. A lost response can leave an uncertain write, so inspect the object before assuming a nonzero exit means nothing was stored.
Upload an object with retry (using createopts)
Save this as object-retry.go. Segments use a fresh byte reader on each attempt; reusing a reader
drained by an earlier request can send an empty body on retry.
package main
import (
"bytes"
"context"
"fmt"
"time"
"github.com/gophercloud/gophercloud/v2"
"github.com/gophercloud/gophercloud/v2/openstack/objectstorage/v1/objects"
)
func uploadObjectWithRetry(ctx context.Context, client *gophercloud.ServiceClient, containerName, objectName string, opts objects.CreateOpts, content []byte) error {
backoff := []time.Duration{time.Second, 2 * time.Second, 5 * time.Second}
var err error
for i, wait := range backoff {
attemptOpts := opts
attemptOpts.Content = bytes.NewReader(content)
attemptOpts.ContentLength = int64(len(content))
if err = objects.Create(ctx, client, containerName, objectName, attemptOpts).Err; err == nil {
return nil
}
if !retryable(err) {
return err
}
if i == len(backoff)-1 {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(wait):
}
}
return fmt.Errorf("failed after %d attempts: %w", len(backoff), err)
}
Handling large files with segmented uploads
Save this as large.go. A DLO manifest is a zero-byte object whose X-Object-Manifest header names
a segment container and prefix. Swift concatenates every matching segment in lexicographic name
order, so the names below use padded numbers. Use CreateOpts.ObjectManifest, not custom metadata,
to set the header. See Swift's DLO documentation.
A fresh random prefix isolates each version. Reusing <objectName>/ would leave trailing segments
when a replacement is shorter, and would change the old manifest's contents before publication.
The manifest is written after the segments. DLO assembly depends on container listings, which are
eventually consistent on a distributed Swift deployment; publishing a manifest alone does not prove
that a reader can already retrieve every byte. Run the byte verifier before treating it as complete.
package main
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"io"
"os"
"github.com/gophercloud/gophercloud/v2"
"github.com/gophercloud/gophercloud/v2/openstack/objectstorage/v1/objects"
)
// A random version keeps replacements separate from a live manifest's segments.
func newSegmentPrefix(objectName string) (string, error) {
version := make([]byte, 16)
if _, err := rand.Read(version); err != nil {
return "", fmt.Errorf("error generating segment prefix: %w", err)
}
return fmt.Sprintf("%s/%s/", objectName, hex.EncodeToString(version)), nil
}
func uploadLargeFile(ctx context.Context, client *gophercloud.ServiceClient, containerName, objectName, filePath string) error {
file, err := os.Open(filePath)
if err != nil {
return fmt.Errorf("error opening file: %w", err)
}
defer file.Close()
stat, err := file.Stat()
if err != nil {
return fmt.Errorf("error getting file info: %w", err)
}
if !stat.Mode().IsRegular() {
return fmt.Errorf("input must be a regular file")
}
// Discover destination failures before sending segments.
if err := ensureContainer(ctx, client, containerName); err != nil {
return fmt.Errorf("error ensuring destination container: %w", err)
}
segmentsContainerName := containerName + "_segments"
if err := ensureContainer(ctx, client, segmentsContainerName); err != nil {
return fmt.Errorf("error ensuring segments container: %w", err)
}
segmentPrefix, err := newSegmentPrefix(objectName)
if err != nil {
return err
}
buffer := make([]byte, 5*1024*1024) // 5 MiB segments
segmentNum := 0
for {
// ReadFull fills each segment except the final one despite short filesystem reads.
n, readErr := io.ReadFull(file, buffer)
if n > 0 {
segmentName := fmt.Sprintf("%s%08d", segmentPrefix, segmentNum)
segmentOpts := objects.CreateOpts{ContentType: "application/octet-stream"}
if err := uploadObjectWithRetry(ctx, client, segmentsContainerName, segmentName, segmentOpts, buffer[:n]); err != nil {
return fmt.Errorf("error uploading segment %d: %w", segmentNum, err)
}
segmentNum++
}
if readErr == io.EOF || readErr == io.ErrUnexpectedEOF {
break
}
if readErr != nil {
return fmt.Errorf("error reading file: %w", readErr)
}
}
// A replacement continues to reference its previous prefix until publication.
manifestOpts := objects.CreateOpts{
ContentType: "application/octet-stream",
ObjectManifest: segmentsContainerName + "/" + segmentPrefix,
}
if err := uploadObjectWithRetry(ctx, client, containerName, objectName, manifestOpts, nil); err != nil {
return fmt.Errorf("error creating manifest for large file: %w", err)
}
return nil
}
Failed uploads and replaced versions leave segments behind. Do not sweep <objectName>/: it also
contains live versions and other runs. An uncertain manifest response is especially dangerous,
because the server may have published it despite the caller's failure. General garbage collection
needs proven ownership, current reference checks, and a grace period longer than active uploads
and readers; this example does not implement that collector.
Ordinary DELETE removes a DLO manifest while retaining its segments. SLO's
?multipart-manifest=delete is not a DLO cleanup operation; Swift 2.38.2's SLO handler returns an outer HTTP 200
with a bulk-delete body reporting 400, leaving the DLO intact. The -mode delete command below
removes only the current manifest and its exact version prefix. It assumes exclusive ownership and
stopped readers/writers, and leaves older or failed versions for separate inspection.
Handling errors and best practices
Rate limiting
Save this wrapper as rate.go. It limits admission of whole-file uploads when reused
by a longer-running caller. It does not limit each retry or segmented request; a single CLI
invocation is not evidence of a sustained request rate.
package main
import (
"context"
"fmt"
"github.com/gophercloud/gophercloud/v2"
"golang.org/x/time/rate"
)
type RateLimitedClient struct {
client *gophercloud.ServiceClient
limiter *rate.Limiter
}
func NewRateLimitedClient(client *gophercloud.ServiceClient, rps float64) *RateLimitedClient {
return &RateLimitedClient{
client: client,
limiter: rate.NewLimiter(rate.Limit(rps), 1),
}
}
func (r *RateLimitedClient) UploadFile(ctx context.Context, containerName, objectName, filePath string) error {
if err := r.limiter.Wait(ctx); err != nil {
return fmt.Errorf("rate limit wait error: %w", err)
}
return uploadWithRetry(ctx, r.client, containerName, objectName, filePath)
}
Verify and delete an uploaded object
Save this as verify.go. Hash the local file and the complete downloaded body separately. A DLO's
ETag is not the original file's MD5, so comparing those two ETags would be the wrong check.
package main
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"fmt"
"io"
"os"
"github.com/gophercloud/gophercloud/v2"
"github.com/gophercloud/gophercloud/v2/openstack/objectstorage/v1/objects"
)
var errContentMismatch = errors.New("downloaded bytes differ from the local file")
func verifyFile(ctx context.Context, client *gophercloud.ServiceClient, containerName, objectName, filePath string) error {
file, err := os.Open(filePath)
if err != nil {
return err
}
defer file.Close()
localHash := sha256.New()
localSize, err := io.Copy(localHash, file)
if err != nil {
return err
}
result := objects.Download(ctx, client, containerName, objectName, nil)
if result.Err != nil {
return result.Err
}
defer result.Body.Close()
remoteHash := sha256.New()
remoteSize, err := io.Copy(remoteHash, result.Body)
if err != nil {
return err
}
if localSize != remoteSize || !bytes.Equal(localHash.Sum(nil), remoteHash.Sum(nil)) {
return errContentMismatch
}
fmt.Printf("Verified %s/%s: %d bytes, sha256 %x\n", containerName, objectName, remoteSize, remoteHash.Sum(nil))
return nil
}
Save this as cleanup.go. Check that a DLO refers to this command's segment container, object
name, and a single random version. List that exact prefix before removing the manifest, then delete
only those segments. The prefix is printed first so you can inspect remaining segments if deletion
fails after the manifest is gone. Containers themselves are retained.
package main
import (
"context"
"encoding/hex"
"fmt"
"strings"
"github.com/gophercloud/gophercloud/v2"
"github.com/gophercloud/gophercloud/v2/openstack/objectstorage/v1/objects"
)
func deleteUpload(ctx context.Context, client *gophercloud.ServiceClient, containerName, objectName string) error {
header, err := objects.Get(ctx, client, containerName, objectName, nil).Extract()
if err != nil {
return err
}
var names []string
segmentsContainer := containerName + "_segments"
if header.ObjectManifest != "" {
parts := strings.Split(header.ObjectManifest, "/")
if len(parts) != 4 || parts[0] != segmentsContainer || parts[1] != objectName ||
len(parts[2]) != 32 || parts[3] != "" {
return fmt.Errorf("refusing a manifest outside this upload's namespace")
}
if _, err := hex.DecodeString(parts[2]); err != nil {
return fmt.Errorf("refusing an invalid version prefix")
}
prefix := objectName + "/" + parts[2] + "/"
pages, err := objects.List(client, segmentsContainer, objects.ListOpts{Prefix: prefix}).AllPages(ctx)
if err != nil {
return err
}
names, err = objects.ExtractNames(pages)
if err != nil {
return err
}
fmt.Printf("Segment prefix: %s\n", header.ObjectManifest)
}
if err := objects.Delete(ctx, client, containerName, objectName, nil).Err; err != nil {
return err
}
for _, name := range names {
if err := objects.Delete(ctx, client, segmentsContainer, name, nil).Err; err != nil {
return err
}
}
return nil
}
Run the complete command
Save this as main.go. The default mode uploads a file; large uses DLO segments, limited
calls the rate-limited wrapper, verify downloads and compares bytes, and delete performs the
scoped cleanup above. Ctrl+C requests cancellation. An error returns a nonzero status without
printing credentials or raw API responses.
package main
import (
"context"
"errors"
"flag"
"fmt"
"os"
"os/signal"
"regexp"
"syscall"
"github.com/gophercloud/gophercloud/v2"
)
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, failureMessage(err))
os.Exit(1)
}
}
func run() error {
mode := flag.String("mode", "file", "file, large, limited, verify, or delete")
auth := flag.String("auth", "yaml", "yaml or env")
flag.Parse()
args := flag.Args()
if *mode != "file" && *mode != "large" && *mode != "limited" && *mode != "verify" && *mode != "delete" {
return fmt.Errorf("unknown mode")
}
if *auth != "yaml" && *auth != "env" {
return fmt.Errorf("unknown authentication mode")
}
if (*mode == "delete" && len(args) != 2) || (*mode != "delete" && len(args) != 3) {
return fmt.Errorf("usage: swift-upload [-mode MODE] [-auth yaml|env] CONTAINER OBJECT [FILE]")
}
containerName, objectName := args[0], args[1]
if !regexp.MustCompile(`^[A-Za-z0-9_-]+$`).MatchString(containerName) ||
!regexp.MustCompile(`^[A-Za-z0-9_.-]+$`).MatchString(objectName) {
return fmt.Errorf("use plain ASCII container and object names")
}
if os.Getenv("OS_REGION_NAME") == "" || (*auth == "yaml" && os.Getenv("OS_CLOUD") == "") {
return fmt.Errorf("set OS_REGION_NAME and, for YAML authentication, OS_CLOUD")
}
if *mode != "delete" {
stat, err := os.Stat(args[2])
if err != nil {
return fmt.Errorf("cannot inspect input: %w", err)
}
if !stat.Mode().IsRegular() {
return fmt.Errorf("input must be a regular file")
}
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
var client *gophercloud.ServiceClient
var err error
if *auth == "env" {
client, err = authenticateWithCredentials(ctx)
} else {
client, err = authenticateWithCloudsYAML(ctx)
}
if err != nil {
return fmt.Errorf("authentication or catalog lookup failed: %w", err)
}
if *mode == "verify" {
return verifyFile(ctx, client, containerName, objectName, args[2])
}
if *mode == "delete" {
if err := deleteUpload(ctx, client, containerName, objectName); err != nil {
return fmt.Errorf("deletion failed: %w", err)
}
fmt.Printf("Deleted %s/%s\n", containerName, objectName)
return nil
}
if err := ensureContainer(ctx, client, containerName); err != nil {
return err
}
switch *mode {
case "large":
err = uploadLargeFile(ctx, client, containerName, objectName, args[2])
case "limited":
err = NewRateLimitedClient(client, 1).UploadFile(ctx, containerName, objectName, args[2])
default:
err = uploadWithRetry(ctx, client, containerName, objectName, args[2])
}
if err != nil {
return err
}
fmt.Printf("Uploaded %s/%s\n", containerName, objectName)
return nil
}
func failureMessage(err error) string {
if errors.Is(err, errContentMismatch) {
return "Verification failed: downloaded bytes differ from the local file."
}
if errors.Is(err, context.Canceled) {
return "Operation canceled; inspect stored objects before retrying."
}
for _, status := range []int{401, 403, 404, 429, 502, 503, 504} {
if gophercloud.ResponseCodeIs(err, status) {
return fmt.Sprintf("Operation failed with HTTP %d; check credentials, access, and the target object.", status)
}
}
return "Operation failed; check arguments, cloud configuration, and the local input file."
}
Create a small binary fixture without overwriting an existing file:
(
set -C
cd swift-upload-example &&
printf '\000\377Swift\n' > example.bin
)
Build, upload, and verify it. Change exports to your owned container and RegionOne to your
catalog region in both commands. Building replaces this project's swift-upload executable; a
failed build stops the block before any previous executable runs.
(
export GOWORK=off
cd swift-upload-example &&
go build -o swift-upload . &&
OS_CLOUD=openstack OS_REGION_NAME=RegionOne ./swift-upload exports example.bin ./example.bin &&
OS_CLOUD=openstack OS_REGION_NAME=RegionOne ./swift-upload -mode verify exports example.bin ./example.bin
)
Expected output for that eight-byte fixture:
Uploaded exports/example.bin
Verified exports/example.bin: 8 bytes, sha256 2949a2f975f056f2652117015349d0768d9eaffcd9eacd14f81147844c47f158
For your own larger file, use the same executable with -mode large, then verify the assembled
object. Uploading the same object name replaces its contents; DLO replacements retain older segment
versions. If verification fails because the DLO listing has not converged, retry verification
without uploading again. A hash mismatch can also mean missing or changed segments, so inspect
the exact prefix before deleting anything.
(
cd swift-upload-example &&
OS_CLOUD=openstack OS_REGION_NAME=RegionOne ./swift-upload -mode large exports backup.bin ./backup.bin &&
OS_CLOUD=openstack OS_REGION_NAME=RegionOne ./swift-upload -mode verify exports backup.bin ./backup.bin
)
After stopping readers and writers, remove the current version you own:
(
cd swift-upload-example &&
OS_CLOUD=openstack OS_REGION_NAME=RegionOne ./swift-upload -mode delete exports backup.bin
)
Read the printed prefix if cleanup fails, and inspect only that owned version. A second delete after the manifest is gone returns 404; it cannot discover leftover segments from that header. Never delete a neighboring version merely because its name starts with the same object name.