Efficient file storage and management are crucial for modern applications. OpenStack Swift, an open-source object storage system, offers a scalable solution for storing and retrieving large amounts of data in the cloud. This DevTip explores how to upload files to OpenStack Swift using Go and the Gophercloud library, a Go SDK for interacting with OpenStack APIs.

Prerequisites

Before we begin, ensure you have:

  • Go installed (version 1.21 or higher)
  • Basic knowledge of Go programming
  • Access to an OpenStack Swift environment
  • OpenStack credentials: username, password, tenant name, and authentication URL

Setting up the Go environment

First, create a new directory for your project and initialize a Go module:

mkdir swift-upload-example
cd swift-upload-example
go mod init example.com/swift-upload

Installing Gophercloud

Gophercloud is an open-source Go SDK for working with OpenStack APIs. Install it using the following command:

go get github.com/gophercloud/gophercloud

Authenticating with OpenStack Swift

Create a new file named main.go and add the following code to authenticate with OpenStack Swift:

package main

import (
	"fmt"

	"github.com/gophercloud/gophercloud"
	"github.com/gophercloud/gophercloud/openstack"
)

func main() {
	// Set up authentication options
	opts := gophercloud.AuthOptions{
		IdentityEndpoint: "https://your-openstack-auth-url",
		Username:         "your-username",
		Password:         "your-password",
		TenantName:       "your-tenant-name",
		DomainName:       "your-domain-name",
	}

	// Authenticate and obtain a provider client
	provider, err := openstack.AuthenticatedClient(opts)
	if err != nil {
		fmt.Println("Error creating OpenStack provider client:", err)
		return
	}

	// Create an Object Storage (Swift) service client
	client, err := openstack.NewObjectStorageV1(provider, gophercloud.EndpointOpts{
		Region: "your-region",
	})
	if err != nil {
		fmt.Println("Error creating Swift service client:", err)
		return
	}

	fmt.Println("Successfully authenticated with OpenStack Swift")

	// Continue with file upload...
}

Replace the placeholder values in AuthOptions with your actual OpenStack credentials. Make sure to specify the correct DomainName and Region if required by your OpenStack environment.

Uploading files to Swift containers

Now, let's add the functionality to upload a file to a Swift container. Update your main.go file with the following code:

package main

import (
	"fmt"
	"os"

	"github.com/gophercloud/gophercloud"
	"github.com/gophercloud/gophercloud/openstack"
	"github.com/gophercloud/gophercloud/openstack/objectstorage/v1/containers"
	"github.com/gophercloud/gophercloud/openstack/objectstorage/v1/objects"
)

func main() {
	// ... (previous authentication code)

	// Upload a file
	containerName := "my-container"
	objectName := "example.txt"
	filePath := "path/to/your/file.txt"

	err := uploadFile(client, containerName, objectName, filePath)
	if err != nil {
		fmt.Println("Error uploading file:", err)
		return
	}

	fmt.Printf("File '%s' uploaded successfully to container '%s'\n", objectName, containerName)
}

// uploadFile uploads a file to the specified container in Swift
func uploadFile(client *gophercloud.ServiceClient, containerName, objectName, filePath string) error {
	// Create the container if it doesn't exist
	_, err := containers.Create(client, containerName, nil).Extract()
	if err != nil {
		return fmt.Errorf("error creating container: %v", err)
	}

	// Open the file
	file, err := os.Open(filePath)
	if err != nil {
		return fmt.Errorf("error opening file: %v", err)
	}
	defer file.Close()

	// Create the object in Swift
	createOpts := objects.CreateOpts{
		Content: file,
	}

	// Upload the file
	_, err = objects.Create(client, containerName, objectName, createOpts).Extract()
	if err != nil {
		return fmt.Errorf("error uploading file: %v", err)
	}

	return nil
}

This code:

  • Imports the necessary packages, including os for file operations.
  • Implements the uploadFile function to check if the container exists and create it if it doesn't.
  • Opens the file specified by filePath and uploads it to the container using the Swift client.

Handling errors and best practices

Error handling

It's crucial to check for errors at each step of the process. This allows you to catch and handle issues early, providing better feedback and preventing unexpected behavior. Always handle errors returned by functions and provide meaningful messages.

Using concurrency for multiple uploads

For uploading multiple files, consider using Go's concurrency features to improve performance. Here's an example of concurrent uploads using goroutines and a WaitGroup:

package main

import (
	"fmt"
	"path/filepath"
	"sync"

	"github.com/gophercloud/gophercloud"
)

func uploadFilesConcurrently(client *gophercloud.ServiceClient, containerName string, files []string) {
	var wg sync.WaitGroup

	for _, filePath := range files {
		wg.Add(1)
		go func(f string) {
			defer wg.Done()
			objectName := filepath.Base(f)
			err := uploadFile(client, containerName, objectName, f)
			if err != nil {
				fmt.Printf("Error uploading '%s': %v\n", f, err)
			} else {
				fmt.Printf("Successfully uploaded '%s'\n", f)
			}
		}(filePath)
	}

	wg.Wait()
}

Implementing retry logic

Implement a retry mechanism for failed uploads, especially when dealing with large files or unstable networks:

package main

import (
	"fmt"
	"time"

	"github.com/gophercloud/gophercloud"
)

func uploadWithRetry(client *gophercloud.ServiceClient, containerName, objectName, filePath string, maxRetries int) error {
	for i := 0; i < maxRetries; i++ {
		err := uploadFile(client, containerName, objectName, filePath)
		if err == nil {
			return nil
		}
		fmt.Printf("Upload attempt %d failed: %v. Retrying...\n", i+1, err)
		time.Sleep(time.Second * time.Duration(i+1)) // Exponential backoff
	}
	return fmt.Errorf("failed to upload file after %d attempts", maxRetries)
}

Conclusion

Uploading files to OpenStack Swift using Go and Gophercloud provides a powerful and flexible way to manage cloud storage in your applications. By following these best practices and error-handling techniques, you can create robust and efficient file upload systems tailored to your needs.

While this guide focuses on OpenStack Swift, Transloadit offers a range of file handling services that can simplify your workflow even further. Check out our file exporting service for more advanced and user-friendly options.