Effortless file export to OpenStack Swift in .NET C#

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands ·

Upload files through Swift’s HTTP API with .NET’s built-in HttpClient. This example creates real segments and publishes a static large-object manifest only after every segment is acknowledged.

Introduction to OpenStack Swift

Swift stores objects in containers. Its static large-object API combines an ordered list of uploaded segments into one downloadable object. The deployment must enable the SLO middleware for this example.

Setting up your environment

Prerequisites

Install the .NET 10 SDK. Obtain a valid scoped token and the object-storage account URL from your OpenStack authentication workflow. Your account needs permission to create containers and objects. The sample uses 5 MiB segments and at most 1,000 segments (about 4.88 GiB); confirm your deployment’s SLO, object-size, and quota limits allow these values.

dotnet new console --framework net10.0 --name SwiftExport
cd SwiftExport

Secure configuration management

Set SWIFT_STORAGE_URL to the complete HTTPS account endpoint from the service catalog, including its /v1/AUTH_… path. Set SWIFT_TOKEN to the scoped token. Supply them through your shell or secret manager. Token acquisition and renewal belong to your deployment’s Keystone workflow; the program exits on an expired token instead of pretending a retry can renew it.

Implementing Swift storage operations

Replace Program.cs with this complete program. The injected HttpClient belongs to the caller; the exporter disposes each request and response. Redirects are disabled so the token is not forwarded to a different destination.

using System.Net.Http.Headers;
using System.Security.Cryptography;
using System.Text.Json;

public static class Program
{
    public static async Task<int> Main(string[] args)
    {
        using var deadline = new CancellationTokenSource(TimeSpan.FromMinutes(5));
        Console.CancelKeyPress += (_, e) => { e.Cancel = true; deadline.Cancel(); };
        try
        {
            if (args.Length != 3) throw new ArgumentException("Pass container, object key, and local file.");
            var endpoint = new Uri(Required("SWIFT_STORAGE_URL"));
            if (endpoint.Scheme != "https" || endpoint.UserInfo.Length != 0 ||
                endpoint.Query.Length != 0 || endpoint.Fragment.Length != 0)
                throw new ArgumentException("Use the HTTPS account endpoint from the service catalog.");
            using var handler = new SocketsHttpHandler { AllowAutoRedirect = false, MaxConnectionsPerServer = 4 };
            using var http = new HttpClient(handler) { Timeout = Timeout.InfiniteTimeSpan };
            var exporter = new SwiftExporter(http, endpoint, Required("SWIFT_TOKEN"));
            await exporter.UploadAsync(args[0], args[1], args[2], deadline.Token);
            Console.WriteLine("Uploaded.");
            return 0;
        }
        catch (OperationCanceledException) { Console.Error.WriteLine("Upload canceled or timed out."); }
        catch (HttpRequestException error) { Console.Error.WriteLine($"Swift request failed (HTTP {error.StatusCode})."); }
        catch (Exception) { Console.Error.WriteLine("Upload failed. Check the file and Swift configuration."); }
        return 1;
    }

    private static string Required(string name) =>
        Environment.GetEnvironmentVariable(name) is { Length: > 0 } value
            ? value : throw new ArgumentException($"Missing {name}.");
}

public sealed class SwiftExporter(HttpClient http, Uri endpoint, string token)
{
    private const int SegmentSize = 5 * 1024 * 1024;
    private const int MaxSegments = 1000;

    private static string Encode(string value)
    {
        var parts = value.Split('/');
        if (parts.Any(part => part.Length == 0 || part == "." || part == ".."))
            throw new ArgumentException("Use nonempty path segments other than dot or dot-dot.");
        return string.Join("/", parts.Select(Uri.EscapeDataString));
    }

    private async Task<string?> PutAsync(string path, HttpContent content,
        bool container, CancellationToken ct, string? etag = null, string query = "")
    {
        content.Headers.ContentType ??= new MediaTypeHeaderValue("application/octet-stream");
        using var request = new HttpRequestMessage(HttpMethod.Put,
            endpoint.AbsoluteUri.TrimEnd('/') + "/" + Encode(path) + query) { Content = content };
        request.Headers.Add("X-Auth-Token", token);
        if (etag != null) request.Headers.Add("ETag", etag);
        using var response = await http.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, ct);
        var status = (int)response.StatusCode;
        if (status != 201 && !(container && status == 202))
            throw new HttpRequestException("Unexpected Swift response.", null, response.StatusCode);
        return response.Headers.ETag?.Tag.Trim('"');
    }

    public async Task UploadAsync(string container, string key, string file, CancellationToken ct)
    {
        if (container.Contains('/')) throw new ArgumentException("A container cannot contain slashes.");
        Encode(container);
        Encode(key);
        await using var input = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read);
        long length = input.Length;
        if (length > (long)SegmentSize * MaxSegments) throw new IOException("File exceeds the example limit.");
        await PutAsync(container, new ByteArrayContent([]), true, ct);
        if (length == 0)
        {
            await PutAsync(container + "/" + key, new ByteArrayContent([]), false, ct);
            return;
        }

        string segments = container + "_segments";
        await PutAsync(segments, new ByteArrayContent([]), true, ct);
        string version = Guid.NewGuid().ToString("N");
        var manifest = new List<object>();
        var buffer = new byte[SegmentSize];
        long remaining = length;
        for (int index = 0; remaining > 0; index++)
        {
            int count = (int)Math.Min(remaining, SegmentSize);
            await input.ReadExactlyAsync(buffer.AsMemory(0, count), ct);
            string hash = Convert.ToHexString(MD5.HashData(buffer.AsSpan(0, count))).ToLowerInvariant();
            string name = $"{segments}/{version}/{index:D8}";
            using var content = new ByteArrayContent(buffer, 0, count);
            content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
            string? returned = await PutAsync(name, content, false, ct, hash);
            if (!string.Equals(hash, returned, StringComparison.OrdinalIgnoreCase))
                throw new IOException("Swift did not confirm the segment checksum.");
            manifest.Add(new { path = "/" + name, etag = hash, size_bytes = count });
            remaining -= count;
        }
        if (input.Length != length || input.ReadByte() != -1) throw new IOException("The input file changed.");
        using var manifestContent = new StringContent(JsonSerializer.Serialize(manifest));
        manifestContent.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
        await PutAsync(container + "/" + key, manifestContent, false, ct,
            query: "?multipart-manifest=put");
    }
}

Handling large files

Each nonempty file gets a fresh segment prefix in <container>_segments. Sequential uploads keep the payload buffer at 5 MiB. The manifest contains each segment’s path, ETag, and size, which Swift validates before publishing the object. An empty file uses a regular empty PUT because an SLO manifest must contain an object-backed segment.

The remote object key can contain spaces and Unicode. Encoding preserves path separators and rejects empty, dot, and dot-dot segments to avoid URI normalization. Keep the local file unchanged throughout the upload; size checks cannot detect every concurrent same-size edit.

Implementing robust error handling

Container creation accepts HTTP 201 or 202; object and manifest uploads require 201. Quota, authentication, checksum, connection, and cancellation failures exit nonzero. Ctrl+C and the five-minute deadline propagate to file reads and HTTP operations. This example does not retry.

A failed attempt may leave segments behind. A fresh prefix prevents a later attempt from mixing old and new segments. Record and remove only unreferenced upload prefixes through your storage maintenance workflow. Do not blindly delete segments after a manifest request fails: its response may have been lost even though publication succeeded. Replacing an object also leaves its old segments for that maintenance workflow.

Performance optimization

Reuse the client for a batch of exports. The connection limit belongs to SocketsHttpHandler. If you add concurrent uploads, bound the concurrency and budget one segment buffer per operation.

Practical example: complete implementation

With both environment variables configured, run this from the project directory:

printf 'example export\n' > report.txt
dotnet run -- exports 'reports/report #1.txt' report.txt

The command creates exports and exports_segments if necessary, uploads the segments, and finally publishes reports/report #1.txt. Its exit status is zero only after the final successful response.

Conclusion

Swift segmentation is an explicit upload-and-manifest workflow. Using the documented HTTP contract keeps response handling, resource ownership, and cancellation visible in a small .NET application.