Export files to Amazon S3 in Lua with LuaSocket

Tim Koschützki

Co-founder · Berlin, Germany ·

To export a file to Amazon S3 from Lua, let libcurl sign the upload and verify HTTPS, then download the object to check its bytes. This example calls libcurl through LuaJIT’s FFI and uses LuaSocket for retry delays. It handles binary and empty files, temporary AWS credentials, and keys containing spaces, Unicode, and literal percent signs.

Installation and setup

The tested setup is Linux with LuaJIT 2.1.1788460057, LuaRocks 3.13.0, LuaSocket 3.1.0-1, libcurl 8.22.0, and AWS CLI v2 2.37.10. These are tested versions, not intrinsic minimums. Use a current LuaJIT 2.1 build and a patched libcurl release with HTTPS and AWS SigV4 enabled. Older libcurl options alone do not establish compatibility with its S3 signer.

Have LuaJIT and its development headers, LuaRocks, a C compiler, Make, pkg-config, libcurl, a system CA bundle, AWS CLI v2, cmp, and mktemp installed through your trusted system package manager. The LuaJIT development package must provide the luajit pkg-config entry. Work in an empty directory outside an existing Lua project, and install the LuaSocket rock into its own tree:

(
  set -eu
  for tool in luajit luarocks cc make pkg-config aws cmp mktemp; do
    command -v "$tool" >/dev/null || { printf 'Missing prerequisite: %s\n' "$tool" >&2; exit 1; }
  done
  case "$(aws --version)" in
    aws-cli/2.*) ;;
    *) printf 'AWS CLI v2 is required\n' >&2; exit 1 ;;
  esac
  lua_headers=$(pkg-config --variable=includedir luajit)
  test -f "$lua_headers/lua.h"
  luarocks --lua-version=5.1 --tree="$PWD/rocks" install luasocket 3.1.0-1 \
    LUA_INCDIR="$lua_headers" LUA="$(command -v luajit)"
)

If installation fails, fix the reported prerequisite and repeat this block in the same directory; it does not overwrite your Lua source files. The runner below loads LuaSocket from ./rocks instead of relying on an enclosing project’s Lua search paths. Set LIBCURL_PATH to a trusted absolute shared-library path only when libcurl is outside the loader’s normal search path. OpenResty and LuaSec are not required. The module makes a direct HTTPS connection and disables proxies and redirects; your network must allow direct access to S3.

AWS credential setup

Use an existing general purpose bucket and credentials authorized for s3:PutObject and s3:GetObject on your chosen prefix. The runner uses AWS CLI credential resolution to pass credentials to Lua in process memory. Sign in through your organization’s existing AWS CLI workflow first, or supply credentials through your deployment’s secret manager. Temporary credentials need AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN; long-lived credentials omit the token. Do not paste credentials into the program or enable shell tracing.

Set AWS_REGION and S3_BUCKET to the actual bucket region and name. The default endpoint below supports conventional AWS commercial-region buckets whose names contain lowercase letters, digits, and hyphens. Other partitions, dotted names, access points, and directory buckets need their own verified endpoint configuration. This example does not configure IAM or create a bucket.

Using a verified signing transport

libcurl’s CURLOPT_AWS_SIGV4 signs the request. The session token is sent as x-amz-security-token and included in the signed headers. Keys are encoded as UTF-8 bytes while retaining slash separators. The module rejects standalone . and .. path components to avoid URL normalization changing the object key. The signer hashes the exact POSTFIELDS bytes even though CUSTOMREQUEST selects the HTTP method PUT. This avoids maintaining a separate cryptographic implementation in the tutorial.

Uploading from LuaJIT

Save this module as s3_upload.lua. It deliberately limits each object to 16 MiB in memory; larger uploads need a separately verified streaming or multipart implementation. Each attempt has a 10-second connection limit and a 60-second whole-transfer limit, so even a smaller file can time out on a slow connection. endpoint and cafile are optional trusted deployment settings, useful for testing against an HTTPS fixture. Never accept them from an unauthenticated request.

local ffi = require("ffi")
local socket = require("socket")
ffi.cdef[[
typedef void CURL;
struct curl_slist { char *data; struct curl_slist *next; };
int curl_global_init(long flags);
CURL *curl_easy_init(void);
void curl_easy_cleanup(CURL *handle);
int curl_easy_setopt(CURL *handle, int option, ...);
int curl_easy_perform(CURL *handle);
int curl_easy_getinfo(CURL *handle, int info, ...);
struct curl_slist *curl_slist_append(struct curl_slist *list, const char *value);
void curl_slist_free_all(struct curl_slist *list);
]]
local curl = ffi.load(os.getenv("LIBCURL_PATH") or "curl")
assert(curl.curl_global_init(3) == 0, "Cannot initialize libcurl")
-- Stable option identifiers from libcurl's public curl.h ABI.
local O = { URL=10002, PROXY=10004, WRITEFUNCTION=20011, POSTFIELDS=10015,
  HTTPHEADER=10023, CUSTOMREQUEST=10036, SSL_VERIFYPEER=64, CAINFO=10065,
  POSTFIELDSIZE=60, SSL_VERIFYHOST=81, CONNECTTIMEOUT=78, TIMEOUT=13,
  USERNAME=10173, PASSWORD=10174, AWS_SIGV4=10305,
  PROTOCOLS_STR=10318, FOLLOWLOCATION=52 }
local M = {}

function M.encode_key(key)
  assert(type(key) == "string" and #key > 0 and #key <= 1024, "Invalid object key")
  for part in key:gmatch("[^/]+") do
    assert(part ~= "." and part ~= "..", "Dot path components are unsupported")
  end
  return (key:gsub("([^A-Za-z0-9/_.~-])", function(byte)
    return string.format("%%%02X", byte:byte())
  end))
end

local function header(value)
  assert(type(value) == "string" and #value > 0 and not value:find("[%c]"), "Invalid header value")
  return value
end

local function request(config, key, body, content_type)
  local handle = curl.curl_easy_init()
  assert(handle ~= nil, "Cannot allocate curl handle")
  local headers = nil
  local function append_header(value)
    local updated = curl.curl_slist_append(headers, value)
    assert(updated ~= nil, "Cannot allocate curl header")
    headers = updated
  end
  local received = 0
  local writer = ffi.cast("size_t (*)(char *, size_t, size_t, void *)", function(_, size, count)
    local length = tonumber(size * count)
    received = received + length
    if received > 1024 * 1024 then return 0 end
    return length
  end)
  local function option(name, value)
    if type(value) == "number" then value = ffi.new("long", value) end
    assert(curl.curl_easy_setopt(handle, O[name], value) == 0, "Unsupported curl option: " .. name)
  end
  local ok, result = pcall(function()
    local endpoint = config.endpoint or ("https://" .. config.bucket .. ".s3." .. config.region .. ".amazonaws.com")
    assert(endpoint:match("^https://[A-Za-z0-9.-]+:?%d*$"), "Expected a trusted HTTPS origin")
    option("URL", endpoint .. "/" .. M.encode_key(key))
    option("PROTOCOLS_STR", "https")
    option("PROXY", "")
    option("FOLLOWLOCATION", 0)
    option("SSL_VERIFYPEER", 1)
    option("SSL_VERIFYHOST", 2)
    if config.cafile then option("CAINFO", config.cafile) end
    option("CONNECTTIMEOUT", 10)
    option("TIMEOUT", 60)
    option("USERNAME", header(config.access_key))
    option("PASSWORD", header(config.secret_key))
    option("AWS_SIGV4", "aws:amz:" .. config.region .. ":s3")
    option("CUSTOMREQUEST", "PUT")
    option("POSTFIELDSIZE", #body)
    option("POSTFIELDS", body)
    option("WRITEFUNCTION", writer)
    for _, value in ipairs({"Content-Type: " .. header(content_type), "Expect:"}) do
      append_header(value)
    end
    if config.session_token then
      append_header("x-amz-security-token: " .. header(config.session_token))
    end
    option("HTTPHEADER", headers)
    local code = curl.curl_easy_perform(handle)
    local status = ffi.new("long[1]")
    assert(curl.curl_easy_getinfo(handle, 0x200002, status) == 0, "Cannot read HTTP status")
    return { transport=tonumber(code), status=tonumber(status[0]) }
  end)
  curl.curl_easy_cleanup(handle)
  curl.curl_slist_free_all(headers)
  writer:free()
  if not ok then error(result) end
  return result
end
-- libcurl calls a Lua callback; this call path must stay outside compiled FFI traces.
jit.off(request, true)

function M.retryable(result)
  if result.transport ~= 0 then
    return ({[6]=true, [7]=true, [18]=true, [28]=true, [55]=true, [56]=true})[result.transport] == true
  end
  return ({[408]=true, [429]=true, [500]=true, [502]=true, [503]=true, [504]=true})[result.status] == true
end

function M.upload(config, file_path, key, content_type)
  assert(config.region and config.region:match("^[a-z0-9-]+$"), "Invalid region")
  assert(config.bucket and #config.bucket >= 3 and #config.bucket <= 63
    and config.bucket:match("^[a-z0-9][a-z0-9-]*[a-z0-9]$"), "Invalid bucket name")
  local file = assert(io.open(file_path, "rb"))
  local body, read_error = file:read(16 * 1024 * 1024 + 1)
  file:close()
  if read_error then error(read_error) end
  body = body or ""
  assert(#body <= 16 * 1024 * 1024, "Use multipart upload for larger files")
  for attempt = 1, 4 do
    local result = request(config, key, body, content_type)
    if result.transport == 0 and result.status == 200 then return true end
    if attempt == 4 or not M.retryable(result) then
      return nil, "Upload failed (transport " .. result.transport .. ", HTTP " .. result.status .. ")"
    end
    socket.sleep(math.min(8, 0.25 * 2^(attempt - 1)) * (0.5 + math.random()))
  end
end
return M

Save this caller as upload.lua:

local s3 = require("s3_upload")
local ok, err = s3.upload({
  access_key = os.getenv("AWS_ACCESS_KEY_ID"),
  secret_key = os.getenv("AWS_SECRET_ACCESS_KEY"),
  session_token = os.getenv("AWS_SESSION_TOKEN"),
  region = os.getenv("AWS_REGION"),
  bucket = os.getenv("S3_BUCKET")
}, assert(arg[1], "local file required"), assert(arg[2], "object key required"), "application/octet-stream")
assert(ok, err)
print("Uploaded")

Save the following as upload.sh. Run it from the directory containing both Lua files. Its credential export comes from the trusted AWS CLI executable; the script clears a stale session token before loading the resolved credentials. Its settings apply only to the child process.

#!/usr/bin/env bash
set +x
set -eu
for tool in aws luajit; do
  command -v "$tool" >/dev/null || { printf 'Missing prerequisite: %s\n' "$tool" >&2; exit 1; }
done
: "${AWS_REGION:?Set AWS_REGION to the bucket region}"
: "${S3_BUCKET:?Set S3_BUCKET to the bucket name}"
credentials=$(aws configure export-credentials --format env)
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
eval "$credentials"
export LUA_PATH='./?.lua;./rocks/share/lua/5.1/?.lua;./rocks/share/lua/5.1/?/init.lua'
export LUA_CPATH='./rocks/lib/lua/5.1/?.so'
exec luajit upload.lua "$@"

Use an existing regular file up to 16 MiB, or create a tiny binary example. This block refuses to replace an existing report.bin:

(
  set -eu
  test ! -e report.bin
  printf 'September report\000\377\n' > report.bin
)

Replace your-bucket-name with your bucket. Set AWS_PROFILE in the environment if your AWS CLI login uses a named profile. Keep the object key quoted, and pass its original spelling rather than pre-encoding it:

AWS_REGION=us-east-1 S3_BUCKET=your-bucket-name \
bash upload.sh report.bin 'reports/September café %.bin'

The caller prints Uploaded only after S3 returns HTTP 200. Uploading to an existing key replaces its current object; a versioned bucket retains another version. Use a fresh key for your first test.

Check the remote bytes

Save this independent download check as verify.sh. It uses the same AWS CLI credential source, downloads the exact key, compares every byte with the local file, and removes its temporary copy on success or failure:

#!/usr/bin/env bash
set -eu
: "${AWS_REGION:?Set AWS_REGION to the bucket region}"
: "${S3_BUCKET:?Set S3_BUCKET to the bucket name}"
test "$#" -eq 2
test -f "$1"
download=$(mktemp)
trap 'rm -f "$download"' EXIT
aws s3api get-object --bucket "$S3_BUCKET" --key "$2" --region "$AWS_REGION" \
  "$download" >/dev/null
cmp -- "$1" "$download"
printf 'Verified identical bytes\n'

AWS_REGION=us-east-1 S3_BUCKET=your-bucket-name \
bash verify.sh report.bin 'reports/September café %.bin'

Expect Verified identical bytes. A missing object, a denied download, or different bytes produces a nonzero exit instead. Keep the source file unchanged between upload and verification. A successful PutObject stores the complete request body, but this comparison also checks that you supplied the intended file and key.

Error handling and retries

Only HTTP 200 confirms this PutObject operation. Redirects, authentication failures, and other permanent HTTP errors stop immediately. Selected transient transport errors, throttling, and service errors get at most four attempts, each with the time limits above. Certificate validation failures are not retried; repair the CA bundle or endpoint rather than disabling validation. A missing input or an oversized file fails before any upload. The delay uses socket.sleep() with fractional seconds; it never constructs a shell command or formats a fraction with an integer placeholder.

Each retry signs a fresh request over the same bytes and key. A lost response can still create multiple versions in a versioned bucket, or repeat notifications. Reconcile ambiguous outcomes where those effects matter. This bounded example does not implement adaptive throttling, Retry-After handling, automatic credential refresh, or exactly-once delivery.

Content type handling

The caller uses application/octet-stream. To store a PDF’s content type, replace that string in upload.lua with application/pdf; the file and key arguments stay the same. This sets object metadata and does not validate the file’s contents. The module rejects control characters in header values and never returns the service response body as an error.

Conclusion

Run the upload and byte comparison against a fresh key in your authorized test bucket before integrating the module. For files above this example’s memory limit, choose a streaming or multipart client and verify its completion behavior separately; increasing the limit would still buffer the whole file in Lua.