Export files to Amazon S3 in Lua with LuaSocket

Tim Koschützki

Co-founder · Berlin, Germany ·

LuaSocket provides networking primitives and fractional sleeps, but it does not implement AWS authentication or HTTPS certificate validation by itself. This example uses LuaJIT’s FFI to call libcurl’s maintained SigV4 transport. LuaSocket supplies the retry delay. It replaces the previous untested lua-resty-aws constructor and hand-written signature implementation.

Installation and setup

Use LuaJIT 2.1, LuaSocket 3.1.0-1, and a supported libcurl build with HTTPS and AWS SigV4. The example was tested against libcurl 8.21.0. It uses options available since libcurl 7.85.0; use a currently patched version, not that historical minimum. Install libcurl and LuaJIT through your system package manager, then install LuaSocket for the same LuaJIT interpreter:

luarocks --lua-version=5.1 install luasocket 3.1.0-1

LuaRocks must be configured for LuaJIT’s headers and interpreter. Set LIBCURL_PATH to the trusted absolute path of your libcurl shared library if it is not on the dynamic loader’s normal path. This example targets macOS and Linux. It does not require OpenResty or LuaSec.

AWS credential setup

Use credentials authorized to put objects in the intended bucket and prefix. Prefer temporary credentials and supply all three variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. Long-lived credentials omit the session token. Obtain these through your deployment’s secret management; do not embed them in source code.

Set AWS_REGION and S3_BUCKET to the actual bucket region and name. The default endpoint below supports conventional AWS commercial-region buckets whose names contain lowercase letters, digits, and hyphens. Other partitions, dotted names, access points, and directory buckets need their own verified endpoint configuration. IAM permissions and actual bucket behavior require live validation.

Using a verified signing transport

libcurl’s CURLOPT_AWS_SIGV4 signs the request. The session token is sent as x-amz-security-token and included in the signed headers. Keys are encoded as UTF-8 bytes while retaining slash separators. The module rejects standalone . and .. path components because libcurl’s SigV4 mode does not support PATH_AS_IS; silently normalizing those components would change the object key. The signer hashes the exact POSTFIELDS bytes even though the HTTP method is PUT. This avoids maintaining a separate cryptographic implementation in the tutorial.

Uploading from LuaJIT

Save this module as s3_upload.lua. It deliberately limits each object to 16 MiB in memory; larger uploads need a separately verified streaming or multipart implementation. endpoint and cafile are optional trusted deployment settings, useful for testing against an HTTPS fixture. Never accept them from an unauthenticated request.

local ffi = require("ffi")
local socket = require("socket")
ffi.cdef[[
typedef void CURL;
struct curl_slist { char *data; struct curl_slist *next; };
int curl_global_init(long flags);
CURL *curl_easy_init(void);
void curl_easy_cleanup(CURL *handle);
int curl_easy_setopt(CURL *handle, int option, ...);
int curl_easy_perform(CURL *handle);
int curl_easy_getinfo(CURL *handle, int info, ...);
struct curl_slist *curl_slist_append(struct curl_slist *list, const char *value);
void curl_slist_free_all(struct curl_slist *list);
]]
local curl = ffi.load(os.getenv("LIBCURL_PATH") or "curl")
assert(curl.curl_global_init(3) == 0, "Cannot initialize libcurl")
-- Stable option identifiers from libcurl's public curl.h ABI.
local O = { URL=10002, PROXY=10004, WRITEFUNCTION=20011, POSTFIELDS=10015,
  HTTPHEADER=10023, CUSTOMREQUEST=10036, SSL_VERIFYPEER=64, CAINFO=10065,
  POSTFIELDSIZE=60, SSL_VERIFYHOST=81, CONNECTTIMEOUT=78, TIMEOUT=13,
  USERNAME=10173, PASSWORD=10174, AWS_SIGV4=10305,
  PROTOCOLS_STR=10318, FOLLOWLOCATION=52 }
local M = {}

function M.encode_key(key)
  assert(type(key) == "string" and #key > 0 and #key <= 1024, "Invalid object key")
  for part in key:gmatch("[^/]+") do
    assert(part ~= "." and part ~= "..", "Dot path components are unsupported")
  end
  return (key:gsub("([^A-Za-z0-9/_.~-])", function(byte)
    return string.format("%%%02X", byte:byte())
  end))
end

local function header(value)
  assert(type(value) == "string" and #value > 0 and not value:find("[%c]"), "Invalid header value")
  return value
end

local function request(config, key, body, content_type)
  local handle = curl.curl_easy_init()
  assert(handle ~= nil, "Cannot allocate curl handle")
  local headers = nil
  local function append_header(value)
    local updated = curl.curl_slist_append(headers, value)
    assert(updated ~= nil, "Cannot allocate curl header")
    headers = updated
  end
  local received = 0
  local writer = ffi.cast("size_t (*)(char *, size_t, size_t, void *)", function(_, size, count)
    local length = tonumber(size * count)
    received = received + length
    if received > 1024 * 1024 then return 0 end
    return length
  end)
  local function option(name, value)
    if type(value) == "number" then value = ffi.new("long", value) end
    assert(curl.curl_easy_setopt(handle, O[name], value) == 0, "Unsupported curl option: " .. name)
  end
  local ok, result = pcall(function()
    local endpoint = config.endpoint or ("https://" .. config.bucket .. ".s3." .. config.region .. ".amazonaws.com")
    assert(endpoint:match("^https://[A-Za-z0-9.-]+:?%d*$"), "Expected a trusted HTTPS origin")
    option("URL", endpoint .. "/" .. M.encode_key(key))
    option("PROTOCOLS_STR", "https")
    option("PROXY", "")
    option("FOLLOWLOCATION", 0)
    option("SSL_VERIFYPEER", 1)
    option("SSL_VERIFYHOST", 2)
    if config.cafile then option("CAINFO", config.cafile) end
    option("CONNECTTIMEOUT", 10)
    option("TIMEOUT", 60)
    option("USERNAME", header(config.access_key))
    option("PASSWORD", header(config.secret_key))
    option("AWS_SIGV4", "aws:amz:" .. config.region .. ":s3")
    option("CUSTOMREQUEST", "PUT")
    option("POSTFIELDSIZE", #body)
    option("POSTFIELDS", body)
    option("WRITEFUNCTION", writer)
    for _, value in ipairs({"Content-Type: " .. header(content_type), "Expect:"}) do
      append_header(value)
    end
    if config.session_token then
      append_header("x-amz-security-token: " .. header(config.session_token))
    end
    option("HTTPHEADER", headers)
    local code = curl.curl_easy_perform(handle)
    local status = ffi.new("long[1]")
    assert(curl.curl_easy_getinfo(handle, 0x200002, status) == 0, "Cannot read HTTP status")
    return { transport=tonumber(code), status=tonumber(status[0]) }
  end)
  curl.curl_easy_cleanup(handle)
  curl.curl_slist_free_all(headers)
  writer:free()
  if not ok then error(result) end
  return result
end
-- libcurl calls a Lua callback; this call path must stay outside compiled FFI traces.
jit.off(request, true)

function M.retryable(result)
  if result.transport ~= 0 then
    return ({[6]=true, [7]=true, [18]=true, [28]=true, [55]=true, [56]=true})[result.transport] == true
  end
  return ({[408]=true, [429]=true, [500]=true, [502]=true, [503]=true, [504]=true})[result.status] == true
end

function M.upload(config, file_path, key, content_type)
  assert(config.region and config.region:match("^[a-z0-9-]+$"), "Invalid region")
  assert(config.bucket and #config.bucket >= 3 and #config.bucket <= 63
    and config.bucket:match("^[a-z0-9][a-z0-9-]*[a-z0-9]$"), "Invalid bucket name")
  local file = assert(io.open(file_path, "rb"))
  local body, read_error = file:read(16 * 1024 * 1024 + 1)
  file:close()
  if read_error then error(read_error) end
  body = body or ""
  assert(#body <= 16 * 1024 * 1024, "Use multipart upload for larger files")
  for attempt = 1, 4 do
    local result = request(config, key, body, content_type)
    if result.transport == 0 and result.status == 200 then return true end
    if attempt == 4 or not M.retryable(result) then
      return nil, "Upload failed (transport " .. result.transport .. ", HTTP " .. result.status .. ")"
    end
    socket.sleep(math.min(8, 0.25 * 2^(attempt - 1)) * (0.5 + math.random()))
  end
end
return M

Save this caller as upload.lua:

local s3 = require("s3_upload")
local ok, err = s3.upload({
  access_key = os.getenv("AWS_ACCESS_KEY_ID"),
  secret_key = os.getenv("AWS_SECRET_ACCESS_KEY"),
  session_token = os.getenv("AWS_SESSION_TOKEN"),
  region = os.getenv("AWS_REGION"),
  bucket = os.getenv("S3_BUCKET")
}, assert(arg[1], "local file required"), assert(arg[2], "object key required"), "application/octet-stream")
assert(ok, err)
print("Uploaded")

luajit upload.lua report.pdf 'reports/September report.pdf'

Error handling and retries

Only HTTP 200 confirms this PutObject operation. Redirects, authentication failures, and other permanent HTTP errors stop immediately. Selected transient transport errors, throttling, and service errors get at most four attempts. Certificate validation failures are not retried. The delay uses socket.sleep() with fractional seconds; it never constructs a shell command or formats a fraction with an integer placeholder.

Each retry signs a fresh request over the same bytes and key. A lost response can still create multiple versions in a versioned bucket, or repeat notifications. Reconcile ambiguous outcomes where those effects matter. This bounded example does not implement adaptive throttling, Retry-After handling, automatic credential refresh, or exactly-once delivery.

Content type handling

The caller uses application/octet-stream. Supply the known content type when appropriate, such as application/pdf; a filename extension alone does not validate file contents. The module rejects control characters in header values and never returns the service response body as an error.

Conclusion

Use a maintained signing transport and verify encoded keys, temporary credentials, TLS failures, and non-success responses against an isolated HTTPS endpoint. Local protocol checks cannot prove IAM authorization, bucket policies, or object availability on AWS; complete those checks separately with an authorized test bucket.