Stream tar archives in PHP with bounded buffering

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands · Show bio

Use proc_open() to read GNU tar’s output a chunk at a time and send it as an HTTP response. PHP does not need to hold the archive in memory or write a temporary archive. There is a tradeoff: if tar fails after the response starts, the client can receive HTTP 200 and a readable archive that is missing files. A successful download is not proof that the archive is complete.

This walkthrough creates a local download endpoint for a fixed directory of finished build logs, then checks every expected member and its bytes. It was tested on Linux with PHP 8.5.10’s local server and with PHP-FPM 8.4.26 behind Nginx 1.28.3, using GNU tar 1.35. You also need Bash, cURL, and cmp, with proc_open() enabled in PHP. Windows and BSD tar are outside this walkthrough’s scope.

Keep the archive out of PHP’s buffers

PHP’s memory_limit still applies. The benefit here is that the transfer loop retains only an 8 KiB chunk rather than an archive-sized string. The separate tar process, PHP runtime, web server, and operating system also consume memory. This is bounded application buffering, not a way to remove every memory limit.

PharData⁠ provides an API for manipulating on-disk tar and ZIP archives. Using it does not inherently mean loading every file’s contents into one string. For this write-once download, an external tar process gives us a straightforward pipe and a creation exit status.

Save this as setup.sh in a working directory, then run bash setup.sh. It creates a new php-tar-demo directory and refuses to reuse an existing one. The input contains a text file, an empty file, and binary bytes. Subsequent files belong inside php-tar-demo.

#!/usr/bin/env bash
set -euo pipefail
mkdir php-tar-demo
cd php-tar-demo
mkdir public logs logs/build-123
printf 'Build 123 passed\n' > logs/build-123/build.log
: > logs/build-123/empty.log
printf '\000\001\177\200\377\n' > logs/build-123/payload.bin

Stream tar output with proc_open()

Save this as tar-streaming.php. The array form of proc_open()⁠ launches the program without a shell. Absolute directory paths and -- keep filenames from becoming command options. Use a trusted server PATH and keep TAR_OPTIONS unset.

<?php
declare(strict_types=1);

/** Both paths must already have been resolved with realpath(). */
function isWithin(string $path, string $root): bool
{
    return $path === $root || str_starts_with($path, rtrim($root, DIRECTORY_SEPARATOR) . DIRECTORY_SEPARATOR);
}

function safeAttachmentName(string $downloadName): string
{
    $name = trim((string) preg_replace('/[^A-Za-z0-9._-]+/', '_', basename($downloadName)), '._');
    return $name === '' ? 'archive.tar' : substr($name, 0, 100);
}

function clearOutputBuffers(): void
{
    while (ob_get_level() > 0) {
        $status = ob_get_status();
        if (($status['flags'] & PHP_OUTPUT_HANDLER_REMOVABLE) === 0 || !ob_end_clean()) {
            throw new RuntimeException('Cannot disable output buffering for this response.');
        }
    }
}

function streamTarArchive(string $directory, string $downloadName = 'archive.tar'): void
{
    $realDir = realpath($directory);
    if ($realDir === false || !is_dir($realDir)) {
        throw new InvalidArgumentException('Directory does not exist or is not accessible.');
    }
    clearOutputBuffers();

    // Discard stderr rather than risk a full, unread pipe blocking tar.
    // The exit status still tells the caller whether creation succeeded.
    $descriptors = [
        0 => ['file', '/dev/null', 'r'],
        1 => ['pipe', 'w'],
        2 => ['file', '/dev/null', 'w'],
    ];
    $process = proc_open(['tar', '-C', $realDir, '-cf', '-', '--', '.'], $descriptors, $pipes);
    if (!is_resource($process)) {
        throw new RuntimeException('Could not start tar.');
    }

    // Let our cleanup run when PHP notices a disconnected client.
    $previousIgnoreAbort = ignore_user_abort(true);
    $reachedEnd = false;
    try {
        header('Content-Type: application/x-tar');
        header('Content-Disposition: attachment; filename="' . safeAttachmentName($downloadName) . '"');
        header('X-Content-Type-Options: nosniff');
        header('Cache-Control: private, no-store');
        header('X-Accel-Buffering: no');

        while (true) {
            $chunk = fread($pipes[1], 8192);
            if ($chunk === false) {
                throw new RuntimeException('Failed to read the tar output stream.');
            }
            if ($chunk === '') {
                break;
            }
            echo $chunk;
            flush();
            if (connection_aborted()) {
                throw new RuntimeException('Client disconnected.');
            }
        }
        $reachedEnd = true;
    } finally {
        fclose($pipes[1]);
        if (!$reachedEnd) {
            proc_terminate($process);
        }
        $exitCode = proc_close($process);
        ignore_user_abort($previousIgnoreAbort !== 0);
    }

    if ($exitCode !== 0) {
        throw new RuntimeException("tar exited with status {$exitCode}");
    }
}

class SecureTarStreamer
{
    private array $allowedRoots;

    public function __construct(array $allowedRoots)
    {
        $this->allowedRoots = array_map(function (string $root): string {
            $resolved = realpath($root);
            if ($resolved === false || !is_dir($resolved)) {
                throw new InvalidArgumentException('Invalid allowed root directory.');
            }
            return $resolved;
        }, $allowedRoots);
    }

    public function send(string $directory, string $downloadName = 'archive.tar'): void
    {
        $path = realpath($directory);
        if ($path === false || !is_dir($path)) {
            throw new InvalidArgumentException('Directory does not exist or is not accessible.');
        }
        foreach ($this->allowedRoots as $root) {
            if (isWithin($path, $root)) {
                streamTarArchive($path, $downloadName);
                return;
            }
        }
        throw new RuntimeException('Access to the specified directory is not allowed.');
    }
}

The pipe is blocking, so an empty read means EOF; false means a read failure. After closing the pipe, proc_close()⁠ waits for the producer and returns its exit status. Neither EOF nor the last successful write tells us that tar succeeded. Discarding stderr costs detailed diagnostics; if you need them, drain stdout and stderr concurrently and retain only a bounded amount of diagnostic text.

Serve a directory of finished build logs

Save this as public/download.php. The opaque log_id selects one configured directory; the request never supplies a filesystem path. Only public/ is served, keeping the source logs and helper outside the document root.

<?php
declare(strict_types=1);
require __DIR__ . '/../tar-streaming.php';

try {
    $logId = $_GET['log_id'] ?? '';
    $directories = ['build-123' => __DIR__ . '/../logs/build-123'];
    if (!is_string($logId) || !isset($directories[$logId])) {
        throw new InvalidArgumentException('Unknown log identifier.');
    }
    $streamer = new SecureTarStreamer([__DIR__ . '/../logs']);
    $streamer->send($directories[$logId], $logId . '.tar');
    error_log('Archive generation completed.');
} catch (Throwable $error) {
    error_log('Archive generation failed: ' . $error->getMessage());
    if (!headers_sent()) {
        header_remove('Content-Disposition');
        http_response_code($error instanceof InvalidArgumentException ? 400 : 500);
        header('Content-Type: text/plain; charset=utf-8');
        header('Cache-Control: private, no-store');
        echo "The archive could not be generated.\n";
    }
    // After headers, adding an error message would contaminate the archive bytes.
}

Harden proc_open() calls

This local example has no authentication. Before mounting it in an application, require both login and authorization for the selected build. Keep archived directories under application control and stop their writers first. Resolving the root blocks a sibling such as logs-evil, but it does not freeze the tree against another process changing it. The command does not follow symlink entries; a linked file’s target contents are not included. Use ordinary files and directories for this example, not sockets or special devices.

Save this as serve.sh and run bash serve.sh from php-tar-demo. It runs in the foreground; stop it with Ctrl+C. If port 8080 is busy, use PORT=8090 bash serve.sh and set the same PORT when running the client script below. A bind failure exits with an error.

#!/usr/bin/env bash
set -euo pipefail
if [ ! -f public/download.php ]; then
  printf 'Run serve.sh from the project directory after saving public/download.php.\n' >&2
  exit 1
fi
exec env -u TAR_OPTIONS php \
  -d memory_limit=32M -d output_buffering=0 -d zlib.output_compression=0 \
  -d display_errors=0 -d log_errors=1 \
  -S "127.0.0.1:${PORT:-8080}" -t public

PHP’s built-in server⁠ is for local development. For an existing PHP-FPM deployment, also disable PHP output buffering and compression, keep display_errors off, and configure request and upstream timeouts. PHP’s flush()⁠ cannot override every downstream buffer. With Nginx, use fastcgi_buffering off and fastcgi_ignore_client_abort off for this route. Nginx also recognizes the helper’s X-Accel-Buffering: no⁠ header unless configured to ignore it. A blocked filesystem read or output write can still delay PHP’s cleanup; do not treat this loop as a wall-clock deadline or a durable background job.

Check what the client actually received

Save this as verify.sh. In a second terminal inside php-tar-demo, run bash verify.sh. It downloads, checks the exact member list, extracts the local fixture, and compares all three files with their originals. It creates downloaded/ once and refuses to overwrite it on a rerun. If a step fails, that directory remains for inspection, possibly with a partial archive; use a new empty project for another run.

#!/usr/bin/env bash
set -euo pipefail
mkdir downloaded
curl -fsS --max-time 30 \
  "http://127.0.0.1:${PORT:-8080}/download.php?log_id=build-123" \
  --output downloaded/build.tar
tar -tf downloaded/build.tar | LC_ALL=C sort > downloaded/members.txt
printf './\n./build.log\n./empty.log\n./payload.bin\n' | cmp - downloaded/members.txt
tar -xf downloaded/build.tar -C downloaded
cmp logs/build-123/build.log downloaded/build.log
cmp logs/build-123/empty.log downloaded/empty.log
cmp logs/build-123/payload.bin downloaded/payload.bin
printf 'All expected members and bytes match.\n'

The final line appears only after the comparisons pass. Listing the archive alone is a weaker check. GNU tar can finish writing an archive while reporting a read error, leaving out the unreadable member. Its documented exit statuses⁠ separate successful creation from changed inputs and failures, but that producer status is not encoded in the HTTP response body.

For example, if the PHP worker cannot read payload.bin, the route can log a tar failure while cURL exits successfully and tar -tf lists the remaining members successfully. The member-list comparison above catches the omission because it has an independent expectation. With a remote export, the recipient needs an independently supplied inventory and content hashes to perform that kind of check; deriving an inventory from the received archive proves nothing about missing source files.

What about interrupted downloads?

An interrupted connection can leave a partial tar file. A late producer failure can instead leave a parseable but incomplete archive. Once PHP has sent headers, headers_sent()⁠ prevents the handler from replacing HTTP 200 with an error status. The handler logs the failure and adds no text to the archive. It cannot promise that cURL or a browser will report a failed download.

The helper checks connection_aborted()⁠ after writing, then closes the pipe and requests that tar terminate. Detection depends on PHP and the web server noticing the disconnect. Even a server-side completion log means only that generation finished without a detected transfer error, not that the client saved the file.

This endpoint does not implement HTTP range requests. Retrying starts a fresh archive, which may have different bytes if the source changed. Resuming requires serving the same stable archive representation; tar’s sequential layout does not itself prohibit HTTP byte ranges.

Report progress from the same job

A second tar -v process measures a different run, even if it reads the same directory. Do not use that as the download’s progress or completion signal. A separate SSE endpoint needs a job identifier and shared state written by the actual producer. Keep a job’s creation status separate from the client’s transfer status. This synchronous example has neither a progress store nor a completion receipt for the client.

Compare approaches

Direct streaming fits a download where avoiding temporary archive storage matters and callers understand the late-failure limitation. For a backup or export that must report creation failure before download begins, generate into a private temporary file, check the producer’s exit status, and publish only a successful result. That still allows bounded PHP memory, but requires disk space for the archive.

Serve the finished, immutable file with its known length and a checksum when recipients need to verify transfer integrity or resume it. Checking the producer first fixes the timing of creation errors; it does not turn changing source files into a consistent snapshot. Freeze the inputs or use a filesystem snapshot when that consistency is part of the export’s promise.