Create CI log archives with Node.js and tar-stream

Tim Koschützki

Co-founder · Berlin, Germany ·

Use tar-stream to write completed CI logs into a tar archive as they arrive. The example below streams the archive to a temporary file, then publishes ci-logs.tar only after every entry has finished. An existing destination is left untouched, including when a log producer fails.

This is a local Node.js workflow for logs supplied as strings or Buffers. Each log fits in memory; the combined archive does not have to. It produces an uncompressed .tar file on disk, not an in-memory archive or a live stream of an unfinished log.

Dependencies

Use Node.js 24.15.0 or a later 24.x release, Corepack with Yarn 4.12.0 available, Bash, and GNU tar for inspection. The walkthrough was tested on Linux with Node.js 24.15.0 and GNU tar 1.35. Use a local filesystem that supports hard links, and a destination directory you control. Network filesystems and Windows are outside this walkthrough’s tested scope.

Paste this into Bash from a parent directory where you want the example project. The subshell keeps your current directory unchanged, and && stops setup if a step fails. If node-tar-demo already exists, choose another parent directory instead of deleting an existing project.

(
  mkdir node-tar-demo &&
  cd node-tar-demo &&
  printf '%s\n' '{"private":true,"type":"module","packageManager":"yarn@4.12.0"}' > package.json &&
  printf '%s\n' 'nodeLinker: node-modules' > .yarnrc.yml &&
  touch yarn.lock &&
  corepack yarn add --exact tar-stream@3.1.7
)

The project explicitly enables ES modules and installs ordinary node_modules for the node command. Its own lockfile keeps Yarn from treating it as part of an enclosing project. Node runs these TypeScript files using native type stripping; this executes the code without type-checking it.

Stream entries and publish the finished archive

Save this complete module as node-tar-demo/archive.ts. A producer receives an AbortSignal and yields one completed log at a time. Start consuming the pack stream before adding entries, and await each entry’s callback before requesting another log. These are the packing operations described in the tar-stream documentation.

import type { Writable } from 'node:stream'

import { createWriteStream } from 'node:fs'
import { link, mkdtemp, rm } from 'node:fs/promises'
import path from 'node:path'
import { pipeline } from 'node:stream/promises'
import tar from 'tar-stream'

export interface LogEntry {
  filename: string
  content: string | Buffer
}

type LogProducer = (signal: AbortSignal) => Iterable<LogEntry> | AsyncIterable<LogEntry>
interface ArchiveOptions {
  signal?: AbortSignal
}

function validateFilename(filename: string): string {
  if (
    !filename || filename.includes('\0') ||
    path.posix.isAbsolute(filename) || path.win32.isAbsolute(filename) ||
    filename.includes(':') || filename.split(/[\\/]/).some((part) => !part || part === '.' || part === '..')
  ) {
    throw new Error('Invalid archive filename')
  }
  return filename.replaceAll('\\', '/')
}

async function addLogToArchive(
  pack: ReturnType<typeof tar.pack>, filename: string, content: string | Buffer,
): Promise<void> {
  return new Promise((resolve, reject) => {
    pack.entry({ name: validateFilename(filename), mode: 0o600 }, content, (error) => {
      if (error) reject(error)
      else resolve()
    }).on('error', reject)
  })
}

export async function createLogArchive(
  getLogs: LogProducer, output: Writable, options: ArchiveOptions = {},
): Promise<void> {
  const controller = new AbortController()
  const signal = options.signal
    ? AbortSignal.any([options.signal, controller.signal])
    : controller.signal
  const pack = tar.pack()
  const completed = pipeline(pack, output, { signal })
  const producing = (async () => {
    signal.throwIfAborted()
    for await (const log of getLogs(signal)) {
      signal.throwIfAborted()
      await addLogToArchive(pack, log.filename, log.content)
    }
    signal.throwIfAborted()
    pack.finalize()
  })()

  try {
    await Promise.all([completed, producing])
  } catch (error) {
    // Stop both sides, then wait for the producer and file handle to finish cleanup.
    controller.abort(error)
    await Promise.allSettled([completed, producing])
    throw error
  }
}

export async function saveLogArchive(
  getLogs: LogProducer, destination: string, options: ArchiveOptions = {},
): Promise<void> {
  options.signal?.throwIfAborted()
  const target = path.resolve(destination)
  const staging = await mkdtemp(path.join(path.dirname(target), '.ci-logs-'))
  const temporary = path.join(staging, 'archive.tar')
  try {
    await createLogArchive(
      getLogs,
      createWriteStream(temporary, { flags: 'wx', mode: 0o600 }),
      options,
    )
    options.signal?.throwIfAborted()
    await link(temporary, target)
  } finally {
    await rm(staging, { recursive: true, force: true })
  }
}

The stream helper coordinates production and writing. Node’s pipeline() handles backpressure and destroys the connected streams on cancellation. If either side fails, the helper also cancels the producer and waits for both tasks to settle. Your producer must pass the signal into any operation that can wait, as the demo’s timer does below. Cancellation cannot stop an arbitrary promise that ignores the signal.

The saving helper uses a hard link to give the completed file its final name. Linux link(2) refuses an existing destination, including a symlink. There is no separate existence check followed by an overwriting rename. The temporary directory is beside the destination so both names are on the same filesystem. Removing the temporary name after publication leaves the final archive intact.

Run and inspect the example

Save this as node-tar-demo/demo.ts. The timer stands in for waiting for another completed CI step; replace ciLogs with your own producer when integrating it. The demo includes an empty entry and a small binary artifact so you can inspect more than text.

import { setTimeout as delay } from 'node:timers/promises'
import type { LogEntry } from './archive.ts'
import { saveLogArchive } from './archive.ts'

async function* ciLogs(signal: AbortSignal): AsyncGenerator<LogEntry> {
  yield { filename: 'build.log', content: 'Build completed successfully.\n' }
  await delay(25, undefined, { signal })
  yield { filename: 'test.log', content: 'All tests passed.\n' }
  yield { filename: 'empty.log', content: Buffer.alloc(0) }
  yield { filename: 'artifacts/status.bin', content: Buffer.from([0, 255, 128, 10]) }
}

async function main(): Promise<void> {
  const controller = new AbortController()
  const cancel = () => controller.abort(new Error('Interrupted'))
  process.once('SIGINT', cancel)
  try {
    const destination = process.argv[2] ?? 'ci-logs.tar'
    await saveLogArchive(ciLogs, destination, {
      signal: AbortSignal.any([controller.signal, AbortSignal.timeout(30_000)]),
    })
    console.log(`Created ${destination}`)
  } finally {
    process.removeListener('SIGINT', cancel)
  }
}

main().catch((error: unknown) => {
  console.error(error instanceof Error ? error.message : 'Archive failed')
  process.exitCode = 1
})

From the parent directory used for setup, paste:

(
  cd node-tar-demo &&
  node demo.ts &&
  tar -tf ci-logs.tar &&
  tar -xOf ci-logs.tar test.log
)

Expected output:

Created ci-logs.tar
build.log
test.log
empty.log
artifacts/status.bin
All tests passed.

Run the same block again to check the collision policy: Node exits with status 1 and an EEXIST message, the inspection commands do not run, and the existing archive retains its bytes. To create another archive, give demo.ts a different destination argument, such as ci-logs-next.tar.

Performance considerations

Awaiting each entry prevents the producer from enqueueing the whole archive while a slow output is still draining. It does not make an individual string or Buffer smaller, and a producer that has already collected every log still retains that memory. Supply completed logs incrementally, with a size bound appropriate for your CI jobs.

Tar groups files; this example does not compress them. It writes the full tar output once to disk. The hard-link publication step adds a filename without copying those bytes. There is no benchmark here establishing a speed improvement over a command-line archiver.

Security considerations

Entry names must be relative file paths. The validator rejects absolute paths, drive/alternate-stream separators, null bytes, empty segments, and . or .. segments before normalizing backslashes. Use distinct names for your logs: this helper does not reject duplicate entries. It creates ordinary file entries with mode 0600, without guessing executable permissions from a filename extension.

Keep secrets out of CI logs before archiving them. Tar provides no encryption, and these name checks do not make extraction of arbitrary third-party archives safe. Choose extraction destinations and symlink policies separately if you later build an extractor.

Handling different file types

Pass UTF-8 text as a string and binary data as a Buffer. tar-stream derives the entry size from those bytes, including an empty Buffer; no text decoding is needed for status.bin. For a file stream, tar needs its byte size before the entry body. This example intentionally accepts completed logs instead of claiming to archive a log whose final size is still unknown.

Troubleshooting common issues

EEXIST: The final destination already exists. Refusal happens at publication, after the logs have been produced. Choose another filename; the script never replaces an existing one.
Producer rejection, invalid entry, or write failure: Both tasks settle before temporary-file cleanup. No new final archive is published. When using createLogArchive directly with another Writable, you own disposal of any partial bytes in that destination.
Timeout or Ctrl+C: The demo requests cancellation and exits unsuccessfully after cleanup. Make external producers observe the supplied signal. Once the final link operation has started, cancellation may arrive too late to prevent publication.
Missing directory or hard-link error: Create the destination’s parent first and check that the local filesystem permits hard links. Do not replace the link with an overwriting rename to suppress the error.
Cleanup failure or abrupt termination: A cleanup error can occur after publication; inspect the final archive before retrying. A crash or SIGKILL can leave a .ci-logs-* directory. Remove that attempt’s temporary directory only after its process has stopped. This example does not promise durability across power loss.

Connect your CI log producer

Keep saveLogArchive as the file-output boundary and replace the demo generator with your CI step results. Yield each completed log, pass cancellation through to waits or requests, and choose an archive filename unique to the job. The same streaming helper can write to another Writable when you supply that destination’s own publication and cleanup policy.