Streamline CI log archiving with Node.js

Tim Koschützki

Co-founder · Berlin, Germany ·

Continuous integration systems generate an enormous amount of log data, making efficient log archiving essential. Node.js streams and tar libraries offer a powerful solution for processing and archiving logs in real-time, minimizing disk I/O and improving performance.

Dependencies

Before we begin, ensure you have Node.js 22 or newer installed. Create a new project and install the required dependencies:

npm init -y
npm install tar-stream@3.1.7

Creating a real-time log archive

The following example writes an archive to a supplied destination while adding entries. Each log's content is already in memory, but the complete archive does not need to be buffered. Consuming the pack stream while producing entries is essential: otherwise larger archives can stall on backpressure before the caller ever receives a stream.

const tar = require('tar-stream')
const path = require('node:path')
const { createWriteStream } = require('node:fs')
const { pipeline } = require('node:stream/promises')

function validateFilename(filename) {
  if (
    !filename || filename.includes('\0') ||
    path.posix.isAbsolute(filename) || path.win32.isAbsolute(filename) ||
    filename.includes(':') || filename.split(/[\\/]/).includes('..')
  ) {
    throw new Error('Invalid archive filename')
  }
  return filename.replaceAll('\\', '/')
}

async function addLogToArchive(pack, filename, content) {
  return new Promise((resolve, reject) => {
    pack.entry({ name: validateFilename(filename) }, content, (err) => {
      if (err) reject(err)
      else resolve()
    }).on('error', reject)
  })
}

async function createLogArchive(logs, output, { signal } = {}) {
  const pack = tar.pack()
  const completed = pipeline(pack, output, { signal })
  const producing = (async () => {
    try {
      for await (const log of logs) {
        await addLogToArchive(pack, log.filename, log.content)
      }
      pack.finalize()
    } catch (error) {
      pack.destroy(error)
      throw error
    }
  })()
  await Promise.all([completed, producing])
}

// Example usage with error handling
async function archiveLogs(output) {
  const logs = [
    {
      filename: 'build.log',
      content: `Build started at ${new Date().toISOString()}
Installing dependencies...
Build completed successfully.`,
    },
    {
      filename: 'test.log',
      content: `Test suite initiated.
All tests passed without errors.
Execution time: 12s.`,
    },
  ]

  await createLogArchive(logs, output)
}

archiveLogs(createWriteStream('ci-logs.tar')).catch(() => {
  console.error('Failed to create the log archive')
  process.exitCode = 1
})

Performance considerations

When working with tar archives in Node.js, keep these performance optimizations in mind:

Use streams for large files to minimize memory usage
Consider using worker threads for compression in high-throughput scenarios
Implement backpressure handling for large datasets
Use appropriate buffer sizes when dealing with binary data

Security considerations

The validateFilename helper rejects parent-directory segments, null bytes, absolute paths on either Windows or POSIX, and drive/alternate-stream separators before creating an entry. Do not normalize away traversal segments before checking them. Keep archive names relative, use trusted content, and apply separate path and symlink safeguards wherever archives are extracted.

Handling different file types

When archiving various file types, you'll need to handle them appropriately:

async function addFileToArchive(pack, filename, content, encoding = 'utf8') {
  const options = {
    name: validateFilename(filename),
    // Set appropriate mode for executable files
    mode: filename.endsWith('.sh') ? 0o755 : 0o644,
  }

  return new Promise((resolve, reject) => {
    // Handle Buffer input for binary files
    const data = Buffer.isBuffer(content) ? content : Buffer.from(content, encoding)
    pack.entry(options, data, (err) => {
      if (err) reject(err)
      else resolve()
    }).on('error', reject)
  })
}

Troubleshooting common issues

Here are solutions to common challenges when working with tar streams:

Memory leaks:

async function handleArchiveStream(archiveStream, outputStream) {
  await pipeline(archiveStream, outputStream)
}

Incomplete archives:

async function createArchiveWithTimeout(logs, outputStream, timeoutMs = 30000) {
  await createLogArchive(logs, outputStream, {
    signal: AbortSignal.timeout(timeoutMs),
  })
}

This aborts the stream pipeline rather than merely rejecting a separate timer promise. Discard partial output after failure. If logs come from an external async source, that source must also observe cancellation so it does not remain blocked waiting for another log.

Conclusion

Node.js streams combined with tar libraries provide an efficient solution for real-time log archiving in CI pipelines. This approach minimizes disk I/O and enhances performance while maintaining flexibility for various use cases. By implementing proper error handling and security measures, you can build robust and reliable archiving systems for your development workflow.

Happy coding!