Efficient file deduplication with SHA-256 and Node.js

Tim Koschützki

Co-founder · Berlin, Germany · Show bio

Use a file’s SHA-256 digest as a unique database key to reject repeat uploads, even when their filenames differ. This walkthrough builds a local Node.js service, sends files with cURL, and checks that its deduplication records survive a restart.

Understand content-based deduplication

The server saves each incoming file under a generated name, streams its bytes through SHA-256, and tries to insert the digest into SQLite. A successful insert keeps the file. A conflicting insert removes the new copy and returns HTTP 409 Conflict.

The important decision happens in one database statement: INSERT … ON CONFLICT(hash) DO NOTHING. SQLite’s UPSERT behavior⁠ lets the unique key arbitrate concurrent uploads. Looking up a digest first and inserting later would let two requests both observe that it is absent. The primary key already supplies uniqueness; a second hash index is unnecessary.

This detects identical bytes. Renaming a file does not change its digest, but re-encoding an image or changing embedded metadata can. It does not find visually similar images.

Why not MD5?

MD5 is unsuitable when collision resistance matters⁠, including when someone could deliberately upload different files with the same digest. SHA-256 provides collision resistance⁠, not a mathematical guarantee that collisions cannot exist. This example treats matching digests as duplicates. If your application requires an exact equality check, compare the stored and incoming bytes before discarding a matching upload and provide a separate way to store a collision.

Set up the project

Use a Linux shell, cURL, Node.js 24.15.0, and Corepack with Yarn 4.12.0 available. The example also runs on Node.js 26.8.1. Node’s built-in TypeScript support⁠ runs the saved .ts file without a compiler step; it does not type-check it.

Paste this into a terminal. It creates a new project and returns to your original directory. If file-deduplication already exists, choose a fresh parent directory; the commands refuse to reuse it.

(
  mkdir file-deduplication &&
  cd file-deduplication &&
  printf '%s\n' '{"private":true,"type":"module","packageManager":"yarn@4.12.0"}' > package.json &&
  printf '%s\n' 'nodeLinker: node-modules' 'enableScripts: true' > .yarnrc.yml &&
  touch yarn.lock &&
  corepack yarn add --exact express@5.2.1 multer@2.4.0 sqlite3@5.1.7
)

Wait for installation to succeed before continuing. sqlite3 includes a native binding; if its prebuilt binary is unavailable for your platform, its installation guide⁠ describes the build prerequisites. The versions above use SQLite 3.44.2 in the tested Linux setup. The local yarn.lock makes this an independent project, and node-modules lets node resolve its packages directly. Install scripts are enabled for the native SQLite binding. The sqlite3 repository is now archived, so treat this as a pinned local example and choose a maintained database driver before adapting it for a new deployed service.

Create a local upload handler

Save the complete block as file-deduplication/server.ts. This service binds to 127.0.0.1 and has no authentication. It accepts arbitrary file bytes, including empty files, and neither processes nor serves them. Keep it local; MIME filtering would not establish that an upload is safe.

import { createHash } from 'node:crypto'
import { createReadStream } from 'node:fs'
import { rm } from 'node:fs/promises'

import express, { type ErrorRequestHandler } from 'express'
import multer from 'multer'
import sqlite3 from 'sqlite3'

async function hashFile(filePath: string): Promise<string> {
  const hash = createHash('sha256')
  for await (const chunk of createReadStream(filePath)) hash.update(chunk)
  return hash.digest('hex')
}

async function storeFile(
  db: sqlite3.Database,
  filePath: string,
  name: string,
  size: number,
): Promise<{ hash: string; stored: boolean }> {
  let stored = false
  try {
    const hash = await hashFile(filePath)
    const changes = await new Promise<number>((resolve, reject) => {
      db.run(
        `INSERT INTO files (hash, original_name, file_path, size)
         VALUES (?, ?, ?, ?) ON CONFLICT(hash) DO NOTHING`,
        [hash, name, filePath, size],
        function (error) {
          if (error) reject(error)
          else resolve(this.changes)
        },
      )
    })
    stored = changes === 1
    return { hash, stored }
  } finally {
    // Finish cleanup before the route sends a duplicate or failure response.
    if (!stored) await rm(filePath, { force: true })
  }
}

async function main(): Promise<void> {
  const port = Number(process.env.PORT ?? '3000')
  if (!Number.isInteger(port) || port < 1 || port > 65535) {
    console.error('PORT must be an integer from 1 to 65535')
    process.exit(1)
  }

  const db = await new Promise<sqlite3.Database>((resolve, reject) => {
    const database = new sqlite3.Database('deduplication.db', (error) => {
      if (error) reject(error)
      else resolve(database)
    })
  })
  await new Promise<void>((resolve, reject) => {
    db.exec(
      `CREATE TABLE IF NOT EXISTS files (
        hash TEXT NOT NULL PRIMARY KEY,
        original_name TEXT NOT NULL,
        file_path TEXT NOT NULL,
        size INTEGER NOT NULL,
        upload_date TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
      )`,
      (error) => (error ? reject(error) : resolve()),
    )
  })

  const app = express()
  const upload = multer({
    dest: 'uploads/',
    limits: { fileSize: 10 * 1024 * 1024, files: 1, fields: 0 },
  })

  app.post('/upload', upload.single('file'), async (req, res) => {
    if (!req.file) return res.status(400).json({ error: 'Send one file in the file field' })
    const { path, originalname, size } = req.file
    const { hash, stored } = await storeFile(db, path, originalname, size)
    if (!stored) return res.status(409).json({ error: 'Duplicate file detected', hash })
    return res.status(201).json({ message: 'File stored', hash, size })
  })

  app.get('/files', (_req, res, next) => {
    db.all(
      'SELECT original_name AS name, hash, size, upload_date FROM files ORDER BY hash',
      (error, rows) => {
        if (error) return next(error)
        res.json(rows)
      },
    )
  })

  const handleError: ErrorRequestHandler = (error: unknown, _req, res, _next) => {
    if (error instanceof multer.MulterError) {
      const status = error.code === 'LIMIT_FILE_SIZE' ? 413 : 400
      res.status(status).json({ error: 'Send one file of at most 10 MiB and no text fields' })
      return
    }
    console.error('Request failed; check the database and upload directory')
    res.status(500).json({ error: 'Could not complete the request' })
  }
  app.use(handleError)

  await new Promise<void>((resolve, reject) => {
    app.listen(port, '127.0.0.1', (error?: Error) => {
      if (error) reject(error)
      else resolve()
    })
  })
  console.log(`Listening on http://127.0.0.1:${port}`)
}

main().catch((error: unknown) => {
  const code = error instanceof Error && 'code' in error ? error.code : 'STARTUP_FAILED'
  console.error(`Could not start server (${code}); check the port and storage permissions`)
  process.exit(1)
})

Multer’s disk storage⁠ assigns a generated filename instead of using the client’s name as a path. SQLite retains the first accepted name as metadata. The regular function callback in db.run is intentional: this.changes belongs to that completed statement⁠ and distinguishes an inserted row from a conflict that did nothing.

Start the server

From the same parent directory where you ran setup, paste:

(
  cd file-deduplication &&
  node server.ts
)

Wait for Listening on http://127.0.0.1:3000. If you get EADDRINUSE, stop your existing listener or set PORT=3001 before node server.ts and use that port in every request below. The startup callback checks errors because Express 5 reports bind failures there⁠. A database or directory permission error must be fixed before the server can start.

Upload, repeat, and change the bytes

In a second terminal, upload six bytes, including the newline:

printf 'hello\n' | curl -sS -i -F 'file=@-;filename=hello.txt' http://127.0.0.1:3000/upload

The response has status 201 Created and this JSON body:

{"message":"File stored","hash":"5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03","size":6}

Upload the same bytes under a different name:

printf 'hello\n' | curl -sS -i -F 'file=@-;filename=renamed.txt' http://127.0.0.1:3000/upload

Expect 409 Conflict, with error set to Duplicate file detected and the same hash. The first stored copy remains. These upload commands intentionally omit cURL’s -f option so you can inspect the expected 409 response; a successful cURL exit alone does not mean the server accepted a file.

Now send different bytes using the first filename, then send an empty file:

printf 'different\n' | curl -sS -i -F 'file=@-;filename=hello.txt' http://127.0.0.1:3000/upload
curl -sS -i -F 'file=@/dev/null;filename=empty.bin' http://127.0.0.1:3000/upload

Both return 201, with sizes 10 and 0. Repeating either request returns 409. There are now three files under file-deduplication/uploads/ and three database records, even though two accepted files have the same original name. List those records with:

curl -fsS http://127.0.0.1:3000/files

After all requests finish, stop the server with Ctrl+C, start it again using the same command, and repeat the first upload. It still returns 409. Both deduplication.db and uploads/ live in the project directory, so keep them together and restart from that directory. The example appends new content and keeps the first copy of repeated content; restarting does not clear either store.

For overlapping uploads of new identical content, the unique insert admits one request with 201. The others receive 409 after their extra files are removed. Missing files, wrong field names, extra files, and text fields receive 400; files larger than 10 MiB receive 413. A storage failure returns a generic 500, so check the server terminal and filesystem before retrying.

Handle large files efficiently

The incremental hash API⁠ lets hashFile read chunks instead of buffering the whole file. Multer first writes the complete upload to disk, then hashing reads it again. Duplicate detection therefore saves retained storage, but does not avoid the upload bandwidth or temporary disk use.

Each hash.update() call consumes CPU on the main thread. The 10 MiB per-file limit bounds this local example’s input size; it does not bound the number of simultaneous requests or total disk usage. Larger workloads need concurrency and storage limits before increasing that cap.

Performance, storage, and security tips

The SQLite insert is atomic, but the file write and database insert are not one transaction. A crash between them can leave an unreferenced file. Deleting or corrupting a stored file does not remove its hash record, so a later upload can be rejected even though the stored bytes are missing. Cleanup can also fail if storage permissions change. This example does not reconcile those cases, verify stored files on every lookup, or promise durability through a power loss.

Back up the database and files as a consistent pair while the service is stopped. Before adapting this for public uploads, add authentication, ownership checks, quotas, and a recovery process that reconciles records with files. A global duplicate response can reveal that someone else uploaded particular content; choose the scope of deduplication deliberately.