Efficient file deduplication with SHA-256 and Node.js
Duplicate files can quickly become a headache in modern applications—eating up storage and slowing queries. In this DevTip we will build a small Node.js server that rejects duplicate uploads by calculating a SHA-256 hash for every incoming file and comparing it against previously-seen ones.
Why not MD5?
MD5 used to be the go-to algorithm for quick integrity checks, but it is no longer considered collision-resistant. For a deduplication system you usually do not need full cryptographic strength, yet picking a broken hash means accepting a (small, but real) risk of two different files producing the same digest. SHA-256 is still fast, ships with Node.js, and eliminates that concern—so that is what we will use throughout the article.
Understand content-based deduplication
Content-based deduplication, a form of content addressing, identifies identical files by their binary content instead of their filenames or metadata. The workflow is simple:
- Receive an upload.
- Stream the bytes through a hash function.
- Look up that hash in a datastore.
- Reject, reuse, or store the file depending on whether the hash already exists.
Because hashes have a fixed length (256 bits for SHA-256) you can use them as lightweight primary keys regardless of the original file size.
Set up the project
mkdir file-deduplication && cd $_
npm init -y
npm install express@^4.21.2 multer@^2.3.0 sqlite3@^5.1.7
We will keep extra dependencies to a minimum—express for routing, multer for multipart parsing,
and sqlite3 for a tiny persistent database. path and crypto come with Node.js.
Create a local upload handler
This local demonstration has no authentication or per-user isolation. Bind it to localhost, and add both before deploying an upload service. Client-supplied MIME types are only a preliminary filter; validate actual file contents before processing or serving uploads.
// index.js
const express = require('express')
const multer = require('multer')
const path = require('path')
const fs = require('fs')
const crypto = require('crypto')
const sqlite3 = require('sqlite3').verbose()
const app = express()
/* -------------------------------------------------------------------------- */
/* Database */
/* -------------------------------------------------------------------------- */
const db = new sqlite3.Database('deduplication.db')
const createFilesTable = `CREATE TABLE IF NOT EXISTS files (
hash TEXT PRIMARY KEY,
original_name TEXT,
filename TEXT,
file_path TEXT,
size INTEGER,
upload_date TEXT
)`
/* -------------------------------------------------------------------------- */
/* Multer configuration */
/* -------------------------------------------------------------------------- */
const storage = multer.diskStorage({
destination(req, file, cb) {
const dir = 'uploads'
fs.mkdirSync(dir, { recursive: true })
cb(null, dir)
},
filename(req, file, cb) {
const unique = Date.now() + '-' + Math.round(Math.random() * 1e9)
cb(null, `${file.fieldname}-${unique}${path.extname(file.originalname)}`)
},
})
const MAX_SIZE = 10 * 1024 * 1024 // 10 MB
const upload = multer({
storage,
limits: { fileSize: MAX_SIZE, files: 1 },
fileFilter(req, file, cb) {
const allowed = ['image/jpeg', 'image/png', 'image/gif', 'application/pdf']
cb(null, allowed.includes(file.mimetype))
},
})
/* -------------------------------------------------------------------------- */
/* Helper: stream hashing */
/* -------------------------------------------------------------------------- */
function hashFile(filePath) {
return new Promise((resolve, reject) => {
const hash = crypto.createHash('sha256')
const stream = fs.createReadStream(filePath)
stream.on('data', (chunk) => hash.update(chunk))
stream.on('end', () => resolve(hash.digest('hex')))
stream.on('error', reject)
})
}
function removeUpload(filePath) {
fs.unlink(filePath, (error) => {
if (error && error.code !== 'ENOENT') console.error('Upload cleanup failed:', error.code)
})
}
/* -------------------------------------------------------------------------- */
/* Routes */
/* -------------------------------------------------------------------------- */
app.post('/upload', (req, res) => {
upload.single('file')(req, res, async (err) => {
if (err instanceof multer.MulterError) {
return res.status(400).json({ error: 'Upload rejected by size or file-count limits' })
}
if (err) {
return res.status(500).json({ error: 'Upload failed' })
}
if (!req.file) {
return res.status(400).json({ error: 'No file uploaded' })
}
const { path: filePath, originalname, filename, size } = req.file
try {
const digest = await hashFile(filePath)
// The unique key arbitrates concurrent identical uploads atomically.
db.run(
`INSERT INTO files (hash, original_name, filename, file_path, size, upload_date)
VALUES (?, ?, ?, ?, ?, datetime('now')) ON CONFLICT(hash) DO NOTHING`,
[digest, originalname, filename, filePath, size],
function (insertErr) {
if (insertErr) {
removeUpload(filePath)
return res.status(500).json({ error: 'Could not store file metadata' })
}
if (this.changes === 0) {
removeUpload(filePath)
return res.status(409).json({ error: 'Duplicate file detected', hash: digest })
}
return res.json({ message: 'File uploaded successfully', hash: digest, size })
},
)
} catch (hashErr) {
removeUpload(filePath)
res.status(500).json({ error: 'Could not hash the uploaded file' })
}
})
})
app.get('/files', (req, res) => {
db.all('SELECT original_name as name, hash, size, upload_date FROM files', (err, rows) => {
if (err) return res.status(500).json({ error: 'Could not list files' })
res.json(rows)
})
})
/* -------------------------------------------------------------------------- */
/* Start server */
/* -------------------------------------------------------------------------- */
const PORT = process.env.PORT || 3000
db.exec(createFilesTable, (error) => {
if (error) {
console.error('Could not initialize the database')
db.close()
process.exitCode = 1
return
}
app.listen(PORT, '127.0.0.1', () => {
console.log(`Server listening on http://localhost:${PORT}`)
})
})
Handle large files efficiently
hashFile streams data, keeping its working buffer bounded. Each hash.update() call still uses
CPU synchronously, while the sqlite3 package performs database work asynchronously. For
gigabyte-scale workloads, consider background workers and bound concurrent uploads. The example
deliberately limits each upload to 10 MB.
Performance, storage, and security tips
- Add an index on the
hashcolumn for faster look-ups when you migrate to PostgreSQL or another full-fledged database. - Keep your upload directory outside the web root and back it up separately from your metadata database.
- Cache recently used hashes in Redis when your workload involves many near-identical files (for example user avatars).
- Sanitize filenames to prevent path traversal vulnerabilities if you use user-provided names directly in file system operations (though our example uses a generated filename).
- Regularly update dependencies to patch known vulnerabilities.
Wrap-up
A hash-based approach to deduplication is compact, accurate, and easy to bolt onto any existing upload endpoint. By switching to SHA-256 and streaming file reads you get collision resistance and constant memory usage with only a handful of lines of code.
If you would rather offload hashing entirely, our 🤖 /file/hash Robot in the Media Cataloging service can generate SHA-256 (and many other) checksums for you—no server maintenance required.