Uploading and processing CSV files via a REST API is a common requirement in modern web applications. This tutorial will guide you through building a Node.js API using Express that can receive CSV files, parse their contents, and store the data. We'll use Multer for handling file uploads and csv-parser for parsing CSV data.

Prerequisites

Before we begin, ensure you have the following installed:

  • Node.js (version 18 or higher)
  • npm (Node Package Manager)

Setting up a Node.js and express server

Let's start by setting up a basic Express server:

  1. Create a new directory for your project and initialize it:
mkdir csv-upload-api
cd csv-upload-api
npm init -y
  1. Install the necessary dependencies:
npm install express multer csv-parser
  1. Create a file named server.js and add the following code:
const express = require('express')
const app = express()
const port = 3000

app.use(express.json())

app.listen(port, () => {
  console.log(`Server is running on port ${port}`)
})

This sets up a basic Express server listening on port 3000.

Implementing the file upload endpoint

Now, let's implement the file upload endpoint using Multer:

  1. Import the required modules and set up Multer:
const multer = require('multer')
const path = require('path')
const fs = require('fs')
const csv = require('csv-parser')

const upload = multer({
  dest: 'uploads/',
  fileFilter: (req, file, cb) => {
    if (path.extname(file.originalname).toLowerCase() === '.csv') {
      cb(null, true)
    } else {
      cb(new Error('Only CSV files are allowed'))
    }
  },
})

This code configures Multer to store uploaded files in the uploads/ directory and only accept files with a .csv extension.

  1. Add the POST endpoint for file uploads:
app.post('/upload', upload.single('file'), (req, res) => {
  if (!req.file) {
    return res.status(400).send('No file uploaded.')
  }
  res.send('File uploaded successfully.')
})

This code sets up a POST endpoint at /upload that accepts a single file with the field name file. The uploaded files will be stored in the uploads/ directory.

Parsing CSV files with csv-parser

After uploading the CSV file, we need to parse its contents. Let's modify our upload endpoint to include CSV parsing:

app.post('/upload', upload.single('file'), (req, res) => {
  if (!req.file) {
    return res.status(400).send('No file uploaded.')
  }

  const results = []

  fs.createReadStream(req.file.path)
    .pipe(csv())
    .on('data', (data) => results.push(data))
    .on('end', () => {
      // Process the parsed data
      console.log(results)
      // Delete the file after processing
      fs.unlink(req.file.path, (err) => {
        if (err) {
          console.error('Error deleting file:', err)
        }
        res.send(`File uploaded and processed. ${results.length} records parsed.`)
      })
    })
    .on('error', (err) => {
      console.error('Error processing file:', err)
      res.status(500).send('Error processing file.')
    })
})

In this updated code, we're reading the uploaded CSV file, parsing it using csv-parser, and pushing each row into the results array. We also handle errors and delete the uploaded file after processing to clean up.

Processing and storing CSV data

Once the CSV data is parsed, you can process and store it as needed. For example, you might save it to a database. Here's a simple example using an in-memory array:

let storedData = []

app.post('/upload', upload.single('file'), (req, res) => {
  if (!req.file) {
    return res.status(400).send('No file uploaded.')
  }

  const results = []

  fs.createReadStream(req.file.path)
    .pipe(csv())
    .on('data', (data) => results.push(data))
    .on('end', () => {
      storedData = results // Store the parsed data
      // Delete the file after processing
      fs.unlink(req.file.path, (err) => {
        if (err) {
          console.error('Error deleting file:', err)
        }
        res.send(`File uploaded and processed. ${results.length} records stored.`)
      })
    })
    .on('error', (err) => {
      console.error('Error processing file:', err)
      res.status(500).send('Error processing file.')
    })
})

// Endpoint to retrieve stored data
app.get('/data', (req, res) => {
  res.json(storedData)
})

In this code, we're storing the parsed CSV data in the storedData array. We also added a GET endpoint at /data to retrieve the stored data. Remember that storing data in memory is suitable for testing purposes only. For production applications, you should store data in a database like MongoDB or PostgreSQL.

Testing the file upload API with postman

To test your API using Postman:

  1. Open Postman and create a new request.
  2. Set the request method to POST and the URL to http://localhost:3000/upload.
  3. In the Body tab, select form-data.
  4. Add a key named file and set its type to File.
  5. Choose a CSV file from your computer.
  6. Send the request.

You should receive a success message if the file is uploaded and processed correctly. You can also send a GET request to http://localhost:3000/data to retrieve the stored data.

Conclusion

In this tutorial, we've built a REST API that can handle CSV file uploads, parse the data, and store it for further processing. This approach can be extended to handle various types of file uploads and data processing tasks.

By leveraging Node.js streams and libraries like Multer and csv-parser, we can efficiently handle large CSV files without overwhelming server resources. This makes it an excellent solution for applications that need to process substantial amounts of data via REST API file uploads.

Efficient file handling and processing are crucial for building scalable applications. If you're looking for a robust solution for handling file uploads and processing, consider checking out Transloadit's file handling services.