Tutorial: file filtering & virus scanning with Transloadit

As people are moving more parts of their life to digital platforms, a lot of consideration is made with regard to how many GBs of data they should subscribe to. Stakes are also getting higher, as keeping your users — and their data — safe has become more important than ever.

Whether we are building native apps, APIs, or web applications, it is our responsibility as developers to invest an extensive amount of thought and time into keeping our users safe from those with malicious intent. At the same time, we also have to ensure that storage space is utilized optimally in order to allow for more free storage across the board.

Transloadit makes it easier for developers to control the storage space used during the process of file encoding, and offers reliable methods for filtering out viruses through the /file/filter and /file/virusscan Robots, respectively.

The /file/filter Robot

The /file/filter Robot lets you reject or redirect files against different encoding Steps. For example, we can ask this Robot to reject files that are encoded in a particular way, or files that are smaller / larger than a specific size.

It largely operates by following the accepts and declines parameters, which can each be set to an array containing three members:

• A value or job variable, such as ${file.mime}, ${file.size}
• One of the following operators: =, <, >, <=, >=, !=, regex, !regex
• A value or job variable, such as 50 or "foo"

Let's take a look at the example below, which rejects files that are larger than 20 MB in size:

{
  "steps": {
    ":original": {
      "robot": "/upload/handle"
    },
    "files_filtered": {
      "use": [":original"],
      "robot": "/file/filter",
      "result": true,
      "declines": [["${file.size}", ">", "20971520"]],
      "error_on_decline": true
    }
  }
}

The /file/virusscan Robot

The /file/virusscan Robot keeps you and your users safe by rejecting millions of potential trojans, viruses, malwares, or other malicious threats before they reach your platform.

This Robot is built on top of ClamAV, the best open source antivirus engine available. We update its signatures on a daily basis.

The example below shows basic usage for this Robot:

{
  "steps": {
    ":original": {
      "robot": "/upload/handle"
    },
    "virus_scanned": {
      "use": [":original"],
      "robot": "/file/virusscan",
      "result": true,
      "error_on_decline": true
    }
  }
}

Creating the Assembly

We want to create an Assembly that combines the two Robots together, to ensure that all file uploads adhere to our desired file checks and virus scans.

First, we'll have to download a test virus file here.

Note that the test virus files in the link above would do no harm to your computer. They simply contain a specific signature created by the EICAR organization (European Expert Group for IT Security) that was specifically designed to test the functional behavior of antivirus software.

These are the Assembly Instructions we will use to test this virus file:

{
  "steps": {
    ":original": {
      "robot": "/upload/handle"
    },
    "files_filtered": {
      "use": [":original"],
      "robot": "/file/filter",
      "result": true,
      "declines": [["${file.size}", ">", "1024"]],
      "error_on_decline": true
    },
    "virus_scanned": {
      "use": [":original"],
      "robot": "/file/virusscan",
      "result": true,
      "error_on_decline": true
    }
  }
}

In the first Step, the /upload/handle Robot uploads a file to Transloadit.

In the next Step, the /file/filter Robot filters out any file with a size larger than 1 MB.

In the final Step, the /file/virusscan Robot rejects any file that has a virus in it.

To try out our newly created Template, we just have to run the following Node.js code, taking the webpage we want as an argument. Feel free to try it out yourself!

const TransloaditClient = require('transloadit')

const transloadit = new TransloaditClient({
  authKey: process.env.TRANSLOADIT_AUTH_KEY,
  authSecret: process.env.TRANSLOADIT_SECRET_KEY,
})

const options = {
  params: {
    steps: {
      ':original': {
        robot: '/upload/handle',
      },
      files_filtered: {
        use: [':original'],
        robot: '/file/filter',
        result: true,
        declines: [['${file.size}', '>', '1024']],
        error_on_decline: true,
      },
      virus_scanned: {
        use: [':original'],
        robot: '/file/virusscan',
        result: true,
        error_on_decline: true,
      },
    },
  },
}

transloadit.addFile('file', process.argv[2])
transloadit.createAssembly(options, (err, result) => {
  if (err) {
    throw err
  }
  console.log({ result })
})

The results

Our Assembly seems to be working perfectly! After running the Node.js code above with two files (the virus file alongside a different file with a size of 2.9MB) the dashboard shows us on both occasions that there were errors regarding both files.

I hope you have enjoyed learning a bit more about how you can combine the /file/filter and /file/virusscan Robots, in an effort to better monitor storage space usage and reliably prevent infected files.