How to automate content moderation using Transloadit (NSFW)
Security needs to be at the forefront of every developer's mind in the modern day and age. Unfortunately, not every user will have good intentions with their file uploads, and it's important to have measures in place to deal with these kinds of files. Today, we're going to take a look at several approaches you can take to stop different types of malicious files from reaching your servers and end-users.
Why is content moderation important?
Content moderation is critical for any online business. Users don't want to be exposed to graphic content unwillingly, and advertisers don't want to be shown next to it. Effective content moderation protects both your brand and your users, yet it can be expensive to hire a full-time team of content moderators. Transloadit can help screen for unwanted content and malware, while metadata and file hashes provide additional signals for your moderation workflow. These checks are not a guarantee of safety or permission to use a file. Let's find out how 👇
Automatically reject NSFW content
You can use our /image/describe Robot to generate a list of tags based on the image provided, and then filter out files that contain tags such as gore, hateful symbols or nudity.
You can check out the full list of tags provided by both AWS and GCP.
Provider labels can change. Before relying on this filter, inspect the /image/describe output for
representative test images using format: "json" in a separate diagnostic Assembly. Match your
filter to the labels actually returned, then use format: "meta" for the filtering pipeline below.
Now, let's see a Template that uses this:
{
"steps": {
":original": {
"robot": "/upload/handle"
},
"described": {
"use": ":original",
"robot": "/image/describe",
"explicit_descriptions": true,
"format": "meta",
"granularity": "list",
"provider": "aws"
},
"filtered": {
"use": "described",
"robot": "/file/filter",
"result": true,
"declines": [
["${file.meta.descriptions}", "includes", "Hate Symbols"],
["${file.meta.descriptions}", "includes", "Explicit Nudity"],
["${file.meta.descriptions}", "includes", "Visually Disturbing"]
],
"error_msg": "One file contains explicit content!",
"error_on_decline": true
}
}
}
Here we're generating a list of descriptors for our image, and if our filter detects any of our top-level categories in this list, the file will be rejected and our Assembly stopped. Only publish the filtered results after the Assembly succeeds, not the original upload. Automated labels can miss content or flag acceptable images, so choose categories and human review rules for your application's requirements.
Checking copyright metadata
Metadata cannot establish whether an uploader has permission to use an image. Copyright information can be absent, removed or inaccurate, and a copyright notice does not itself make an upload unauthorized. Hiding a metadata check is not a security control.
The following Template demonstrates a metadata filter only: files with a nonempty copyright field are excluded from its results, while files without that field pass. A passing result is not copyright clearance. Keep permission checks and any required human review separate from this example; do not use it as an automatic copyright verdict.
{
"steps": {
":original": {
"robot": "/upload/handle"
},
"filter": {
"use": ":original",
"robot": "/file/filter",
"result": true,
"accepts": [["${file.meta.copyright}", "empty", ""]],
"error_on_decline": false
}
}
}
Detecting malware
Naturally, the last thing you want your users uploading is malware. Whether it reaches other users, or your system critical machines, it's important that viruses get stopped as early as possible.
Our /file/virusscan Robot detects malware for you, and cancels the Assembly when a scan detects a threat. Only release results after a successful scan. Scanning reduces risk but cannot guarantee that a file is harmless; keep your other upload controls in place. Consult the Robot documentation and pricing for current availability and charges.
{
"steps": {
":original": {
"robot": "/upload/handle"
},
"virus_scanned": {
"use": ":original",
"robot": "/file/virusscan",
"result": true,
"error_on_decline": true
}
}
}
Excluding known files
In the case that a user is uploading thousands of the same file, and for one reason or another, it's getting through your other layers of defense, you can take the file hash and add it to a denylist. The following example rejects an empty file using its SHA-256 digest. Replace that digest with the SHA-256 of a known unwanted file for your own policy. Changing a file's bytes changes its hash, so an exact-match denylist does not detect modified copies.
Here is a Template that aims to do exactly this.
{
"steps": {
":original": {
"robot": "/upload/handle"
},
"hash_files": {
"use": ":original",
"robot": "/file/hash",
"algorithm": "sha256"
},
"filter_files": {
"use": "hash_files",
"robot": "/file/filter",
"result": true,
"error_on_decline": true,
"declines": [["${file.meta.hash}", "===", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"]]
}
}
}
In short, we generate a hash for each uploaded file. If it equals the hash of a file we're looking to avoid, the file will not pass the filter.
Wrapping up
As you can see, there are several approaches to screening file uploads. Combine appropriate checks with access controls, safe storage and human review instead of relying on any one signal.
We're constantly looking to help make things more secure at Transloadit, so for your next read maybe take a look at our security page to see some of the steps we're taking to accomplish this.