Transcribe speech in audio or video files
🤖/speech/transcribe transcribes speech in audio or video files.

You can use the text that we return in your application, or you can pass the text down to other Robots to filter audio or video files that contain (or do not contain) certain content, or burn the text into images or video for example.
Another common use case is automatically subtitling videos, or making audio searchable.
Set speaker_labels to true when you want the transcript to distinguish recurring speakers:
{
"steps": {
"transcribed": {
"use": ":original",
"robot": "/speech/transcribe",
"provider": "aws",
"format": "json",
"speaker_labels": true,
"max_speakers": 3
}
}
}
Speaker labels are currently supported by the aws, gcp and elevenlabs providers. If you enable
speaker_labels without setting provider, Transloadit uses aws for that Step, or
gcp when vocabulary is also supplied. Labels
are normalized as speaker_1, speaker_2, and so on:
{
"text": "Hello there. Hi!",
"words": [
{ "text": "Hello", "startTime": 0, "endTime": 0.5, "speaker": "speaker_1" },
{ "text": "there", "startTime": 0.6, "endTime": 1, "speaker": "speaker_1" },
{ "text": "Hi!", "startTime": 1.2, "endTime": 1.8, "speaker": "speaker_2" }
],
"segments": [
{ "text": "Hello there", "startTime": 0, "endTime": 1, "speaker": "speaker_1" },
{ "text": "Hi!", "startTime": 1.2, "endTime": 1.8, "speaker": "speaker_2" }
]
}
The other formats show speakers too: text returns one Speaker 1: … paragraph per turn,
webvtt wraps each cue in a <v Speaker 1> voice span, and srt names the speaker when the
turn changes.
ElevenLabs
Set provider to "elevenlabs" to use ElevenLabs Scribe (scribe_v2). Besides speech, it
tags audio events such as (laughter) or (applause) in the transcript, and it detects the
spoken language automatically unless you set source_language. Files can be up to 10 hours long.
For production, we recommend using your own ElevenLabs account. Store your key as
elevenlabs_api_key in a Template Credential of the AI Providers type and reference it with
credentials. With your own account, you have your own ElevenLabs limits and credits.
ElevenLabs bills you directly, and Transloadit charges only its fee: 30% of ElevenLabs’ list price
of $0.0037 per minute (about $0.0011 per minute, at least $0.0005 per Step).
Transloadit’s shared ElevenLabs account is the default and a quick way to try things out. Its credits and capacity are shared by all customers, so your Step can be delayed or refused for reasons outside your control. With this account, Transloadit bills ElevenLabs’ list price of $0.0037 per minute plus a 30% fee.
With either account, Transloadit retries HTTP 429 limit responses up to twice; if the limit persists, the Step fails. Exhausted credits fail without retrying. When the Step’s regular data volume charge is higher than the pricing above, that applies instead. If ElevenLabs rejects your key, the Step fails; Transloadit never falls back to its own key.
{
"steps": {
"transcribed": {
"use": ":original",
"robot": "/speech/transcribe",
"provider": "elevenlabs",
"credentials": "my_elevenlabs_credentials",
"format": "srt"
}
}
}
Usage example
Transcribe French speech from uploaded audio or video, and save it to a text file:
{
"steps": {
"transcribed": {
"format": "text",
"provider": "replicate",
"robot": "/speech/transcribe",
"target_language": "french",
"use": ":original"
}
}
}Parameters
interpolateboolean | Record<string, boolean>Controls whether Assembly Variables are interpolated for individual instruction fields.
By default, most Robot instruction fields interpolate Assembly Variables. Set this to
falseto treat every instruction field as literal text, or set an individual field path tofalseto treat only that field as literal text. For Robot-specific fields that are literal by default, set this totrueor set that field path totrueto opt back into interpolation.Use field names such as
path, or dotted paths such asffmpeg.vffor nested objects.output_metaRecord<string, boolean> | boolean | Array<string>Requests CPU-intensive metadata, disabled by default for faster Assemblies.
For images,
"has_transparency": truedetects transparency and"dominant_colors": trueextracts hex colors. Add"dominant_colors_sampled": truefor faster, approximate colors from downscaled frames. Sampling requiresdominant_colorsand changes neither image bytes nor billing. Its method and palette may vary. Omit or disable it for full-resolution colors.For images, you can also add
"blurhash": trueto extract a BlurHash string — a compact representation of a placeholder for the image, useful for showing a blurred preview while the full image loads.For images,
"thumbhash": trueinstead extracts a base64-encoded ThumbHash intometa.thumbhash, together withmeta.has_alpha(whether an alpha channel exists, even when fully opaque). It describes EXIF-oriented pixels and uses the first frame of animated images. Extraction is best-effort: images above 40 megapixels, unsupported formats, or a failed/bounded decode produce no placeholder. Successful extraction adds a metadata charge equivalent to 20% of that file's bytes. No ThumbHash surcharge applies when disabled or when no hash is produced.Set this option on the Step producing the image, such as
/upload/handlefor uploaded originals or/image/resizefor processed outputs. Setting it only on/transloadit/storedoes not request extraction: storage preserves the producer's metadata. Transloadit Storage persists a generated hash with its immutable version and returns it in stored results and native asset reads. Direct S3 uploads do not generate placeholders. A placeholder contains image information, so protect it with the same access controls as the full image.For videos, you can add the
"colorspace": trueparameter to extract the colorspace of the output video.For videos, you can also add
"interlaced": trueto detect whether the video is interlaced. This combines the cheap ffprobefield_orderflag with a boundedidetsampling pass over the first frames of the source, exposinginterlaced,field_order, and a diagnosticinterlace_detectionobject underfile.meta. This is computationally expensive and billed accordingly.For audio, you can add
"mean_volume": trueto get a single value representing the mean average volume of the audio file.You can also set this to
falseto skip metadata extraction and speed up transcoding.user_metaRecord<string, any>(default:{})Adds custom JSON metadata to each emitted file without changing its contents. Nested objects and arrays are supported.
Inheritance depends on the Robot. Values merge with existing
user_metaon the output file; the current Step replaces matching top-level keys. Assign required keys explicitly when a Robot creates fresh outputs.In processing Steps,
${file.*}refers to the first input and${result.*}to the emitted file. Values are evaluated per output after the Robot runs, before subsequent metadata extraction and temporary storage. On:original, values are evaluated per upload before metadata extraction.Downstream Steps read
${file.user_meta.key}. See Custom metadata for a complete example and inheritance rules.resultboolean(default:false)Whether the results of this Step should be present in the Assembly Status JSON
queuebatchSetting the queue to 'batch', manually downgrades the priority of jobs for this step to avoid consuming Priority job slots for jobs that don't need zero queue waiting times
force_acceptboolean(default:false)Force a Robot to accept a file type it would have ignored.
By default, Robots ignore files they are not familiar with. 🤖/video/encode, for example, will happily ignore input images.
With the
force_acceptparameter set totrue, you can force Robots to accept all files thrown at them. This will typically lead to errors and should only be used for debugging or combatting edge cases.ignore_errorsboolean | Array<meta | execute>(default:[])Ignore errors during specific phases of processing.
Setting this to
["meta"]will cause the Robot to ignore errors during metadata extraction.Setting this to
["execute"]will cause the Robot to ignore errors during the main execution phase.Setting this to
trueis equivalent to["meta", "execute"]and will ignore errors in both phases.usestring | Array<string> | Array<object> | objectSpecifies which Step(s) to use as input.
- You can pick any names for Steps except
":original"(reserved for user uploads handled by Transloadit) - You can provide several Steps as input with arrays:
{ "use": [ ":original", "encoded", "resized" ] } - You can also tag input Steps with
asto pass semantic intent to robots:{ "use": [ { "name": ":original", "as": "image" }, { "name": ":original", "as": "mask" } ] }
TipThat's likely all you need to know about
use, but you can view Advanced use cases.- You can pick any names for Steps except
providerauto | aws | elevenlabs | gcp | replicate(default:"auto")Chooses the best provider based on your request.
Set this to
"aws","gcp","replicate", or"elevenlabs"to force a specific provider. When set to"auto", Transloadit uses"replicate"by default and uses"aws"whenspeaker_labelsis enabled. If bothspeaker_labelsand a nonemptyvocabularyare set,"auto"uses"gcp"."auto"never selects"elevenlabs".credentialsstringThe name of a Template Credential of the AI Providers type that holds your own
elevenlabs_api_key. Only the"elevenlabs"provider uses it.vocabularyArray<string>Words or phrases to favor during speech recognition, using the spelling you want in the output. For example,
["Transloadit", "Uppy", "tus"]helps the provider recognize brand and product names.Supply up to 50 distinct terms, each between 1 and 100 characters after trimming surrounding whitespace. Terms must not contain control characters or case-insensitive duplicates. The terms joined with
", "must fit within 200 UTF-8 bytes, keeping all terms within Whisper’s prompt budget. Omitting this parameter or supplying an empty array leaves recognition unchanged.Supported by
"replicate","gcp"and"elevenlabs". Replicate uses a prompt-capable WhisperX large-v3 model with aligned word timestamps when vocabulary is supplied. GCP uses inline boosted phrase hints. ElevenLabs uses keyterm prompting, which accepts terms of at most 49 characters and five words each, and adds $0.0008 per minute to the ElevenLabs list price. Explicit"aws"requests with vocabulary return a validation error because AWS requires a pre-created Custom Vocabulary. Use"auto","replicate", or"gcp"instead.Hints improve recognition but do not guarantee that a term is recognized. Afterwards, complete case-insensitive matches use your supplied spelling in every output format, including JSON words and speaker segments, without changing timestamps. Similar-sounding words are never replaced:
"transloadit"becomes"Transloadit", but"Transloaded"stays unchanged.granularityfull | list(default:"full")Both
"full"and"list"currently return the same complete transcription response."list"remains accepted for backward compatibility.formatjson | meta | srt | text | webvtt(default:"json")Output format for the transcription.
"text"outputs a plain text file that you can store and process."json"outputs a JSON file containing timestamped words. Whenspeaker_labelsis enabled, words can includespeakerlabels and the JSON can also include groupedsegmentsby speaker."srt"and"webvtt"output subtitle files of those respective file types, which can be stored separately or used in other encoding Steps."meta"does not return a file, but stores the data inside Transloadit's file object (under${file.meta.transcription.text},${file.meta.transcription.words}, and, when speaker labels are available,${file.meta.transcription.segments}) that's passed around between encoding Steps, so that you can use the values to burn the data into videos, filter on them, etc.
speaker_labelsboolean(default:false)When enabled, Transloadit asks the transcription provider to distinguish different speakers. JSON and meta output can then include
speakerlabels such as"speaker_1"on individual words, plus groupedsegmentsby speaker. Text, SRT, and WebVTT output behavior is unchanged.Speaker labels identify recurring voices, not real person names. Accuracy depends on audio quality, background noise, overlapping speech, and the number of speakers.
max_speakersstring | number(default:10)The maximum number of speakers to detect when
speaker_labelsis enabled.source_languagestring(default:"en-US")Sets the spoken language for the
"aws","gcp"and"elevenlabs"providers. The transcription stays in the spoken language; this parameter does not translate speech. ElevenLabs uses only the language part, such as"de"for"de-DE", and detects the language itself when you omit this parameter.The language should be specified in the BCP-47 format, such as
"en-GB","de-DE"or"fr-FR". Please also consult the list of supported languages for thegcpprovider and theawsprovider.target_languagestringSets the spoken language hint for the
"replicate"provider. Omit this parameter to detect the spoken language automatically. Set it to the full lowercase English language name, such as"french", when the spoken language is known. Despite its name, this parameter does not translate speech.For the
"aws"and"gcp"providers, usesource_languageinstead.
Demos
Related blog posts
- Tech preview: new AI Robots for enhanced media processing
- New feature: auto-transcribe videos with subtitles
- Building a screen reader plugin with /text/speak Robot
- Building an AI-powered video dubber with Transloadit
- Transloadit’s 2021 milestones and progress
- Build a Reddit video subtitling bot with Transloadit
- Creating engaging audio visualizations with Transloadit
- What’s new in video: transitions, chapters, CMAF, and more
- Social-style captions with the spoken word highlighted
- Transcripts that know who’s talking