Transloadit

What is Speech-to-Text?

Speech-to-text, also called automatic speech recognition or ASR, converts spoken audio into written language. Results may include timestamps, confidence values, detected languages, punctuation, and speaker labels.

How Speech-to-Text works

Many recognizers first convert the waveform into short-time spectral features, then predict the most probable sequence of text tokens for them. Traditional systems trained separate acoustic, pronunciation, and language models, while many current systems train a single end-to-end neural network that maps speech directly to text. Whisper, for example, resamples audio to 16 kHz, computes a log-Mel spectrogram, and decodes text from 30-second windows, so longer recordings are transcribed window by window. Punctuation, capitalization, word timing, and language detection may be produced alongside the transcript, but they are separate outputs with their own uncertainty. Recognition concerns what was said; identifying who said it is a different task, speaker diarization, that can be combined afterward. Services such as Amazon Transcribe also distinguish batch transcription of stored files from streaming transcription of live audio, and the two modes can differ in supported formats and languages.

Key facts

  1. How should I compare recognizers? The standard metric is word error rate (WER): substitutions, deletions, and insertions divided by the number of words in a reference transcript, so lower is better and 0 is a perfect match. WER also counts harmless differences in punctuation, contractions, or number formatting unless both texts are normalized first; the Whisper authors saw WER fall by up to 50 percent on several datasets from normalization alone. Scores also vary widely between datasets, so compare candidates on your own recordings, with the same reference transcripts and the same normalization.
  2. Can a transcript contain words nobody said? With some models, yes. The Whisper authors list repeat loops, dropped words at the start or end of a segment, and complete hallucination unrelated to the audio as known failure modes, particularly in long recordings. An independent study ran sentence-length clips of interview speech, averaging 10 seconds, from speakers with and without aphasia through the Whisper API in 2023. It found invented phrases or sentences in roughly 1 percent of the clips, more often for speakers with longer pauses, and found none comparable when the affected clips were run through services from Google, Amazon, Microsoft, AssemblyAI, and RevAI. That is a rate per short clip, so it does not predict how often an hour-long recording is affected. Review long pauses and non-speech passages; in the WhisperX evaluation, segmenting audio with voice activity detection before decoding reduced hallucination and repetition.
  3. Will I get a time for every word? Not from every recognizer, and precision varies. Whisper predicts start and end times for whole segments; word times are inferred afterward from attention weights, which the WhisperX authors measured as less precise than forced alignment with a separate phoneme model. Google’s Speech-to-Text v1 API sets word offsets only when enableWordTimeOffsets is true and calls the feature experimental with accuracy that can vary, while Amazon Transcribe returns start and end times for every word. Segment times are enough to search a transcript or jump to a passage; word-by-word highlighting and transcript-based editing need word-level times. Subtitle and caption files are timed per phrase: W3C guidance describes captions of one or two lines, each with the time at which that phrase is displayed.
  4. Can I use confidence scores to decide what needs review? Only after testing thresholds on your own audio. Amazon Transcribe returns a confidence score for every word, but Google’s Speech-to-Text v1 API reference says its confidence value is not guaranteed to be accurate or to be provided, and that 0.0 is a sentinel meaning the value was not set, not that the word is wrong. Each provider computes its own score, so a threshold tuned for one recognizer should not be reused with another.

When Speech-to-Text matters

Use transcripts for captions, search, summaries, accessibility, moderation, or analytics. Accuracy can decline with noise, accents, overlapping speech, or specialized vocabulary, so consequential output needs review.

Working with Speech-to-Text

Before production

  1. Check the recognizer’s input guidance before adding cleanup or re-encoding stages. Google recommends lossless FLAC or LINEAR16 audio at the recording’s native sample rate and says noise reduction applied before recognition typically reduces accuracy; Amazon Transcribe also recommends lossless formats.
  2. If you supply custom vocabulary or phrase boosts for names and jargon, re-test on recordings that do not contain those terms. Google’s documentation warns that boosting can make a phrase appear in the transcript even though it was not spoken.
  3. Have a person correct transcripts before publishing them as captions. W3C accessibility guidance says automatically generated captions do not meet accessibility requirements unless they are confirmed to be fully accurate. The review covers more than the recognized words: W3C describes captions as synchronized text for speech and for the non-speech audio information needed to understand the content, and its explanation of the WCAG captions requirement adds that captions identify who is speaking.

How Transloadit helps with Speech-to-Text

Transloadit’s /speech/transcribe Robot transcribes speech in audio or video files. Its format parameter selects the result: json with timestamped words, plain text, srt or webvtt subtitle files, or meta, which stores the transcript on the file so that later Steps can use it.

Recognition is carried out by the provider selected with provider, so language support and features such as speaker labels depend on that provider. The spoken language is set with a different parameter per provider (source_language or target_language), and neither translates the speech. The Robot documentation lists the current providers and parameters.

Explore Transloadit’s audio capabilities

Sources

Turn media knowledge into a working pipeline

Connect uploads, processing, AI, storage, and delivery through one declarative API — with the encoding stack, scaling, and format churn handled for you.

Try Transloadit for free