How to build a reliable document intake and OCR pipeline

Start with an intake contract

A document pipeline becomes difficult when the first step accepts anything and the last step assumes everything was understood. Define the intake contract before choosing Robots: which sources are allowed, what the application calls a document, which metadata is required, how large an upload may be, and which outputs downstream systems expect. Enforce a page-count limit in the application before creating the Assembly; the sample below does not cap pages. Large or page-heavy PDFs can exceed the Robot’s fixed polling window while waiting for an asynchronous provider job, even if that job later finishes. Retrying unchanged input does not widen this window; handle repeated timeouts as a failure and reduce or split the input before another attempt.

Keep the uploaded original as the immutable reference. A normalized PDF, an OCR text file, and a search record are derivatives with different lifetimes and failure modes. Give them a shared application identity so a reviewer can move from a search hit to the source bytes that produced it. Page-specific navigation requires a separate page mapping.

Route each input to the right Robot

A filename extension does not tell you whether a PDF contains selectable text or only scanned pixels. Use Transloadit’s resolved ${file.mime} and the application’s policy to route inputs. MIME resolution combines embedded metadata, extension-based information, and content sniffing; the extension can win when embedded metadata is unavailable. This gate is not proof of the file’s contents. If your intake contract requires strict content validation, validate the stored bytes in your application. /file/filter can route files with accepts or declines conditions on ${file.mime}; use error_on_decline when an unsupported input should fail the Assembly.

PDFs can go directly to /document/ocr. Use /image/ocr for short text in photographs and screenshots; its documentation warns that dense document images may be less accurate. For office inputs such as DOCX, use /document/convert to convert to PDF, checking its documented input formats first. This workflow uses its PDF output; other output formats are available, but PDF input is not supported. A Robot can skip inputs it does not accept without failing the Assembly, so add an explicit MIME gate and verify required outputs.

Preserve provenance through derived outputs

Text without provenance is difficult to trust. Store the source document ID, output version, and extraction timestamp with the extracted text. The AWS sample below returns document-level JSON without page attribution. For page-specific search hits, the application must split PDFs before upload and record the original page numbers; this preparation is outside the guide. The sample allows one input per Assembly and uses fixed object names. A page pipeline needs one Assembly per page with page-qualified object keys and a mapping to the parent document. Reusing the same document_id and output_version for multiple pages would overwrite their objects. Never infer the mapping from OCR array order. If the source is replaced, create a new version and let the indexer retire the old record only after the new output is verified.

Keep OCR text separate from the normalized document. A reviewer may need the original scan, the converted PDF, and the extracted text side by side. That relationship is also useful when a different OCR provider produces a better result.

Make review and indexing explicit

The Assembly Status ok value ASSEMBLY_COMPLETED, observed through polling or notifications, means no unignored error stopped the Assembly and its declared jobs, including storage jobs, finished. A Step can skip an input, and ignore_errors can suppress a Step failure, so completion alone does not prove that the required artifacts exist. It also does not mean your application’s search index accepted the text or a human approved a sensitive record. Publish a downstream event only after the outputs you require exist and have passed their checks.

/document/ocr does not expose confidence scores, so this Template cannot route on a provider confidence threshold. Use application checks such as missing output files, empty extracted text, implausibly little text for the reported page count, missing expected phrases, or a high-consequence document class to trigger review. For ordinary documents, an automated path can still record the checks and retain a way to reprocess the source.

Version corrections without losing evidence

OCR corrections and source replacements should create a new derived version. Keep the previous source, output, and review decision long enough for your retention policy so an index result can be explained after a correction.

Use the document ID and output version as the indexing key. Reuse a version only when retrying the same processing attempt; allocate a new version for corrections, changed providers, or deliberate reprocessing. Serialize writes for a version so concurrent retries cannot overwrite a newer accepted result.

Start with a bounded PDF Template

The example below implements only the PDF branch: it allows at most one upload, rejects non-PDF MIME types, requests AWS OCR JSON, and exports the source and JSON to separate fixed keys under one application-controlled document version. Save this JSON as stored Template content, not as complete Assembly request parameters. Your server supplies the Auth Key and signature when creating the Assembly. Test representative PDFs for output shape, usable text, and failure behavior before extending the recipe. auth.max_size limits total expected upload bytes, including request overhead: 104857600 bytes is 100 MiB, not a per-file allowance. It does not guarantee rejection before a resumable upload transfers bytes, or that OCR will finish.

Create a Transloadit account and save the Assembly Instructions as a Template. Create an S3 Template Credential named document-output with key, secret, bucket, and bucket_region; see Template Credentials for setup. For this recipe, the server must construct and submit the entire Assembly request, including the upload, rather than hand signed params to a browser. It must supply and validate non-empty, authorized, path-safe document_id and output_version values in the request’s fields. These become the ${fields.document_id} and ${fields.output_version} variables in the storage paths. Multipart fields also contribute Assembly fields and are not covered by the params signature, so signing alone does not authorize a storage prefix. Do not forward client-supplied fields. Verify returned storage keys against the server’s expected prefix before indexing. Save allow_steps_override: false in the stored Template to prevent runtime Step overrides when using template_id. Keep the bucket private with Block Public Access enabled and check reader authorization; acl: bucket-default omits an object ACL and relies on the bucket’s access policy.

This sample pins provider: aws to make the provider choice explicit, not to guarantee identical responses. The default is auto. Use AWS Textract’s current limits⁠ as the authority for AWS language support. Check provider support before choosing AWS or GCP; their JSON structures differ. Test your document languages and layouts with the chosen provider, and expect responses to change as provider models evolve.

The sample uses format: "json" with granularity: "list" to export an array of recognized line strings for indexing and text-volume checks. This avoids the default full-granularity artifact’s mixture of page, line, and word entries, which can repeat text. format: "text" instead exports plain text and forces list granularity. Neither output includes page attribution or confidence. If you choose full granularity, its optional polygon points use one width and height from PDF metadata, becoming zeros when those dimensions are missing. Validate them before overlays, and never infer page identity from polygons or array order.

The AWS implementation builds this artifact from one provider response and does not follow pagination tokens. Long or dense PDFs can therefore have incomplete output even when OCR succeeds. Check both exported objects and extracted text volume against the reported page count, and send suspicious results to review. Those checks flag problems but do not prove complete coverage; compare against known text in representative source documents before publishing.

The fixed source.pdf and ocr.json object keys make retries target the same storage locations for a document version. They do not deduplicate Assemblies or make OCR deterministic; the application must serialize retries and keep the accepted result mapping. Compare the providers’ charges using Transloadit pricing, including page volume and other processing in the workflow. OCR’s provider-specific page charge is calculated from the input’s reported page_count; compare current charges before estimating the complete workflow.

{
  "allow_steps_override": false,
  "auth": {
    "max_number_of_files": 1,
    "max_size": 104857600
  },
  "steps": {
    ":original": {
      "robot": "/upload/handle"
    },
    "pdf_only": {
      "use": ":original",
      "robot": "/file/filter",
      "accepts": [["${file.mime}", "==", "application/pdf"]],
      "error_on_decline": true,
      "error_msg": "Upload a PDF document"
    },
    "extracted_text": {
      "use": "pdf_only",
      "robot": "/document/ocr",
      "provider": "aws",
      "format": "json",
      "granularity": "list",
      "result": true
    },
    "stored_source": {
      "use": "pdf_only",
      "robot": "/s3/store",
      "credentials": "document-output",
      "path": "documents/${fields.document_id}/${fields.output_version}/source.pdf",
      "acl": "bucket-default",
      "result": true
    },
    "stored_text": {
      "use": "extracted_text",
      "robot": "/s3/store",
      "credentials": "document-output",
      "path": "documents/${fields.document_id}/${fields.output_version}/ocr.json",
      "acl": "bucket-default",
      "result": true
    }
  }
}