How to extract text from documents and images at scale

Route each input to the Robot that accepts it

Text extraction fails most often at the routing step rather than at recognition. /document/ocr currently accepts PDFs only, so a Word file, a PowerPoint deck, or a TIFF sent straight to it will not produce text. Convert those with /document/convert first, and send photographs and screenshots to /image/ocr instead. A single /file/filter Step in front of the pipeline is usually enough to split the traffic by type.

Both Robots take the same three parameters. provider selects between AWS and GCP, and defaults to automatic selection. granularity chooses between one block of text and a positioned list of fragments. format decides whether the text comes back as a file or is attached to the file object for later Steps to read.

{
  "steps": {
    ":original": {
      "robot": "/upload/handle"
    },
    "pdfs": {
      "use": ":original",
      "robot": "/file/filter",
      "accepts": [
        ["${file.mime}", "regex", "application/pdf"]
      ]
    },
    "photos": {
      "use": ":original",
      "robot": "/file/filter",
      "accepts": [
        ["${file.mime}", "regex", "^image/"]
      ]
    },
    "deskewed": {
      "use": "pdfs",
      "robot": "/document/autorotate"
    },
    "pdf_text": {
      "use": "deskewed",
      "robot": "/document/ocr",
      "format": "json",
      "granularity": "full"
    },
    "photo_text": {
      "use": "photos",
      "robot": "/image/ocr",
      "format": "json",
      "granularity": "full"
    }
  }
}

Choose the output shape before you choose the provider

The format parameter has more effect on the surrounding pipeline than any other choice. json and text both return a file, which suits archiving and indexing. meta returns no file at all and instead stores the strings on the file object under ${file.meta.recognized_text}, where later Steps can read them. That is what makes it possible to filter on recognised content, or to burn extracted text into an image, without running recognition twice.

granularity follows from the same question. full returns one block, which is what a search index wants. list returns positioned fragments, which is what you need to redact a region, pull a specific field, or highlight a match in a viewer. Choosing list when you only need a search index costs nothing but produces output that is more work to consume.

{
  "steps": {
    ":original": {
      "robot": "/upload/handle"
    },
    "recognized": {
      "use": ":original",
      "robot": "/document/ocr",
      "format": "meta",
      "granularity": "full",
      "provider": "gcp"
    },
    "only_invoices": {
      "use": "recognized",
      "robot": "/file/filter",
      "accepts": [
        ["${file.meta.recognized_text}", "regex", "(?i)invoice"]
      ]
    }
  }
}

Prepare the input rather than switching providers

When recognition quality disappoints, the reflex is to try the other provider. Input preparation usually pays better. Scans arrive rotated, skewed, and unevenly lit, and every one of those costs accuracy before the model sees the page. Running /document/autorotate ahead of recognition corrects orientation, and normalising contrast on photographed pages removes a common source of dropped lines.

Cost behaves the same way. OCR carries a minimum charge of one megabyte per file, so a hundred single-page scans submitted individually cost considerably more than the same hundred pages batched through one Assembly. Batch where the workload allows it, and keep interactive recognition separate from bulk backfills so a large import cannot delay a user waiting on a single upload.

Treat recognised text as evidence, not as a record

Both Robots call third-party AI services, and those providers retrain their models. The same PDF can return slightly different text months later. That is manageable as long as the system does not assume otherwise: store the recognised text next to the source file along with the provider and the date it was produced, so a later difference is visible rather than silent.

The same property makes exact-match assertions a poor choice in tests. Assert that a known phrase appears, or that a field parses, rather than comparing a full transcript. Where a value has legal or financial consequence, keep the source document as the record and treat the extracted text as an index into it.

Pull individual fields instead of whole transcripts

Plenty of workloads do not want the text of a document at all. They want an invoice number, a date, a total, or the region of a page that must be blacked out before the file is shared. That is where granularity: "list" earns its cost: it returns fragments with positions, so a downstream Step can select by location rather than by parsing a single undifferentiated block.

Anchoring on layout alone is fragile, because a supplier redesigns a form and every coordinate moves. Anchoring on a nearby label and using position only to disambiguate between repeated matches survives that redesign. Where the document set is genuinely unpredictable, passing the recognised text to a model with a strict output schema is more robust than an expanding collection of regular expressions.

Decide what happens when a page cannot be read

Every archive contains pages that recognition cannot handle: a photograph of a screen, a fax of a fax, handwriting in a margin. Returning empty text for those is correct behaviour, and the pipeline needs somewhere for them to go. Treating an empty or very short result as a routing signal rather than as a failure keeps the batch moving and puts the exceptions in front of a person.

Retrying the same file against the same provider rarely helps, because nothing about the input changed. Retrying after a preparation Step such as autorotation sometimes does. Recording which files produced nothing, and how many, turns an invisible quality problem into a number that can be watched over time.

Frequently asked questions