Turn any document into LLM-ready Markdown
Language models read Markdown well. A heading shows where a topic starts, a list stays a list, and a paragraph is one thought. The documents your users upload look nothing like that. A PDF repeats a header and a page number on every page and breaks sentences at the end of each line. A scan has no text at all. Feed that to a model as it is, and every chunk in your RAG index carries the noise: you pay for it in tokens and in worse answers.
Transloadit now turns those documents into clean Markdown. PDFs, scans, photos of a page, Word and PowerPoint files: one command, per document or per page.
One command
First, authenticate the Transloadit CLI. Then run:
npx -y @transloadit/node assemblies create \
--template builtin/extract-text \
--field text_format=md \
-i report.pdf -o out/
builtin/extract-text is a ready-made Builtin Template around
🤖 /document/extract. The Markdown lands in out/report-text.md.
We ran it on NASA’s Artemis II press kit, 27 pages of mission overview, timelines and crew information. It came back in about five seconds. This is the lower half of page 3:
And this is what the command returned for it:
### Mission Priorities
The Artemis II test flight will confirm the systems necessary to support astronauts in deep space exploration and prepare to establish a sustained presence on the Moon. There are five main priorities for Artemis II:
1. Crew: Demonstrate the ability of systems and teams to sustain the flight crew in the flight environment, and through their return to Earth.
2. Systems: Demonstrate systems and operations essential to a crewed lunar campaign. This ranges from ground systems to hardware in space, and operations spanning from development to launch, flight, and recovery.
3. Hardware and Data: Retrieve flight hardware and data, assessing performance for future missions.
4. Emergency Operations: Demonstrate emergency system capabilities and validate associated operations to the extent practical, such as abort operations and rescue procedures, as needed.
5. Data and Subsystems: Complete additional objectives to verify subsystems and validate data.
A few things happened on the way:
- Headings are headings. Text that is clearly larger than the body text becomes a Markdown heading, and the level follows the size.
- Paragraphs are whole again. The PDF stores each printed line separately. The Markdown has one line per paragraph, so a sentence is never cut in half.
- Lists stay lists. The five priorities come back as a numbered list.
- The page furniture is gone. The footer “Artemis II Press Kit” and the page number repeat on every page of the PDF. They are not in the Markdown.
Bold and italic inside a paragraph are not carried over: the bold labels in the list come back as plain text.
Markdown is one of four formats. text_format also takes txt, html and json.
One file per page, for RAG
For retrieval you usually want smaller pieces, and a page number to cite. Add two fields:
npx -y @transloadit/node assemblies create \
--template builtin/extract-text \
--field text_format=md \
--field text_granularity=page \
--field text_method=native \
-i report.pdf -o out/
Now every page is its own file:
$ ls -v out/report/native_text/ | head -n 3
report-page-1.md
report-page-2.md
report-page-3.md
$ ls out/report/native_text/ | wc -l
27
Heading levels are measured across the whole document, so a section heading has the same level on
page 3 as on page 20. text_method=native tells the Robot to read the text that is stored in the
PDF. Per-page output requires it. For scanned PDFs there is a per-page pipeline further down.
Scans and photos
A scan is a picture of text. Give the same command a JPEG, a PNG or a PDF without any text in it, and Transloadit runs OCR for you:
npx -y @transloadit/node assemblies create \
--template builtin/extract-text \
--field text_format=md \
-i scan.jpg -o out/
We tried it on a typewritten page from NASA’s 1969 press kit for Apollo 11:
APOLLO 11
The United States will launch a three-man spacecraft toward the Moon on July 16 with the goal of landing two astronaut-explorers on the lunar surface four days later.
If the mission--called Apollo 11 is successful, man will accomplish his long-time dream of walking on another celestial body.
OCR output comes back as paragraphs: lines that belong together are joined. It is not perfect: look closely and you will see that the second dash after “Apollo 11” went missing.
Some scanned PDFs do contain text, left behind by an older OCR pass, and it is often poor. Add
--field text_method=ocr to ignore that text and read the pages again.
Word and PowerPoint
Office files work the same way. Pass a .docx or a .pptx, and the Template first converts it to
PDF with 🤖 /document/convert, then extracts the Markdown:
npx -y @transloadit/node assemblies create \
--template builtin/extract-text \
--field text_format=md \
-i countdown.docx -o out/
NASA’s explainer of the Artemis I launch countdown, a Word file, came back as clean paragraphs in under five seconds. A PowerPoint deck of 42 slides and 320 MB took 27 seconds. From a deck you get the text on the slides, with the titles as headings. Pictures and diagrams are not described.
Bigger pipelines
builtin/extract-text covers the common cases. When you need more control, write the Steps
yourself. This pipeline gives you one Markdown file per page of a scanned PDF:
🤖 /document/split cuts the PDF into single pages, and
/document/extract runs OCR on each of them. Save it as scan-pages.json:
{
"steps": {
":original": {
"robot": "/upload/handle"
},
"pages": {
"use": ":original",
"robot": "/document/split",
"split": true
},
"markdown": {
"use": "pages",
"robot": "/document/extract",
"extract": ["text"],
"text_method": "ocr",
"ocr_provider": "aws",
"text_format": "md",
"result": true
}
}
}
Then run it:
npx -y @transloadit/node assemblies create --steps scan-pages.json -i scans.pdf -o out/
Four scanned pages took ten seconds:
$ find out -type f | sort
out/scans/markdown/scans_page_1-text.md
out/scans/markdown/scans_page_2-text.md
out/scans/markdown/scans_page_3-text.md
out/scans/markdown/scans_page_4-text.md
From here it is ordinary Assembly work. Pick pages with the pages parameter of /document/split,
combine several PDFs into one first with 🤖 /document/merge, or add
a storage Robot so the Markdown lands in your own bucket, next to the
upload it came from.
Get started
Run the first command with a document of your own, or try the Steps in the Template editor. The 🤖 /document/extract documentation lists every parameter.
The example documents are NASA’s Artemis II press kit and Apollo 11 press kit. Using them here does not mean that NASA endorses Transloadit.
