Batch process and migrate large media libraries
Import hundreds of gigabytes or terabytes from existing cloud storage, run one automated workflow across many files, and migrate the results back.
Read large media libraries from S3, Google Cloud, Azure, SFTP, HTTP, and other supported sources.
Submit Assembly-sized batches while Transloadit schedules concurrent jobs across a fleet that can scale to 1,500 machines.
Export processed files to the original storage location or move them to a new provider and prefix.
Build momentum
Give your existing library a new lease on life.
Roll out new formats and workflows across the files you already have.
Launch new formats sooner
Prepare your existing library for the devices your audience uses.
Make old content discoverable
Add transcripts, previews, and metadata to files already in storage.
Move your migration forward
Import, process, and export files without manual download cycles.
Keep releases consistent
Apply one saved Template across the files in your batch.
Free up engineering time
Let managed workers handle processing while your team ships features.
Plan for bigger libraries
Plan your batch capacity before large backfills and seasonal peaks.
DSC_08452-1200w.jpg420 KB
contract-p1.png180 KB
keynote-720p.mp488 MB
quarterly-report.pdf310 KBFrom hundreds of gigabytes to terabyte-scale migrations
Use Transloadit as a managed file processing API for a one-time cloud storage migration, a large media backfill, or recurring bulk file processing. Import from your existing storage, transform each file, and export the results without moving the entire library through your application servers.
Inventory the source
Choose the buckets, prefixes, or object paths to migrate and estimate the total files and bytes.
Paginate into Assemblies
Paginate large storage prefixes into jobs of roughly 500–1,000 files and submit them at a controlled rate.
Process concurrently
Apply one saved Template across the migration while Batch Job Slots govern parallel processing in each region.
Export and reconcile
Write results to the same or a new storage destination, then verify each Assembly before marking files complete.
Your application
- Inventory
- Paginate
- Submit
- Assemblies submitted
Transloadit
- Import
- Schedule
- Transform
- Export
- Report
- Results returned
Your application
- Record IDs
- Approve
- Publish
You own the ends, we own the middle
Your application keeps the parts that depend on your data: choosing what to migrate, submitting it in Assembly-sized batches at a rate you control, and recording what came back. That is an inventory loop and an idempotent callback handler.
Everything between those two points is ours: importing the bytes, scheduling across the fleet and its queues, running the same Template over every file, exporting the results, and accounting for errors per Assembly. See how queues and concurrent processing work.
Your workflow. Your controls.
Batch processing without the pain
Stop running one-off scripts for every backfill. Reuse a workflow you can inspect.
Start small, then scale up
Test your Template on a small sample, then process millions of files with Transloadit.
Import from your storage
Read existing files from supported buckets and remote storage providers.
Reuse your processing logic
Run the same Template for new uploads and existing files.
Get more from each file
Create multiple formats, thumbnails, and metadata in one workflow.
Keep track of each Assembly
Inspect processing status and errors to decide what needs another run.
Know the moment files are ready
Use per-file callbacks and webhook requests when processing finishes.
Make room for busy days
Batch Job Slots keep work moving. Extra jobs queue when your slots are busy.
Bring your own storage
Send results to Amazon S3, Cloudflare R2, Google Cloud Storage, Azure, and more.
Keep spending in check
Set a soft bill limit for alerts or a hard limit to stop processing.
Start a large-scale batch processing migration
Frequently asked questions
What can I do with a batch of existing files?
Re-encode videos, resize images, transcribe audio, extract metadata, or apply another saved workflow to your library. Start with a Template that produces the results you need. Explore available processing Robots.
Where can my source files live?
Use supported import Robots to read from Amazon S3, Cloudflare R2, Google Cloud Storage, Azure, and other sources. The import options determine how files are selected. Find your storage integration.
Should I process the whole library at once?
Start with a representative sample so you can check quality, output paths, and usage. Then expand the batch with the same Template. Large libraries may need multiple Assemblies rather than one oversized job. Read about Assemblies.
What happens when all my Batch Job Slots are busy?
Batch imports use the Batch Queue and your plan’s Batch Job Slots. Extra jobs wait until a slot is available. Contact us about batch capacity before a large backfill. Learn how queues and slots work.
How do I find failed jobs or know when processing finishes?
Use Assembly status responses and notifications to track completion and errors. Your application can decide which failed Assemblies need to be retried. Read the webhook guide.
Can I export results to a different storage provider?
Yes. Combine the source provider’s import Robot with the destination provider’s export Robot. Choose output paths deliberately and verify the results before removing originals. See how to save processed files.
How can I estimate and control costs?
Rates vary by operation and usage. Set a soft bill limit for alerts or a hard limit to stop processing. For larger workloads, ask about volume pricing and batch capacity. Estimate your workflow costs.