Key takeaways
- Direct analysis is simplest when sources are already small, valid, and consistently encoded.
- Preprocessing controls orientation, dimensions, format, and transfer cost before model inference.
- Alt text should communicate purpose in context, not list every detected object.
Image description can serve alt text, search, moderation, or catalog enrichment, but those tasks need different prompts and levels of detail. The safest architecture separates deterministic file preparation from probabilistic interpretation.
What matters most
- Store model, prompt version, confidence or review state, and the source relationship.
Define what a description must accomplish
Image description is not one task. Accessibility text explains the image's purpose in a particular page, catalog metadata supports filtering and retrieval, moderation looks for policy categories, and an internal inventory records visible facts. Before selecting a model, define the audience, maximum length, required fields, forbidden inferences, and what should happen when evidence is ambiguous.
A single generated sentence should not serve every consumer. An ecommerce search index may need object labels, colors, and product type, while alt text may only need to say what the image contributes beside its caption. Moderation needs policy-specific categories and review states rather than polished prose. Separate these outputs so a later prompt or policy change does not silently alter unrelated product behavior.
Accessibility
Describe the information or function the image contributes in its surrounding context.
Search
Capture normalized, factual terms that improve retrieval without turning uncertain guesses into facts.
Moderation
Return policy categories, evidence, and a review route rather than an automatic final judgment.
Choose labels or contextual language
Use Transloadit's /image/describe Robot when object labels are the desired output. It can return a flat list or a fuller response with confidence values, and results can be written as JSON or placed in file metadata for later Steps. It can also request explicit-content descriptions, but provider categories differ and should be mapped into your own moderation policy.
Use an image-capable model through /ai/chat when the output requires context, relationships, or a task-specific natural-language response. Supply a focused instruction and a JSON Schema when downstream code needs structured fields. Model support and behavior can evolve, so record the selected model, prompt version, and source derivative. Do not treat a valid JSON response as proof that its claims are correct.
Prepare a bounded analysis derivative
Deterministic preprocessing makes model inputs more consistent. Correct orientation, reject invalid files, convert uncommon encodings, and resize within a documented pixel budget. A smaller derivative reduces transfer time and inference cost, while a fixed format makes failures easier to reproduce. Keep the original file linked to the derivative so analysis can be repeated when requirements change.
Aggressive preprocessing can remove the evidence the model needs. Small signs, product labels, diagrams, and distant people may become unreadable after downscaling. Cropping can exclude context that changes the meaning of a scene. Test multiple sizes on representative images and preserve aspect ratio unless the task explicitly requires a crop. Never overwrite the source with the analysis rendition.
Orientation
Apply camera orientation consistently so people, text, and objects are presented upright.
Resolution
Set a maximum that controls cost while retaining the smallest important details in the evaluation set.
Color and transparency
Check whether flattening transparency or converting color spaces changes meaningful content.
Write prompts that separate evidence from inference
Ask for observable details first, then optional interpretations in separate fields. For example, a catalog schema might include visible_objects, visible_text, uncertainties, and suggested_alt_text. Tell the model to omit identity, medical status, emotion, and protected traits unless the task has a legitimate basis and an approved review process. Include length and vocabulary constraints that match the destination.
Treat filenames, nearby page text, EXIF fields, and user-provided captions as untrusted context. They may be inaccurate or contain prompt-like instructions. Label contextual data clearly and tell the model that it cannot override the task instructions or visible evidence. If page context is required for alt text, provide only the relevant heading, caption, and link purpose instead of an entire untrusted document.
Write useful alt text, including no text
Good alt text communicates the image's function without narrating every pixel. It normally avoids filler such as “image of,” does not repeat a nearby caption, and includes visible words only when those words matter. A linked logo may need the organization's name and link purpose, while a chart requires its main conclusion or an adjacent data table rather than a list of shapes and colors.
Decorative images should usually use empty alternative text so screen readers skip them. An AI system cannot reliably decide decoration from pixels because the decision depends on page context. Make decoration an explicit content-authoring choice. Complex images, screenshots, maps, and diagrams may need a short alt attribute plus a longer explanation maintained by a person.
Functional images
Describe the action or destination when an image behaves as a button or link.
Text in images
Include meaningful visible text and provide equivalent content elsewhere when it is lengthy.
Decorative images
Use empty alt text rather than generating a description that adds noise.
Handle hallucinations and sensitive content
Vision models can invent text, objects, identities, relationships, and intentions. Confidence values from one model are not universal probabilities, and fluent wording can hide uncertainty. Store uncertain observations separately and send high-impact results to review. Never use a generated description alone to make decisions about employment, eligibility, health, identity, or law enforcement.
Descriptions of people need especially conservative rules. Avoid guessing names, ethnicity, gender identity, disability, health, religion, or emotional state from appearance. Face detection is not identification, and a scene that resembles an event does not prove what happened. Provide a correction path for people affected by published metadata and remove sensitive analysis when its retention is no longer justified.
Evaluate quality by task and content segment
Build an evaluation set from the images the product actually receives. Include low light, rotated files, screenshots, illustrations, dense text, ambiguous scenes, multiple languages, sensitive subjects, and decorative assets. Have qualified reviewers score factual accuracy, missing critical details, unsupported claims, readability, policy compliance, and usefulness in context. Measure each content segment separately so strong results on simple photos do not hide failures on diagrams.
Avoid tests that expect one exact sentence from a probabilistic model. Instead, test schema validity, prohibited claims, required concepts, length, and human ratings. Pin prompt and application versions, record the model selected for every result, and rerun the evaluation before changing models or preprocessing. Conduct screen-reader reviews on real pages because an accurate description can still create a poor experience when repeated or placed incorrectly.
Operate a reviewable production workflow
Run analysis asynchronously with clear deadlines and bounded retries. Store the source identifier, derivative checksum, task type, model, prompt version, raw structured result, reviewer decision, and publication state. Cache by the source checksum and analysis configuration so the same image is not repeatedly sent for inference. Invalidate the cache when the source, model, prompt, or policy changes.
Monitor latency, inference cost, schema failures, refusal rates, reviewer corrections, and drift by image category. A timeout should leave the source available and mark description generation as pending or unavailable, not publish an empty guess. Protect private images with least-privilege storage access, short retention for temporary derivatives, redacted logs, and explicit controls over which provider may receive sensitive inputs.
Failure fallback
Queue a retry or human description instead of presenting unreviewed placeholder text as fact.
Cost control
Resize appropriately, cache results, cap retries, and reserve expensive contextual models for tasks that need them.
Provenance
Keep machine-generated, reviewed, edited, and published states distinct so changes remain auditable.
Technical details worth knowing
- A factual inventory, an accessibility alt description, and marketing copy are different outputs. The same image should be described differently according to context and audience.
- Vision models can infer nonexistent text, objects, identity, or relationships. High-impact metadata should preserve confidence and allow review instead of becoming unquestioned fact.
- Metadata outside the pixels, including filenames and nearby text, can improve context but can also inject misleading instructions and should not override visible evidence automatically.
- Useful alt text normally omits phrases such as “image of,” avoids speculation, and includes text visible in the image when that text matters to the page.
- A decorative image should usually have empty alternative text rather than an AI-generated description that creates noise for screen-reader users.
- Descriptions of people require special caution around identity, health, emotion, protected traits, and other sensitive inferences that pixels do not establish reliably.
A practical approach
- 1
Define whether the output supports accessibility, search, moderation, or internal analysis.
- 2
Prepare a bounded derivative without destroying details required by the task.
- 3
Evaluate prompts and models on representative content, including ambiguous and sensitive images.
- 4
Require review where a wrong description could mislead, offend, or block access.
When Transloadit is useful
Use /image/describe when object labels and confidence scores are enough. For a contextual natural-language description, pass a bounded image to an image-capable /ai/chat model with a task-specific prompt and output schema. Preserve the source, model, and review state either way.
Architecture boundary
Both /image/describe and image-capable /ai/chat models are probabilistic, and their output can change as providers evolve. Generated descriptions need accessibility and factual review before publication.
Frequently asked questions
Should AI-generated captions be used directly as alt text?
Not by default. Alt text depends on page purpose and surrounding content, so generated text should pass contextual rules and human review where an error would impair access or mislead users.
When should I use `/image/describe` instead of `/ai/chat`?
Use /image/describe for object labels, confidence-bearing label output, or supported explicit-content categories. Use an image-capable /ai/chat model when you need contextual prose or a task-specific structured interpretation.
How much should an image be resized before analysis?
There is no universal size. Choose the smallest derivative that preserves the details required by the task, then validate it on text-heavy, distant, and ambiguous examples from your own content.
Can a confidence score eliminate human review?
No. Confidence is model-specific and may be poorly calibrated for your inputs. Use thresholds to route work, but retain review for sensitive, consequential, or uncertain results.
What should happen for a decorative image?
The authoring system should mark it as decorative and provide empty alternative text. Generating a description would add unnecessary screen-reader output.