AI and generated media # Calibrating AI moderation thresholds, review, and appeals Design AI moderation as a policy and review system, not a single model score that silently publishes or deletes user media. Published August 13, 2026 ## Key takeaways * Separate model categories, platform policy, and enforcement actions so each can be reviewed and changed independently. * Evaluate false positives and false negatives by policy class and user segment rather than quoting one global accuracy score. * Use reversible quarantine and appeal paths for consequential decisions instead of irreversible model-only enforcement. AI moderation can triage volume, but every threshold encodes a policy tradeoff. The system must separate provider categories, contextual rules, reviewer judgment, user notification, and appeal rather than hiding them behind “safe” or “unsafe.” ## In this guide 1. [Separate provider signals from platform policy](#ai-moderation-systems-section-1) 2. [Preserve only the evidence a policy requires](#ai-moderation-systems-section-2) 3. [Calibrate thresholds for each action and category](#ai-moderation-systems-section-3) 4. [Collect explicit-content signals before policy routing](#ai-moderation-systems-section-4) 5. [Count reversals, misses, and reviewer disagreement](#ai-moderation-systems-section-5) 6. [Make enforcement reversible and appealable](#ai-moderation-systems-section-6) 7. [Version policy independently from model settings](#ai-moderation-systems-section-7) ## Separate provider signals from platform policy This guide focuses on calibrating thresholds and review capacity; the broader moderation-workflow guide covers intake architecture. Start with a versioned policy that names categories, surfaces, audiences, and available actions. Provider labels are observations that policy may use, not prewritten enforcement decisions. Normalize provider categories into an application vocabulary without discarding the raw response. Keep the signal, policy evaluation, reviewer decision, enforcement action, and appeal as linked records. That separation allows a model or policy change to be analyzed without rewriting what originally happened. ## Preserve only the evidence a policy requires Create a reviewed calibration set that represents languages, media types, user groups, common benign lookalikes, and severe rare cases. Restrict access because the set may contain sensitive material. Define retention and reviewer protections before collecting more examples than the policy genuinely needs. Record source identity, applicable policy, provider and model, input derivative, category scores, reviewer label, and disagreement. Avoid copying raw media into logs, alerts, or queue payloads. Evidence must be sufficient for correction and appeal without becoming an uncontrolled secondary content store. ## Calibrate thresholds for each action and category Plot precision and recall or equivalent error tradeoffs per category, then choose separate boundaries for automatic allow, human review, and any action eligible for automatic restriction. A wide review band reduces model-only decisions but increases queue volume and delay; capacity is therefore part of threshold design. Weight false positives and false negatives by consequence rather than optimizing one accuracy number. High-impact account penalties need stronger evidence and authorization than holding one upload. Recalculate expected review volume on realistic traffic before release so an apparently safe threshold does not create an unusable backlog. ## Collect explicit-content signals before policy routing For supported images, `/image/describe` with `explicit_descriptions: true` returns only explicit descriptions; `granularity: "full"` includes confidence values. Run a separate ordinary-description pass if the application needs non-explicit labels. The Assembly result still requires application-owned normalization and policy evaluation. Use `/file/filter` only for transparent routing under configured conditions, while the application owns quarantine, review, appeal, and publication. Transloadit does not supply a universal moderation policy or specialist coverage for every medium and category. Keep unsupported signals in explicit pending or unavailable states. Collect explicit-content signals with confidence values ``` { "steps": { ":original": { "robot": "/upload/handle" }, "moderation_signals": { "use": ":original", "robot": "/image/describe", "explicit_descriptions": true, "format": "meta", "granularity": "full", "provider": "aws" } } } ``` ## Count reversals, misses, and reviewer disagreement Evaluate thresholds on a frozen holdout set before release, then monitor reviewer reversals, appeals, sampled allowed content, score distributions, and known incidents. Segment results by category, surface, language, media type, and user population so common easy cases cannot hide a damaging subgroup failure. Reviewer disagreement is evidence of ambiguity, not noise to erase. Maintain adjudication guidance and record when policy language, available context, or reviewer wellbeing limits certainty. Appeal outcomes are useful but selection-biased because only some affected users appeal; do not treat them as a random sample. ## Make enforcement reversible and appealable Quarantine or defer according to the documented surface policy when a required classifier is unavailable. Never let a timeout accidentally choose fail-open or permanent deletion. Automatic actions should be bounded, idempotent, and reversible, with stronger authorization for consequences beyond one media item. Notify affected users with a sanitized reason and provide an appeal path where policy or law requires it. An appeal appends a new decision linked to the original evidence and applicable policy; it must not erase history. Reviewers need sufficient context without unnecessary exposure to sensitive content. ## Version policy independently from model settings Store policy version, provider configuration, category mapping, thresholds, review guidance, and enforcement code as separate but linked release inputs. Compare old and new thresholds in shadow mode before changing actions. A provider update can shift score distributions even when category names and schemas stay stable. Alert on score shifts, review-band volume, queue age, reversal, appeal, provider error, and cost. Keep the prior decision path available during rollback and identify which outcomes require re-evaluation after a material policy or model change. Thresholds should not move solely to make backlog charts look better. ## Technical details worth knowing * Task boundary: AI moderation estimates whether media may match policy-relevant categories so a platform can prioritize review or apply bounded rules. A model estimates categories from content; moderation policy decides what is allowed in context, and enforcement applies a reversible product action. * Input contract: Send only content required by the policy, preserve source evidence securely, and keep user text or filenames from overriding moderation instructions. Input preparation must be evaluated with the model because preprocessing can remove evidence as well as noise. * Output contract: Return provider observations, normalized policy categories, evidence references, threshold version, proposed action, and human review state. A valid response does not prove that the recommendation is authorized, useful, or safe to execute. * Method choice: Combine narrow classifiers with deterministic rules and trained review, reserving automatic actions for categories and thresholds validated on the platform’s own risk profile. Model names alone do not describe the training data, thresholds, latency, licensing, or failure behavior of a deployed system. * Evaluation: Measure false positives and false negatives per policy class, language, content segment, and severity, then include appeal outcomes and reviewer disagreement. Aggregate scores should be segmented by content type so common easy examples do not hide failures on important edge cases. * Failure and safety: Model or provider failure should quarantine or defer risky content according to policy, not default invisibly to publish or permanent deletion. Protect reviewers, limit retention and access to sensitive material, avoid identity inference, and provide notice and appeal where users are affected. /image/describe uses AWS or GCP, so evaluate provider egress, region, and retention before sending sensitive uploads. * Operations: Version policy separately from model configuration, audit enforcement and appeals, monitor segment drift, and preserve reversible states for corrected decisions. ## A practical approach 1. 1\ Write the decision, output schema, and rejection criteria for a moderation decision pipeline. 2. 2\ Build a representative a moderation decision pipeline evaluation set and preserve each source, preprocessing choice, and provenance record. 3. 3\ Benchmark the complete workflow on representative evidence and compare the result with predefined task-specific acceptance criteria. 4. 4\ Release a moderation decision pipeline behind explicit review and fallback paths, then monitor the operating signals that determine whether it remains useful. A four-stage media workflow ## When Transloadit is useful Use /image/describe with explicit\_descriptions set to true for documented explicit categories and granularity set to full for confidence values. That pass returns only explicit descriptions, so run a separate ordinary-label pass if the application needs both. Use /file/filter for transparent routing while the application owns review and appeal. ## Architecture boundary Transloadit can expose supported explicit-content descriptions through /image/describe and can route files with /file/filter, but it does not define a platform’s policy or make a universally reliable moderation decision. ## Frequently asked questions ### Should one confidence threshold control every moderation action? No. Categories and consequences have different error costs. Define separate allow, review, and eligible restriction boundaries from representative evidence and available review capacity. ### Does `explicit_descriptions: true` return ordinary labels too? No. It returns only explicit descriptions. Run a separate non-explicit description pass when the application needs ordinary image labels as well. ### Can an unavailable classifier default to publishing? Only if the documented policy for that surface explicitly accepts that exposure risk. Make fail-open, fail-closed, delay, or review behavior deliberate rather than letting timeout choose it. ### Are appeal outcomes a sufficient moderation evaluation set? No. Appeals reveal important errors but are selection-biased. Combine them with reviewed holdouts, sampled outcomes, reversals, incidents, and segmented score distributions. ## Build the workflow Move from the concept to a tested Assembly with Robot documentation and working demos. ### Relevant Robots * [/image/describe](/docs/robots/image-describe.md) * [/file/filter](/docs/robots/file-filter.md) * [Read the API documentation](/docs.md) * [Explore working demos](/demos.md) * [Create a free workspace](/c/signup/) AI and generated media ## Continue with related guides * [AI content moderation in an upload workflow](/guides/ai-content-moderation-workflows.md)\ Place AI moderation inside a controlled upload workflow with confidence thresholds and human review. * [Automated content moderation: architecture and failure handling](/guides/automated-content-moderation.md)\ Build automated moderation as a layered system of file checks, classifiers, policy decisions, and review queues. * [How to give AI agents safe media-processing capabilities](/guides/transloadit-for-ai-agents.md)\ Give AI agents constrained media capabilities through stable Templates, structured inputs, and observable Assembly results. * [Two ways to describe images with AI](/guides/describe-images-with-ai.md)\ Compare direct vision-model descriptions with a preprocessing pipeline that creates cleaner, bounded analysis inputs. * [Creating 3D models from images: a practical pipeline](/guides/create-3d-models-from-images.md)\ Plan a photo-to-3D capture and processing workflow without confusing image preparation with 3D reconstruction. * [AI image analysis: tasks, architecture, and safeguards](/guides/ai-image-analysis.md)\ Understand classification, detection, OCR, captioning, embeddings, and moderation as different image-analysis tasks.