How to Write Context-Specific Moderation Checks for Shieldstral
Turn moderation requirements into focused questions, supply the relevant content, and use Shieldstral’s confidence scores to rank or threshold results.
한 줄 답
How should developers structure content-moderation checks with Mistral AI’s Shieldstral?
For each Shieldstral moderation check, state the evaluation context and strictness, ask one yes-or-no policy question, and provide the content to assess. Keep separate policies as separate questions. Use the resulting yes/no probability as a continuous safety score that can be thresholded or used to rank cases by confidence.
핵심 요약
- Express each moderation policy as a single yes-or-no question instead of relying on a fixed taxonomy.
- Put the deployment context, desired strictness, and any definition of unsafe content in the instruction.
- Supply a prompt, response, prompt-response pair, or image with optional text as the document being evaluated.
- Use Shieldstral’s continuous yes/no probability to threshold decisions or rank cases by confidence.
- Shieldstral applies to text, images, and combined text-and-image inputs without retraining for each new policy.
A fixed list of harm categories can become a poor fit when an AI product changes audience or purpose. Content that is acceptable in a cybersecurity research tool may be harmful in a mental-health platform. Shieldstral addresses that problem by letting developers provide the policy as plain-language input when the model runs.
This guidance applies specifically to Mistral AI’s Shieldstral, not to Le Chat generally. Shieldstral is a 3B open-weights safety classifier released under the Apache 2.0 license. It evaluates text, images, and combined text-and-image material through one question-answering format.
Build every check from three parts
A Shieldstral request consists of an instruction, a query, and a document. Each part has a distinct job.
<Instruct>establishes the evaluation context and strictness. It can also define what should count as unsafe.<Query>asks one yes-or-no question about the policy being tested.<Document>contains the material to judge.
The document can be a user prompt, an assistant response, a prompt-response pair, or an image with optional text. Match that choice to the actual moderation task. Use a prompt when screening an incoming request, a response when checking generated output, and a prompt-response pair when the relationship between the two matters.
Turn policies into narrow questions
Write the query so that a yes answer has one clear meaning. The source gives examples such as asking whether content promotes physical violence, whether an image is safe for a minor, or whether an assistant refused a request.
This structure is useful when an application needs several kinds of checks. Treat each policy boundary as its own question rather than compressing unrelated concerns into one broad query. A focused question makes the returned probability easier to interpret because it corresponds to a specific condition.
Put contextual details in the instruction. If a product needs strict handling for adversarial jailbreaks but more lenient treatment for response-quality review, state the appropriate strictness for the check being run. If a term such as unsafe could be ambiguous, provide its relevant definition there as well.
Because the policy lives in the request, it can be changed for a new deployment context without retraining Shieldstral. That makes it possible to reuse one checkpoint while adjusting questions and instructions for different applications or audiences.
Use the score as more than a label
At inference time, Shieldstral reads the logits for yes and no and applies softmax normalization to produce a continuous safety score. The verdict comes from a single token and a single forward pass.
That score supports two practical patterns. A system can apply a threshold when it needs a decision boundary, or rank evaluated items by confidence when it needs to prioritize results. The source does not prescribe a universal threshold, so the supplied material does not support treating one cutoff as correct for every product.
The same interface can cover prompt classification, response moderation, refusal detection, and toxicity detection. Keep the question aligned with the task. A refusal check should ask whether the assistant refused; a content-safety check should ask about the specific unsafe property.
Keep modality and context together
For image moderation, provide the image as the document and include optional text when it contributes to the evaluation. Shieldstral uses the same natural-language policy interface for text, images, and text-plus-image inputs, so the policy does not need to be converted into a separate fixed image taxonomy.
For conversational review, use a prompt-response pair when evaluating only one side would remove necessary context. For simple input screening or output screening, provide the relevant prompt or response alone. This keeps the document closely matched to the decision the query asks the model to make.
Check the publication date and current Shieldstral documentation before implementation, because vendor features change.
자주 묻는 질문
- Which product does this guidance apply to?
- It applies to Mistral AI’s Shieldstral, a 3B open-weights multimodal safety classifier. The source does not describe this workflow as a Le Chat feature.
- Does Shieldstral require retraining when a moderation policy changes?
- No. Policies are supplied in plain language at inference time, allowing the same checkpoint to be retargeted without retraining.
- What content can Shieldstral evaluate?
- The document can be a prompt, a response, a prompt-response pair, or an image with optional text.
- Does Shieldstral return only a safe or unsafe label?
- It reads the yes and no logits and normalizes them into a continuous probability. That score can support thresholds or confidence-based ranking.
출처
- Introducing Shieldstral. | Mistral AI — Mistral AI