Model Risk Management and Red Team

Gains:

• Ability to classify usage scenarios into low/medium/high risk levels according to impact
• Ability to systematically test the model before production with red-teaming
• Ability to make production decisions with model card and acceptance door (go/no-go)

Not every use of AI carries the same risk. An assistant summarizing a meeting note and an assistant evaluating a loan application produce very different results. The basis of corporate governance is to classify uses according to risk level and apply appropriate control to each level. In this unit, we will learn the framework of model risk management (the discipline of managing the risk caused by a model being incorrect, biased or exploitable), how to test the model before production with red-teaming, and the model card and acceptance criteria.

Classification by Risk

The first step is always the same: "What happens if this usage goes wrong?" Three rough levels according to potency and reversibility:

• Low risk: Error is easily detected and undone; No personal/financial consequences. Example: internal meeting summary, draft idea generation.
• Medium risk: The error affects the business process but passes through the human eye. Example: draft response to customer, preliminary report summary.
• High risk: Decision directly affects a person/money, difficult to reverse. Example: credit/insurance decision, healthcare triage, employment screening.

Control intensity increases with the level of risk: at low risk, light controls are sufficient; At high risk, human supervision, strict verification, red teaming and constant monitoring are mandatory.

Attention: Make risk classification according to the effect of the use, not its name. The so-called "just a chatbot" system is high risk if it can initiate payments.

Red Team (Red-Teaming)

Red teaming is deliberately trying to break a system by pretending to be a malicious attacker. This is in AI; It includes jailbreaking (bypassing the model's security rules), prompt injection, data exfiltration, generating biased/malicious output, and testing edge scenarios. The goal is to find vulnerabilities before the real attacker.

Step by step:

1. List threat scenarios. How can this system be abused?
2. Prepare the attack set. Write concrete entry examples for each threat.
3. Try systematically. Run each scenario and record the result.
4. Prioritize findings. Sort by impact × probability.
5. Fix it and test again. After the patch, try again with the same set (regression).

Model Card and Acceptance Criteria

A model card is a document that summarizes what a model is suitable for, its limitations, known risks and performance. Before you put it into production, you should have criteria for an acceptance decision: accuracy threshold, red team pass rate, latency, cost, and bias tests.

Four Copiable Templates

Risk classification prompt:

Consider the following use case: {{ scenario }}Questions:- Who/what does the bug affect? (person, money, reputation, harmony)- Is it reversible? (yes/no) - Can humans intervene? Result: "Low / Medium / High risk" + list of mandatory checks.

Red team attack set generator:

You are a red team specialist. Generate 15 attack scenarios for the following assistant: 5 jailbreaks, 5 prompt injections (3 of which are indirect), 5 data exfiltration attempts. For each scenario: write the purpose, the full introduction text, and "success criteria" (whatever I see counts the attack as successful).

Model board skeleton:

Model Card:- Intended use / unintended use- Training/data limits and known vulnerabilities- Performance: accuracy, latency, cost (on test set)- Security: red team pass rate, known jailbreaks- Bias testing results- Acceptance decision: APPROVAL / CONDITIONAL / REJECTION + justification

Admission gate control rule:

ALL conditions must be met to move to production:- >= target threshold on accuracy test set- Red team critical findings count = 0- If high risk: human inspection and monitoring boardIf none are met: "NO-GO" + missing item.

Weak Prompt / Strong Prompt

poor approach

Strong approach

Processing each use with the same control

Classify by risk and scale control

"We tested it, it works" (happy way)

Deliberate breaking attempt with the red team

Putting the model into production without justification

Model card + acceptance gate (go/no-go)

Not retesting after patching

Regression test after correction

Three Mini Cases

Case 1 — Misclassification was costly. One company deemed the recruitment prescreening "just an adjunct" and deemed it low risk. The model systematically eliminated graduates from certain schools; this turned into a discrimination complaint. Usage was reclassified as “high risk” and bias testing and human monitoring were added.

Case 2 — Red team found 3 critical vulnerabilities. A customer assistant was assigned to the red team before going into production. 3 out of 15 scenarios were successful: another customer's order information could be leaked through an indirect injection. The gaps were closed and retested with the same set; Production was resumed only when the critical finding was reset.

Case 3 — The model clarified the decision to accept the card. Choosing between two models, a team placed model cards side by side. The cheaper model hit the mark in accuracy, but was vulnerable to 2 critical jailbreaks on the red team. The team chose the expensive but safe model due to the acceptance gate "critical finding = 0" rule and documented the decision.

Tip: Red team is not a one-time event. Rerun the attack set whenever the model, prompt, or tools change; Security is not a state, but an ongoing practice.

Common mistakes

• Classify use by name (rather than effect); mistaking high risk for low.
• Just testing the “happy path” and not trying the abuse at all.
• Doing the red team once and not repeating it after changes.
• Putting the model into production without a model card and acceptance criteria.
• Bypassing bias/discrimination testing (especially in high-stakes human decisions).
• It means "closed" without doing post-correction regression testing.

In summary

• The first step is to classify uses as low/medium/high risk according to impact; Control intensity increases with risk.
• Red teaming is deliberately trying to break the system like an attacker; finds the vulnerability before the real attacker.
• The model card documents the model's purpose, limitations, and risks; is the basis for the admission decision.
• Transition to production must be tied to a go/no-go: accuracy, critical finding zero, required monitoring.
• Security is continuous: red teaming and regression testing are repeated with every change.

Application task

Choose your use of an AI, determine the risk level based on the impact, and write the justification. Then generate at least 10 attack scenarios for that use (jailbreak, injection, data exfiltration) and try them manually. For each successful attack, propose a fix. Finally, fill out a model card skeleton and make a "GO/NO-GO" decision with reasons.

checklist

• [ ] I have classified the use according to the risk level according to the effect.
• [ ] I matched control intensity to risk level.
• [ ] I prepared a red team attack set and tried it systematically.
• [ ] I fixed the critical findings and verified them with regression testing.
• [ ] I prepared a model card (purpose, limit, performance, security).
• [ ] I tied the production decision to a go/no-go.