Gains:
- Ability to define metrics that monitor usage, security, quality and performance signals
- Ability to detect output quality drift with baseline and sampling
- Ability to set alarm and feedback loop for anomalies and jailbreak waves
Putting an AI system into production is the beginning, not the end. Even if the model remains the same, the world changes: user behavior, incoming data, attack techniques, and business context are constantly shifting. Yesterday's correct answer may be wrong today. So the final pillar of security is continuous monitoring and observability — the ability to see from the outside what's going on inside the system. In this unit, we will learn what metrics to monitor, how to capture output quality drift, and how to alert for anomalies.
Why Continuous Monitoring?
In classical software, "does it work" is a binary question: either it answers or it doesn't. In AI, while the system appears to “work,” it can silently deteriorate: answers slowly become inaccurate, costs escalate, jailbreak attempts increase. The only way to capture these is to constantly measure the right signals.
Attention: The most dangerous malfunction is the silent one, not the noisy one. The system does not throw errors, its quality just decreases. If you don't set up monitoring, the first person to notice will be your customer or auditor, not you.
Four Signal Families to Watch
- Usage and cost: Request volume, token consumption, cost per user. Sudden jump; It could be a sign of abuse, a loopy integration, or a leaky switch.
- Security signals: Jailbreak/injection attempts, vehicle calls rejected, authorization errors. An increase may indicate an active attack campaign.
- Quality and drift: Decrease in output quality over time (drift). For example, verification pass rate, correction rate in human approval, user satisfaction.
- Performance: Latency, error rate, timeout. It directly affects user experience and cost.
What is Drift and How to Catch It?
Drift is when the quality of the model's inputs or output shifts unnoticed over time. There are two types: data drift (the distribution of incoming requests changes — new topic, new language) and quality drift (output for the same job gradually gets worse). A baseline is required to capture: record the normal range of metrics when the system is healthy; Let the deviation become an alarm.
Step by Step: Setting Up Monitoring
- Measure the baseline. Record the normal range of each signal when the system is healthy.
- Define threshold and alarm. Which deviation will warn whom and how?
- Sampling + human inspection. Have a human review a sample of the outputs regularly (quality drift is often only visible).
- Install a dashboard. Monitor four signal families on one screen.
- Feedback loop. Tie findings from monitoring to prompt/control improvement.
Four Copiable Templates
Quality sampling evaluation prompt (drift tracking with LLM-as-judge):
Below are 20 random printables from this week. Rate each as "good / acceptable / bad" and write a short justification. Finally I will compare the bad rate with last week's rate; If there is a pattern (recurrence of the same type of mistake) that stands out this week, mark it.<outputs>{{ examples }}</outputs>
Anomaly summary prompt:
Examine the following daily metrics: number of requests, tokens, cost, tool call rejected, jailbreak attempts, average latency. Mark any metric that deviates more than 30% from the baseline as "ANOMALIT" and estimate the possible cause (attack, bug, abuse).<metrics>{{ daily_data }}</metrics>
Alarm threshold definition rule:
Define alarms for each signal:- Cost: if exceeds 2x daily average -> high priority alert- Jailbreak attempts: if exceeds 10 per hour -> notify security team- Verification pass rate: if falls below 90% -> quality review- Latency: if p95 exceeds target by 2x -> performance review
Drift research prompt:
The verification pass rate has dropped from 94% to 78% in the last 2 weeks. Help me answer these questions: (1) Has a new topic/language/format appeared in the incoming requests? (2) Are errors concentrated in a particular category? (3) Does the timing coincide with a prompt/model/tool change? Name the data to be checked for each.
Weak Prompt / Strong Prompt
poor approach
Strong approach
"If there is an error, we will see"
Baseline + threshold + proactive alarm
Just checking to see if the system is standing.
Monitoring four families of signals (usage, security, quality, performance)
Not sampling the output quality at all
Regular human sampling + LLM-as-judge
Not collecting and looking at metrics
Dashboard + feedback loop
Three Mini Cases
Case 1 — Cost alarm caught the leaking key. A company's daily token cost tripled overnight. The threshold alarm alerted the security team; investigation showed that a test key had been leaked and used by a bot. The key was revoked in 25 minutes; If there had been no alarm, the bill would have been noticed at the end of the month.
Case 2 — Silent quality drift. A support assistant's verification pass rate quietly dropped from 95% to 80% in three weeks. Weekly sampling captured this; The reason was that customers started asking about a new product line and the model's knowledge base on it was incomplete. The rate recovered when the knowledge base was updated.
Case 3 — The jailbreak wave was early. Injection attempts made on an assistant increased from 2 to 40 per hour in one day. Security alarm triggered; It was seen that a "recipe" for cracking the system was shared in a forum. The team updated the defense prompt and rate-limited suspicious accounts; The wave died down before it turned into a real leak.
Tip: Don't settle for just machine metrics. Quality drift is often caught by just having a human read the sample outputs. A small routine of reviewing 15-20 random printouts per week will catch the most expensive silent failures early.
Common mistakes
- Not putting it into production and setting up monitoring ("it's working, okay").
- Not being able to identify the anomaly without measuring the baseline.
- Missing the quality drift by only looking at "does it stand up".
- Not sampling the output quality through human eyes at all.
- Not raising an alarm and finding out the problem from the customer/supervisor.
- Not connecting monitoring findings to improvement (no feedback loop).
In summary
- AI systems can quietly deteriorate; The most dangerous malfunction is the one that does not throw errors, but only reduces quality.
- Track four families of signals: usage/cost, safety, quality/drift, and performance.
- Drift (the drift of input or output quality over time) is captured only compared to a baseline.
- Regular human sampling in addition to machine metrics captures quality drift.
- Connect monitoring to the alarm and feedback loop; Measuring and not looking is not monitoring.
Application task
Choose at least one metric from each of the four signal families for your own AI system and write down their current (or estimated) baselines. Define an alarm threshold for each metric. Then take 15 of your last semester's outputs and score them with the sampling prompt above; Note the “bad” rate. Let this be your first baseline against which to compare drift in the future.
checklist
- [ ] I defined metrics from four signal families (usage, security, quality, performance).
- [ ] I set a baseline and alarm threshold for each metric.
- [ ] I regularly sample the output quality through human eyes.
- [ ] I monitor the signals on a single screen with a display panel.
- [ ] Alarm goes to security team for anomalies and jailbreak waves.
- [ ] I attribute the monitoring findings to the prompt/control improvement.