Gains:
- Ability to understand the three pillars of observability (metric, log, trace) and the four golden signals and have artificial intelligence generate PromQL queries, alarm rules and dashboards
- Ability to prevent alarm fatigue by keeping alarms action-oriented and at the right urgency and testing thresholds against your own system's historical data
- Ability to prevent privacy and secret leakage by masking sensitive areas before giving the logs to artificial intelligence
While a system may appear to be working, it may be dying inside: memory slowly filling up, response times increasing, error rate creeping up. The only way to notice this is to constantly monitor the system. A more advanced concept is observability: the ability to understand what's going on inside the system by looking at its external signs. There are three pillars of observability, and the DevOps professional uses all three:
- Metric: Numerical values measured over time — CPU usage, number of requests, response time, error rate. "How much?" answers the question.
- Log: Text event records produced by the system—"user logged in", "database connection lost". "What exactly happened?" answers the question.
- Trace: The path a request follows while passing from service to service within the system and the duration of each step. “Where is the slowness?” answers the question.
Most common tools: Prometheus for metrics, Grafana for visualization, Loki/ELK for log, Jaeger/OpenTelemetry for trace. AI is very skilled at writing the query languages (especially Prometheus' PromQL), alarm rules, and dashboard configurations for these tools. It's also where AI is at its strongest: summarizing large chunks of logs and metrics and flagging anomalies.
Let's clarify the difference between monitoring and observability in one sentence: monitoring is asking questions you already know (“Is the CPU past 90%?”); observability is being able to ask questions you didn't already know ("why is this weird slowness only happening for a certain customer at a certain time?"). Modern systems are so complex that you cannot predict all modes of failure; Therefore, the ability to collect rich metrics, logs, and traces and then query them in depth — that is, observability — becomes critical. This is where AI comes into play when answering the "previously unknown question": it quickly scans the raw data you have, suggests patterns and anomalies, and you get to the root cause by verifying these clues.
Step by step: what and how to monitor?
- Choose the right metrics. In the industry, "four golden signals" are taken as basis: latency, traffic, errors, saturation — how full the resource is. These summarize the health of most services.
- Collect metrics. Let the application present an endpoint that Prometheus can read.
- Set up dashboards. Visualize these metrics in Grafana.
- Write alarm rules. Who will be warned when a threshold is exceeded and how?
- Centralize logs. Make all service logs searchable in one place.
- Reduce noise. Too much alarm creates “alert fatigue”; The important alarm disappears.
Tip: A good alarm meets two things: it's actionable and has the right urgency. An alarm that wakes someone up at 3am must be something that actually requires nighttime intervention. Don't wake anyone up for something that doesn't require action on its own, like "CPU 70%"; display it on the board.
How to write an alarm rule?
An alert consists of three components: condition (which metric exceeds which threshold and for how long), duration (“for 5 minutes” to avoid triggering momentary fluctuations), and importance/action (to whom, through which channel). AI masterfully establishes these three with the right context. For example, translating a rule like “critical alarm if error rate exceeds 5% for 5 minutes” into PromQL is a split-second task for the AI — but you decide if the threshold is right for your system.
Caution: The alarm thresholds suggested by the AI are general assumptions. Your system's normal load, tolerance and work impact are different. Before you put a threshold directly into prod, you look at your historical data and ask "how many times has this threshold been triggered in the past, how many of those were real problems?" Answer the question.
Log privacy: critical warning
Logs are the most frequently overlooked source of leaks. A log line may accidentally contain a password, a credit card number, or personal data (under KVKK/GDPR). When pasting logs into an AI for analysis:
- Mask sensitive areas. Replace values such as token, password, email, ID number with <REDACTED>.
- Give examples, not all. Instead of a million lines, a few hundred representative lines are often enough.
- Choose an institution-approved vehicle. Especially for production logs, use a tool whose data does not go to training.
Four golden signals and alarm tables
signal
measured by
Example alarm threshold
urgency
latency
response time
p95 > 800 ms, 5 min
high
traffic
Request/sec
Sudden 300% increase/decrease
medium
Error
Failed request rate
> 5%, 5 min
critical
Saturation
resource occupancy
Disk > 85%
high
three mini cases
Case 1 — 400 lines of log summarized in 30 seconds. A service had slowed down. The engineer gave the masked 400 lines of log to the AI and said, "summarize the recurring error patterns and time intensity." AI showed that a particular external API call times out every 30 seconds. Root cause found in 30 seconds; Scanning logs manually would take half an hour.
Case 2 — alarm fatigue solved. One team was receiving 200 alarms a day and was ignoring them all — until a real outage alarm was also overlooked. Give the AI all the alert rules and ask "which ones are not actionable and which ones can be combined?" they asked. The number of alarms decreased to 12 per day; Every alarm was now taken seriously.
Case 3 — wrong threshold caught early. YZ suggested "Warn when 95% full" for the disk. The engineer looked at historical data: once the disk reached 95% there was little time for intervention. It lowered the threshold to 80% and added a second alarm based on “growth rate.” Verification prevented an actual midnight outage.
Four copyable templates
1) Log summarization (masked):
Analyze the log example below (I masked sensitive values with <REDACTED>). Give me: (1) recurring error patterns, (2) concentration over time, (3) most likely root cause, and (4) 3 metrics I will look at to verify. Log: [LINES]
2) Alarm rule generation:
Write an alarm rule for Prometheus/Alertmanager: Generate [SEVERITY] alarm if [THRESHOLD] exceeds [METRIC][DURATION]. The rule should be action-oriented and include an annotation and runbook link field. Explain PromQL and write why this threshold is reasonable.
3) Writing/declaring PromQL query:
Write a PromQL query that measures: [EX. 5xxerror rate percentage in the last 5 minutes]. Explain the query step by step. Then tell me what the healthy range for this value should be.
4) Dashboard design:
Design a Grafana dashboard for [SERVICE]: with which panels should I display the four golden signals(latency, traffic, error, saturation)?Suggest metric, visualization type and reasonable threshold for each panel. Purpose: to see the health status of a guard in 10 seconds.
Weak prompt / Strong prompt
Weak: "What's in that log?" (followed by 5000 lines of raw log, tokens in it)
Result: you leak secrets and the AI gives an untargeted, superficial summary.
Strong: "Find recurring error patterns and time intensity in the 300-line masked log example below; tell me the most likely root cause and the metrics I'll look at to verify. I made the tokens <REDACTED>."
Difference: the second prompt gives a masked and focused example, asking for a clear analysis output; It is both safe and useful.
Common mistakes
- Pasting the log into AI without masking it. The most common secret/personal data leak.
- Setting alarms for everything. Alarm fatigue buries real alarm.
- Non-actionable alarm. It is warning noise that no one can do anything about.
- Accepting the threshold of AI without question. The threshold should be set according to your system's history.
- Just looking at the metric. Without log and trace, the root cause cannot be found most of the time.
- Not setting an alarm time (for). Momentary fluctuations produce false alarms.
In summary
Observability; It is the ability to understand the inside of the system from the outside with metrics, logs and traces. The four golden signals (latency, traffic, error, saturation) summarize the health of most services. AI is very powerful at writing PromQL queries, alarm rules and dashboards, and at summarizing large chunks of logs and finding anomalies. But it is your responsibility to verify alarm thresholds against your own system's history, keep alarms action-oriented, and never share logs without masking them.
Application task
For a service (or a sample service): (1) Have an alarm rule generated for the error rate with the "Alarm rule generation" template and set the suggested threshold to "how many times has it triggered in the past?" Test it with the question; (2) mask a log sample you have and have it analyzed with the "Log summarization" template; (3) note which metric you will look at to confirm the most likely root cause.
checklist
- [ ] I chose the metrics to track based on four golden signals.
- [ ] I masked all the logs I gave to the AI in terms of sensitive areas.
- [ ] I verified that each alarm was action-oriented and of the correct urgency.
- [ ] I tested the alarm thresholds against my system's historical data.
- [ ] I filtered instantaneous fluctuations by adding for (duration) to the alarms.
- [ ] I used metric + log + trace together for root cause.