Gains:
- Ability to recognize AI-specific attack surfaces (prompt injection, data poisoning, confidential data leakage, membership extraction) and design layered defenses
- Ability to apply privacy as a design principle: data minimization, masking, access control and retention period
- Ability to conduct security work solely for defensive purposes, disclose vulnerabilities responsibly, and avoid unauthorized use
A machine learning system carries all the security risks of traditional software and adds unique new attack surfaces. The model can be fooled by an input, the training data can be poisoned, and confidential information can leak into the output. In this unit, we consider AI systems from a defense perspective: recognizing attacks, hardening the system, protecting privacy. This information is not for unauthorized access or attack, but to keep your own systems safe.
AI-specific attack surfaces
In addition to classic security (authentication, authorization, encryption), ML systems are vulnerable to:
- Prompt injection: Instruction hidden in the input to the LLM misses the model. The most common and most practical LLM security risk.
- Data poisoning: An attacker introduces a hidden backdoor or bias into the model by inserting bad samples into the training data.
- Model inference and inversion: An attacker reconstructs training data or model behavior by sending multiple queries to the model.
- Membership inference: Inferring whether a particular person's data is used in education — a privacy violation.
- Sensitive data leak: The model reveals confidential information (name, identity, secret) in the training data in the output.
There are defenses for each of these risks; The key is to consider risk at the design stage.
Prompt injection: the most immediate threat
There are two types of prompt injection:
- Direct: The user personally enters text such as "ignore previous instructions".
- Indirect: The bad instruction is hidden in an external context (web page, document, email) that the model processes. Particularly dangerous for agents and RAG because the model handles external content reliably.
Defense layers:
- Parsing: Separate system instruction and user/external data with clear delimiters; mark external content as "data, not commands".
- Minimum powers: Limit how much damage the model can do even if it is captured (vehicle powers in unit 5).
- Output control: Verify what the model produces before you use it — especially if it translates into an action.
- Human approval: Tie high-risk actions to approval.
Caution: You cannot completely solve prompt injection with a single defense; Layered defense (defense in depth) is required. Critical assumption: "The model might be fooled at some point; so what's the worst that would happen if it were fooled, and how do I limit that?"
Weak approach / Strong approach
Weak: "I typed 'ignore bad instructions' at the system prompt and we're safe."
Strong: "We wrapped external content with <data> tags and said 'ignore instructions within'. We also limited the model's tools to minimal authorization, tied irreversible actions to human approval, logged all tool calls, and subjected the output to rule checks before use. We rely on layers, not a single defense."
The difference: the strong approach knows that a one-line instruction will not be enough and builds layers that limit the damage.
Privacy: data is protected from the start
Privacy is not a feature added later, it is a design principle (privacy by design). Basic applications:
- Data minimization: Do not collect and store more personal data than necessary. Data that is not collected cannot be leaked.
- Anonymization and masking: Mask or remove personal identifiers (name, ID, email) before giving them to the model.
- Access control: Limit and log who accesses the data and model (RAG access control on unit 4).
- Retention period: Determine by policy how long you retain data; Delete the expired one.
Differential privacy (a technique that prevents a single individual's data from significantly affecting the output by adding controlled noise during training) and federated learning (an approach that trains on devices without moving the data to the center) are advanced privacy techniques; should be considered when working with sensitive data.
Tip: Before processing any data, ask: "If this personal data is leaked, who will suffer what harm?" If the damage is serious, either do not collect the data at all or process it by masking it. The safest data is data that has never been collected.
Training data and model supply chain security
As much as your model, the components you use are also a safety issue:
- Data source trust: Is the training data reliable or could it be poisoned? Audit public data sets.
- Third-party models and libraries: A pre-trained model or dependency you downloaded may be malicious. Check its source, signature, and known vulnerabilities.
- Supply chain: Every tool and package in your ML pipeline is a link of trust; You are as safe as the weakest link.
Responsible disclosure and ethical boundaries
When you find a vulnerability — on your own system or a vendor's system — the correct course is responsible disclosure: privately reporting the vulnerability to the relevant party and giving it time to fix it, not exploiting or disseminating it. Using artificial intelligence or the security information you have acquired for unauthorized access, data leakage, or unauthorized intervention into someone else's system is illegal and against professional ethics. The security content of this module is entirely for defense, detection and hardening purposes.
three mini cases
Case 1 - Limitation of indirect injection. A RAG support bot was rendering the web content. Hidden instructions were buried on one page. The model was partially fooled, but the bot had no write privileges (minimal privileges) and the output was passed through rule checking before being displayed to the user; It turned out to be harmful and was caught. Layered defense prevented a single failure from becoming a disaster.
Case 2 - Confidential data leak. A team fine-tuned customer support logs into a model without masking them (unit 6). The model started generating real customer names in irrelevant questions. There was also a risk of membership removal. Model withdrawn, data masked, retention policy corrected. Lesson: confidential data should not enter education.
Case 3 - Poisonous data set. One team trained on a publicly available dataset without auditing it. There were poisonous samples on set that fooled the model when it saw a specific trigger word (backdoor). After adding auditing and anomaly scanning, these samples were captured. Lesson: check the data source, don't blindly trust.
Copiable templates
Check this LLM/agent system for prompt injection.- Are system instructions and user/external data clearly separated?- Is external content marked as "data" or is it handled as a command?- What is the worst that would happen if the model is fooled (authorization limit)?- Are irreversible actions subject to human approval?- Is the output inspected before use?System: [description]. List layered defensive deficiencies.
Audit this data processing flow for confidentiality.- Is each collected personal field really necessary (minimization)?- Which fields should be masked in the data going to the model?- Is there access control and logging?- Is the retention period defined?Flow: [description]. Suggest correction for each deficiency.
In this text, find the personal data that needs to be masked before sending it to the model. Fields: name, email, phone, ID/passport number, address, card number, IP. List each finding with its type and recommended mask. Do not replace the rest of the text.Text: [text]
Generate a security checklist before putting this third-party model/library into production.- Are the source and publisher trusted, signature verified?- Scanned for known vulnerabilities (CVE)?- What privileges/access does it need, can it be minimized?Component: [name/source]
Risk-defense table
Risk
defense
layer
prompt injection
Parsing + minimal privilege + output control
Design + runtime
data poisoning
Source control + anomaly scanning
data line
Confidential data leak
Masking + data minimization
Data + training
Membership extraction
Differential privacy
Education
excessive authority
Minimum authorization + approval
agent design
supply chain
Component inspection + signature
addiction
Common mistakes
- Thinking that you have solved prompt injection with a single line. Layered defense is a must.
- Processing/training confidential data without masking it. Permanently infiltrates the model.
- Considering external content trustworthy. Indirect injection gate.
- Not checking the data source. Poisoning goes unnoticed.
- Blindly trusting the third-party component. Supply chain gap.
- Thinking that privacy will be added later. It should start from design.
In summary
In addition to classical security risks, AI systems carry unique threats such as prompt injection, data poisoning, confidential data leakage and membership extraction. None of them can be solved by a single measure; layered defenses (parsing, least authorization, output control, human approval) required. Privacy is a design principle: minimize data, mask it, limit access, impose retention periods. Control the component and data supply chain. All this information is for defence, detection and consolidation; Explain vulnerabilities responsibly, never exploit.
Application task
Check an LLM/agent system (your own project or example) for prompt injection: are system instructions and external data separated, what is the authorization limit if the model is tricked, are irreversible actions confirmed? Add at least two layers of defense. Separately, find and mask any personal fields that need to be masked in a sample data going to the model. Check the source and known vulnerabilities of any third-party component you use.
checklist
- [ ] System instruction and external/user data are clearly separated.
- [ ] External content is marked as data, not commands.
- [ ] Even if the model is fooled, the damage is limited to minimal authority.
- [ ] Personal data masked/minimized; storage period defined.
- [ ] Data source and third-party components have been checked.
- [ ] My security work is for defense purposes; I explain the gaps responsibly.