Prejudice, Fairness and Discrimination: The Model's Blind Spots

Gains:

• Understanding how bias in artificial intelligence models arises from data and design and why it creates legal and ethical risks in banking
• Ability to recognize protected features, proxy variables and indirect discrimination and question model decisions in terms of fairness
• Ability to understand how justice metrics, excluded groups and the right to object are woven into the model governance and maintain the discrimination red line

An AI model is not neutral; It is a mirror of the data learned. If the data carries past inequalities, the model learns them and carries them into the future. In banking, this is not an abstract ethical debate: if a credit model systematically disadvantages a particular group, this is both unlawful (prohibition of discrimination) and an ethical breach, bringing reputational and sanctions risks to the bank. The purpose of this unit is to show how the model becomes biased, direct and indirect forms of discrimination, the proxy variable trap, and ways to weave fairness into model governance. One red line must be said from the outset: discrimination, no matter how "statistical" it may seem, is unacceptable.

Where does prejudice come from?

• Data bias: If past decisions were unfair (e.g. a particular group was given less credit in the past), the model assumes this is “normal” and continues. This is called historical bias.
• Sampling bias: If some groups are underrepresented in the data, the model learns poorly and inaccurately about them.
• Label bias: If the definition of “good customer” itself is biased, the model optimizes for the bias.
• Design bias: Human choices about which variables to use can also carry bias.

Tip: "The model is based on data, so it is objective" is a misconception. Data is a snapshot of the past; If the past was unjust, the "objective" model justifies the injustice with mathematics. Objectivity is not a guarantee of impartiality.

Direct and indirect discrimination

• Direct discrimination: The model's direct use of a protected characteristic (gender, ethnicity, religion, age). This is expressly prohibited.
• Indirect discrimination (surrogate variable / proxy): Even if the model does not use the protected feature directly, it produces the same result by using another variable that is strongly related to it (zip code, name, shopping category, working district). Zip code is often a proxy for ethnicity or income level. The model says "neighborhood" but the result is "origin" discrimination.

Variable

apparently

can be a proxy

Postcode / district

geography

Ethnicity, income level

Name

ID

origin, gender

Shopping category

behavior

lifestyle, belief

graduated school

Education

Socioeconomic origin

That's why alarm should sound when you see a statement like "neighbourhood is risky" in a justification: it may appear to be a legitimate criterion but carry indirect discrimination.

Caution: Removing the protected feature from the data does not end the discrimination. Surrogate variables can return it. Justice is not achieved by "I deleted the sensitive column"; This is achieved by testing the results of model decisions between groups.

Weaving justice into governance

1. Fairness metrics. Compare the model's approval/rejection rates and error rates between groups. Does one group systematically experience higher rejections?
2. Proxy control. Examine the relationship of the variables used to the protected properties.
3. Explainability. Justification must be produced for every decision; A "black box" denial is untenable.
4. Right to object. The customer should be given the right to learn the rationale for the decision and request a human review.
5. Excluded groups. It should also be checked that groups that are underrepresented in the data are not mistreated.

Four copyable templates

1) Screening for discrimination in justification:

Check the following credit decision rationale: is there a direct/indirect reference to a protected characteristic (age, gender, origin, religion) or a proxy variable (postcode, neighbourhood, name, shopping type)? Flag risky statements and explain why they pose a risk of discrimination. Reason: [text]

2) Variable list proxy control:

Examine this list of variables used in a credit model. Which ones could be a proxy for a protected characteristic and pose a risk of indirect discrimination? Write your justification for each risky variant. List: [variables]

3) Summary of decision in terms of fairness (neutral, explainable):

Your role: assistant who drafts an explainable decision rationale. Rely only on legitimate, non-discriminatory criteria (debt-to-income ratio, payment history, collateral). DO NOT reference protected property and surrogate variable. Write the rationale in plain language that the customer will understand. I will give the approval.

4) Objection response draft:

A customer appealed the credit denial and wants justification. Draft a respectful and descriptive response based on legitimate criteria; Remind the customer of his or her right to human review and correction. Do not use any discriminatory implications. Verified rejection reason: [reason]

Weak prompt / Strong prompt

Weak prompt:

Applications in this region are very risky, let's use district information more heavily in the model so that the accuracy can be increased.

This approach strengthens indirect discrimination based on district; It legitimizes an unlawful outcome in the name of "hit".

Powerful prompt:

Your role: justice audit assistant. Flag the variables used that pose the risk of surrogate variables. Suggest which metrics I should follow to compare decisions between groups in terms of approval/rejection rate. Never reinforce a discriminatory criterion; redirect to legitimate and explainable criteria.

Strong will looks for proxy risk, introduces fairness metrics, and rejects discrimination.

three mini cases

Case 1 — Historical bias. One model gives a particular occupational group a systematically low score because it has been given less credit in the past. A fairness audit shows that the rejection rate for this group is 30% higher than others at similar income levels. The model is rebalanced with legitimate income and payment criteria.

Case 2 — Surrogate variable. Zip code is a powerful variable in a model. The audit shows that the zip code overlaps with certain ethnically concentrated neighborhoods, producing indirect origin discrimination. The variable is removed and replaced by legitimate financial benchmarks; performance remains at an acceptable level, discrimination is eliminated.

Case 3 — Right to object. A rejected client asks for justification. The bank provides the explainable justification (high debt-to-income ratio) and accepts the request for human review. Review reveals a misread of a document; The decision is corrected. Explainability and the right to object compensate for a mistake with justice.

Fairness metrics: what and how we measure

“Is the model fair?” There is no single answer to the question; There are multiple definitions of justice, and they sometimes conflict with each other. Here are a few approaches that are practically followed in banking:

• Approval/rejection rate comparison: Is the approval rate systematically different between different groups with similar financial profiles (e.g. gender groups)? A large and unexplained difference is a sign of bias.
• Error rate equality: Is the model's false rejection (rejecting a legitimate application) rate balanced across groups? If one group is unfairly rejected more often than another, there is a problem.
• Calibration: Do the customers the model calls “high risk” really default at similar rates in each group? The meaning of the score should not shift from group to group.

These metrics sometimes cannot be provided at the same time; Which definition of justice to prioritize is not a technical but an ethical and legal decision and requires human governance.

Tip: Measure the fairness metric at regular intervals, not just when you build the model. A model may go live fairly and become biased over time due to data drift. Justice is not a matter of "establishment" but a matter of "monitoring".

Common mistakes

• The "objective data" fallacy. Assuming the model is neutral; data carries historical bias.
• Just delete the sensitive column. Removing the protected property and overlooking proxy variables.
• Not tracking fairness metrics. Not comparing the rejection/error rate between groups at all.
• Defending the black box decision. Considering a decision that cannot be justified as legitimate.
• Removing the right to object. Disabling human review because "it slows down the process."

Attention: Discrimination often arises not from malice but from carelessness. A variable added because it "increases accuracy" may unknowingly exclude a group. Therefore justice is based not on good faith but on regular testing.

In summary

AI models carry the biases of the data they learn from, and discrimination in banking is a legal, ethical and reputational risk. Direct discrimination is using a protected feature; Indirect discrimination, on the other hand, is producing the same result with proxy variables (zip code, name). Justice is not about erasing the sensitive column; Ensured by fairness metrics, proxy control, explainability, and right to object. In one sentence: The pattern repeats the past; It is man's responsibility to uphold justice and reject discrimination.

Application task

Write a list of 8-10 variables for a credit model (some with intentional proxy risk: zip code, name, shopping category). 2. Perform a proxy check with the template and mark the risky ones with their reasons. Then write a discriminatory justification text and scan it with template 1. Finally, plan in a paragraph which fairness metrics you will track to compare the model's rejection rate between groups.

checklist

• [ ] I have checked that the model/rationale does not use protected properties.
• [ ] I also screened the risk of proxy variables.
• [ ] I planned to compare the approval/rejection and error rate between groups.
• [ ] I ensured that every decision had an explainable rationale.
• [ ] I gave the customer the right to object and human review.
• [ ] I did not even cross the discrimination red line on the grounds of "hit".