Gains:
- Ability to recognize different data sources (database, API, file, web scraping) and the pitfalls of each and understand the schema correctly
- Ability to perform repeatable sampling by evaluating whether the sample represents the population and selection bias
- Ability to eliminate data leakage at the collection stage and observe legal/ethical boundaries by asking the question 'Will I have it at the time of prediction' in each column?
Every analysis is as good as the quality of the data you collect. Even the most advanced model in the world will produce unreliable results if it works with data that is collected incorrectly, sampled biasedly, or contains information about the future. In computer science, this principle is summarized as "garbage in, garbage out" (garbage in, garbage out). In this unit, we'll cover the data collection phase: understanding the source, sampling, asking quality questions, and being alert to the risk of data leakage from day one. Artificial intelligence is a powerful aid at this stage; Writes SQL query, summarizes API document, drafts data contract. But it is the human being who decides what data you collect and whether that data represents you.
Getting to know data sources
Data comes from different places, and each source has its own pitfalls. Database (structured data stored in tables, usually queried with SQL) is the most common source; It is reliable, but it is necessary to understand its scheme well. API (Application Programming Interface) provides live data but carries the risk of speed limits and format changes. Files (CSV, Excel, JSON) are flexible but prone to format inconsistency. Web scraping is powerful, but it has legal and ethical limits; Not every site can be scraped.
Attention: For web scraping and automatic data collection, comply with the site's terms of use, robots.txt file and KVKK/GDPR. Unauthorized data collection creates legal liability. In the context of information security, use data collection tools only on systems for which you are authorized and for defense/analysis purposes; Unauthorized access or scraping is prohibited.
Understanding the schema: getting acquainted with the data
Before collecting a data set, you must understand its schema (the names of the columns, their data types, their meanings, and their relationships with each other). AI is very useful here in creating a "data dictionary" — a table explaining what each column means. But the explanations AI produces are predictions; Confirm the true meaning of each column with the team that produced the data. For example, a column named "status" might contain 0/1/2; Only the originating team knows whether these are "pending/approved/cancelled" or something else.
The following table summarizes the basic resource types and cautions:
Source
strong point
trap
How AI helps
SQL database
Structural, reliable
Complex JOINs
Writes a query draft
API
live data
Speed limit, shape change
Document summaries, pull code
CSV/Excel
Flexible, fast
Format inconsistency
Read/parse code
web scraping
Wide reach
Legal/ethical limit
Parsing draft (within authority)
Log/event data
detailed
huge volume
Filtering query
Illustration: does the part represent the whole?
Most of the time, you work with a sample (a subset selected from the population) rather than the whole data. The critical question is: does this sample represent the population? Selection bias is the most common trap. For example, if you only sample users from the mobile app, you won't see web users and your results will be misleading. Random sampling (each record has an equal chance of being selected) is safest in most cases; but in time series data, splitting is done chronologically rather than randomly (we will see this in Units 7 and 10).
Leak awareness from day one
Data leakage is the source of most disasters and usually arises during the data collection phase. Example: when predicting "was it cancelled", if you add the "cancellation date" column to the data, the model looks into the future. During the gathering phase, ask one question for each column: “Will I actually have this information at the time I make the prediction?” If the answer is no, that column is leaking. We will cover this topic in depth in Unit 10; But awareness should start from day one.
three mini cases
Case 1 — The problem of representation. One bank collected data only on approved loans for its credit risk model (18,500 records). Rejections were not in the data. The model was wrong in the real world because it never saw how rejects would behave. Lesson: the sample should be representative of the entire population from which you are making your decision.
Case 2 — Silent form change. A team was pulling price data from an API every day. One day, the API provider changed the currency from USD to EUR, but the domain name remained the same. Data was collected in the wrong unit for 12 days; 3,200 lines were corrupted. Lesson: Regularly check volume and format consistency in API data.
Case 3 — Early leakage. An analyst included the "reason for account closure" column when collecting data for a "churn" estimate. This column was filled only after the customer left. The model yielded 97% accuracy on the test set; It didn't work in production because that column was empty at prediction time. Lesson: ask each column the question "do I have it at the time of the prediction?"
Four copyable templates
1) Data dictionary extraction:
Your role: data scientist assistant. Below are the column names and sample (anonymous) values of a table. For each column, list its estimated meaning, datatype, and potential quality risks in a table. Mark the columns you are not sure of as "confirmation required"; meaning making.Columns: [paste here]
2) Sampling code (random, repeatable):
I have pandas df. Write code that extracts a representative 5% random sample from 200,000 rows. Use random_state=42 (for reproducibility). Add code to check that the class distribution of the sample is similar to the population.
3) Leak scanning question:
I'll give you this list of columns. My goal is to predict "is it cancelled" (0/1). For each column, evaluate whether I will actually have it at the time of the prediction and mark it as "safe / suspicious / leak". Write your rationale in one sentence. Columns: [list]
4) SQL pull query draft:
I have "orders" and "customers" tables in PostgreSQL. Write a JOIN query that combines the orders of the last 90 days with the customer city and returns the total amount and number of orders per city. Explain the date filter and how NULL cities are handled. I will run the query and verify it.
Weak prompt / Strong prompt
Weak prompt:
Pull me a good sample data from this database.
"Good" is ambiguous; Which painting, which period, which size, which purpose is not clear. AI will only produce a generic, possibly wrong query.
Powerful prompt:
Your role: SQL assistant. I have a "transactions" table: columns id, customer_id, date (timestamp), amount (numeric), channel (text: 'web'/'mobile'). Task: Write a repeatable (deterministic with ORDER BY) query that returns 10,000 representative rows from each channel for the year 2024. Purpose: channel comparative analysis. List the assumptions of your query.
Here the table, purpose, size and repeatability are clear.
Common mistakes
- Not questioning the representativeness of the sample. Easily accessible data is not accurate data; selection bias distorts the result.
- Adapting column meanings to AI. The source team knows the meaning; Do not use the AI prediction without confirming it.
- Not tracking API format/unit change. The silent change collects corrupt data for days.
- Ignoring the leak at the collection stage. If the question "Do I have it at the time of prediction" is not asked early, the model will give false success.
- Collecting unauthorized or illegal data. Violation of robots.txt, terms of use and KVKK is a serious risk.
Tip: Keep a one-page “data card” for each new data source: source, pull date, number of rows, known boundaries, and columns at risk of leakage. This card saves the "what was this data" question and reproducibility months later.
In summary
The quality of the analysis is limited by the quality of the data collected. Know the source (database, API, file, scrape) and schema well; make sure the sample is representative of the population; Eliminate leakage from day one by asking each column “do I have it at the time of prediction?” AI is a great accelerator for query and document work, but humans decide what data to collect and its representativeness. Limits of authority, law and confidentiality always come first.
Application task
Choose a data source (from your own business or hypothetical). Get a draft of a data dictionary from AI with the “data dictionary extraction” template above; Then manually evaluate each column to see if it has been leaked. Try to find at least one suspicious/leak column and write in one sentence why it is risky.
checklist
- [ ] Have I confirmed the data source and schema with the source team?
- [ ] Have I checked that the sample is representative of the population?
- [ ] Have I asked each column the question "will I have it at the time of the estimate?"
- [ ] Have I made the sampling repeatable (fixed seed)?
- [ ] Have I checked the legal/ethical (authority, robots.txt, KVKK) limits of collection?