Gains:
- Ability to explain the steps in cleaning, coding and analyzing electronic health record (EHR) data with artificial intelligence.
- Ability to recognize clinical data risks such as missing data, bias, class imbalance, and temporal leakage
- Ability to validate predictive model outputs through calibration, subgroup performance, and clinical usefulness
The memory of modern hospitals is the electronic health record (EHR): a digital collection of diagnoses, lab results, medications, procedures, nurse notes and doctor observations. This data is both a treasure and a minefield for artificial intelligence. Treasure because it contains millions of sick annual patterns; minefield because clinical data is disorganized, incomplete, biased, and full of temporal traps. This unit includes cleaning, coding and analyzing EHR data with artificial intelligence; the most insidious errors (temporal leakage, bias, class imbalance); and we will see how the predictive model outputs are validated.
Let's remind from the beginning: a risk model may say "this patient has a high probability of readmission"; but this is a decision support output, not a diagnosis or treatment decision. AI does not diagnose; The decision is up to the physician and clinical team.
Steps to Working with EHR Data
Step 1 — Subtract and merge. Data comes from different systems (laboratory, pharmacy, radiology); is combined with the patient ID. At this stage, confidentiality is the highest priority (see Unit 10).
Step 2 — Cleaning. Missing values, unit inconsistencies (confusion of mg/dL and mmol/L), duplicate records, and unlikely values (age 200, blood pressure 0) are eliminated. AI is strong here in outlier flagging and free text structuring.
Step 3 — Coding and standardization. Diagnoses are mapped to standard codes such as ICD (International Classification of Diseases), and medications are mapped to standard dictionaries. Extracting structured information from free text notes is called natural language processing (NLP); AI is valuable here, but its output requires validation.
Step 4 — Feature engineering. Model inputs are generated from raw data: last laboratory value, trend, number of medications, comorbidity score, etc.
Step 5 — Modeling and evaluation. The predictive model is built and—the most critical part—evaluated in a careful, leak-free manner.
The Three Most Insidious Mistakes
Temporal leakage. Putting information into the feature that does not yet exist at the time of prediction. For example, using discharge diagnosis or last-day laboratory testing to estimate the risk of death at admission. The model looks perfect in the laboratory, but crashes in real use because it has seen the "future".
Prejudice. The data reflect past clinical decisions; If these decisions are already unequal, the model learns and reinforces this. For example, if a certain group has historically been ordered less for testing, the model might think that the disease is "lower" in that group.
Class imbalance. If the event of interest (e.g. a rare complication) is 1% of the data, a model that always says "no event" would look 99% accurate but would be useless. Instead of accuracy, sensitivity, precision, and event-based metrics are required.
Evaluation: Discrimination is Not Enough, Calibration is a Must
There are two different questions. Discrimination (typical measure of AUC): does the model correctly rank patients by risk? Calibration: do the probabilities given by the model match the actual frequencies? Do approximately 20% of the patients who say "20% risk" actually have an event? Since decisions are made based on thresholds in the clinic, a well-ranking but poorly calibrated model will lead to incorrect threshold decisions. Additionally, subgroup performance (age, gender, ethnicity, hospital) should be reported separately and the question of clinical usefulness (does using this model really produce better results?) should be asked.
Three Mini Cases: By the Numbers
Case 1 — Success inflated by leakage. A readmission model yielded an AUC of 0.92 in internal testing; it looked great. The review found that features included “post-discharge outpatient appointment”; This information was not available at the time of prediction. Once the leak was cleaned up, the AUC dropped to 0.71 — a realistic and usable value.
Case 2 — Calibration drift. While a sepsis early warning score was well calibrated at the hospital where it was developed, the patient profile showed twice the actual event rate for the same score at a different hospital. The score ranked correctly but the probabilities were wrong; the threshold could not be used without recalibration.
Case 3 — Saving time with NLP. One research team was manually extracting smoking history from 12,000 patient notes; Approximately 2 minutes per note, 400 hours total. NLP-assisted extraction reduced this to a few hours; The team only hand-checked a randomly selected validation sample that the model left “unclear.” Critical was the meticulous examination of the validation sample.
Weak Prompt / Strong Prompt
Weak prompt:
Build a risk model from this patient data and report results.[data]
Powerful prompt:
Your role: You are a clinical data analysis assistant (YOU DON'T DECIDE). Critique a prediction model PLAN for the following list of anonymous variables:- Mark whether each variable is present at the time of prediction (risk of temporal leakage).- List class imbalance and potential sources of bias.- Suggest discrimination + calibration + subgroup + clinical usefulness for evaluation.- State that the decision rests with the clinical team. Anonymous variables and target:[list]
Four Copiable Templates
1) Leak inspection:
Classify the following list of features as "available at the time of prediction" / "future information (leak)" and justify it. Goal and moment of prediction: [definition]. Features: [list]
2) Data quality report:
Check for missing value rate, missing values, unit inconsistency and duplicate record for this anonymous table; A quality summary table appears. Table: [data]
3) NLP inference verification scheme:
Write a validation plan for an NLP step that extracts [variable] from free-text annotations:random sample size, gold standard labeling, fit metric, error analysis.
4) Evaluation framework:
Write a draft evaluation framework for this clinical model: discrimination (AUC), calibration curve, subgroup performance, decision curve analysis. Model: [description]
Model's Role: By Task Type
Task type
AI contribution
criticality
verification
Data cleaning/outlier
high
low
spot check
NLP extraction from notes
high
medium
Sample validation
Risk/prediction scoring
medium
high
Calibration + subgroup
Resource/capacity planning
medium
medium
Business unit approval
Treatment decision
limited
very high
Full physician approval
Tip: If a clinical model's AUC is unexpectedly high (e.g. above 0.95), look for temporal leakage before celebrating. In the real world, such high values are usually a sign of information leakage.
Caution: The model may learn and reinforce inequalities in historical data. High overall accuracy may imply systematic harm in certain patient groups; Performance should always be reported in subgroups.
Common mistakes
- Not noticing temporal leakage. Knowledge of the future produces false success in the laboratory.
- Be satisfied with accuracy. Accuracy is misleading with unbalanced data; Sensitivity and calibration required.
- Not reporting subgroups. The overall metric hides bias and inequality.
- Skipping calibration. Even a good ranking model can make mistakes in its threshold decisions.
- Leaving the decision to the model. The output is support; The treatment decision is up to the clinical team.
In summary
- EHR data is powerful but patchy, incomplete, and biased; cleaning and coding are critical steps.
- Temporal leakage is the most insidious error; Future knowledge produces false success in the laboratory.
- Class imbalance and bias make the accuracy metric misleading.
- Evaluation should include discrimination, calibration, subgrouping, and clinical usefulness.
- Forecast outputs are support; Diagnosis and treatment decisions belong to the clinical team.
Application task
Construct a prediction goal and a list of ten variables (intentionally include a “prospect” variable, e.g. discharge diagnosis). Use the powerful prompt to check the model for leaks and see if it captures this variable. Then write an evaluation framework for this model in three items, including discrimination, calibration, and subgrouping.
checklist
- [ ] I understand the extraction, cleaning, coding, and modeling steps of working with EHR data.
- [ ] I can recognize the temporal leak and inspect the property list.
- [ ] I realized how class imbalance and prejudice mislead accuracy.
- [ ] I know the difference between discrimination and calibration and the necessity of both.
- [ ] I can position the predictive output as support, not decision.