Gains:
- Ability to measure and report uncertainty with ensemble, sensitivity analysis, cross-validation and blind testing
- Ability to prevent data leakage by separating it on a well/field basis and ensure that the reported accuracy reflects true generalization.
- Ability to calibrate the artificial intelligence trust score with the reliability diagram and apply the discipline of not using an uncalibrated score as a probability
There is a recurring theme in each unit of this module: there are no "sure answers" in geophysics, only models limited by uncertainty. This unit turns that theme into a discipline. Model uncertainty is the existence of different subsurface models that can explain the same data and each model carries a confidence interval. Validation is testing with independent evidence whether a model or AI output is actually valid. Calibration is to ensure that the confidence/probability values given by a model match the actual results. In this unit, we will examine how artificial intelligence (AI) can both measure and hide uncertainty; and we will see why a solid verification-calibration framework makes AI reliable.
Sources of uncertainty
Geophysical uncertainty comes from several layers:
- Data uncertainty: Measurement noise, limited coverage, calibration error.
- Model uncertainty (polysemy): Fitting of more than one subsurface structure to the same data.
- Method/parameter uncertainty: Processing, inversion, and regularization choices change the outcome.
- Uncertainty specific to AI: The model's ability to remain confident but fail when it moves outside the training distribution (distribution shift).
The job of a good geophysicist is not to eliminate this uncertainty, but to measure, communicate and narrow it. AI can make this easy (generating multiple scenarios) but can also easily hide it by producing a single confident answer.
Ways to measure uncertainty
A few practical tools:
- Ensemble: Multiple runs with different starts, parameters, or models; The spread of results gives uncertainty.
- Sensitivity analysis: Changing an input (parameter, subset of data) and seeing how much the result changes.
- Cross-validation: Dividing the data into pieces, training with one part and testing with the other; Measures generalization power.
- Blind test: Testing the model with an independent well/field that it has never seen in training.
- Probabilistic output: Providing the result as a distribution/confidence interval, not a single value.
Tip: When the AI gives a result, always ask: "What and how much do I need to move in the input to change this result?" If the result reverses with a small parameter change, that result is fragile and uncertainty is high.
Calibration: does the confidence score hold true?
AI models often give a confidence/probability (“90% fault”). But this number is misleading if it is not calibrated: the model is actually right in perhaps 60% of the cases it says are "90%". Calibration is to align this number to the actual result rate. In practice, a reliability diagram is drawn: a well-calibrated model is really 90% right when it says "90%". In geophysics, it is essential to calibrate the AI confidence score with independent data before making it the basis for a decision.
Caution: High accuracy is not the same as good calibration. A model may be generally accurate but overconfident; The important thing in critical decisions is that it is truly reliable when it says "95%". Do not treat an uncalibrated confidence score as a probability.
Golden rules of verification
For robust verification: (1) Independence — test data should not interfere with the training/tuning process at all (data leakage — inflates the result). (2) Blind test — field/well that the model does not see. (3) Physical consistency — the result should not contradict physics and geology. (4) Reproducibility — the same result with the same input, and can be produced by someone else. (5) Documentation — record of what data, which parameter, which model version was used.
three mini cases
Case 1 — Data leak fallacy. One team reported that the facies classifier yielded 94% accuracy. The investigation showed that neighboring samples from the same well were involved in both training and testing (data leakage). When broken down by well, accuracy dropped to 71% — that was the true generalization. The leak made the model look better than it actually was.
Case 2 — Overconfident model. One fault detector gave "95% confidence" in its signs. The reliability diagram showed that the "95%" marks were actually 70% accurate. Once the model was calibrated, the confidence scores became realistic and the commentators correctly adjusted how much they would trust each sign.
Case 3 — Fragile inversion. A gravity model looked very convincing. The sensitivity analysis showed that the main structure disappeared when the regularization weight changed by 20%: the structure did not come from the data, but from the parameter selection. The team flagged the structure as “low confidence” and requested additional data.
Four copyable templates
1) Uncertainty measurement plan:
Your role: geophysical uncertainty consultant. I have an [inversion / classification / prediction] result. Which methods (ensemble, sensitivity, cross-validation, blindtest) do I apply to measure uncertainty and in what order? Explain what each one tells me and how to report the result.
2) Calibration check:
My AI model gives a confidence/probability score. How do I test whether this score is calibrated? How do I construct a reliability diagram, recognize "overconfidence" or "overcautiousness" and align the score to the actual outcome rate?
3) Data leak audit:
I trained a geophysical classifier. How do I find and prevent the risk of data leakage (same well/field samples entering both training and testing)? How do I set up separation by well/field? How do I know if reported accuracy reflects true generalization?
4) Result fragility testing:
I have an inversion/model result. I want to understand how fragile this result is. By changing which parameters and how much does the main structure change? How do I distinguish whether the structure comes from data or parameters/premise? Give a sensitivity protocol.
Weak prompt / Strong prompt
Weak prompt:
See if my model has high accuracy.
A single truth number; leakage hides calibration and generalization, giving false confidence.
Powerful prompt:
Your role: model validation expert. Input: [classifier generates N well,confidence scores]. Task: (1) propose well-based allocation against data leakage; (2) install blind well testing; (3) calibrate the confidence score with the reliability diagram; (4) test how sensitive the result is to the parameter. Reduction to a single truth number; Report generalization, calibration and fragility separately.
Leakage, blind testing, calibration and sensitivity request make verification honest.
Uncertainty tools
vehicle
What does it measure?
Main risk
output
ensemble
Model propagation
Account cost
Range/distribution
Sensitivity
Parameter effect
Missed input
Fragility
cross validation
generalization
data leak
realistic accuracy
blind test
independent success
Insufficient test set
trust
Calibration
trust reality
overconfidence
aligned probability
Common mistakes
- Not noticing data leakage. It inflates righteousness; Separate by well/field.
- Using the confidence score without calibrating it. “90%” may not actually be 90%.
- Relying on the single truth number. Generalization and calibration are two different things.
- Not testing vulnerability. The structure lost by the parameter is not real information.
- Not reporting uncertainty. Presenting the result without a confidence interval is misleading.
In summary
Uncertainty is the ineradicable fact of geophysics; the job is to measure it, communicate it, and narrow it down. AI facilitates this with ensemble, sensitivity, and probabilistic output, but can also disguise it with a single confident answer. A solid frame; preventing data leakage, measuring generalization with blind testing, calibrating the confidence score, and testing the fragility of the result. Uncalibrated trust, unverified truthfulness, and concealed uncertainty are the three most dangerous illusions. Reliable geophysics is geophysics that honestly demonstrates its uncertainty.
Application task
Select an AI geophysical result (classification, inversion or prediction). Plan ensemble/sensitivity/blind testing steps with the "Uncertainty measurement plan" template. Then test your train-test distinction with the “Data leak audit” template. If the model gives a confidence score, evaluate whether the score is realistic with the "Calibration check" template and write your result with a confidence interval.
checklist
- [ ] I measured uncertainty with ensemble/sensitivity.
- [ ] I prevented data leakage by separating it on a well/field basis.
- [ ] I tested generalization with a blind test.
- [ ] I calibrated the confidence score and checked its realism.
- [ ] I reported the result with a confidence interval, not a single value.