Table 6 Validation methods
From: Prognostic models for breast cancer: a systematic review
Domain | Measure | Description | Internal validation | External validation |
|---|---|---|---|---|
Overall performance | Measuring the distance between the predicted and actual outcomes [9] | 3 studies | 2 studies | |
| Â | R2 | The amount of variability in outcomes that is explained by the model [9] | 1 study | 1 study |
Brier score | A measure of the average discrepancy between the true disease status and the predicted probability of developing the disease [85] | 2 studies | 1 study | |
Calibration | The level of agreement between the observed and predicted outcomes [9] | 12 studies | 32 studies | |
| Â | Calibration plot | Having predictions on the x axis, and the observed outcome on the y axis [9] | 7 studies | 20 studies |
SMR (Standardised mortality ratio) | The difference from the predicted calibration line and the ideal line in calibration plot [69] | 0 study | 1 study | |
E/O | Ratio between the predicted and observed outcomes [100] | 3 studies | 2 studies | |
E-O | Absolute difference between the predicted and observed outcomes | 2 studies | 28 studies | |
Hosmer-Lemeshow goodness-of-fit test | The ability of a model to fit a given set of data [9] | 4 studies | 5 studies | |
Discrimination | The extent to which the model can discriminate patients with the outcome and those without the outcome [9] | 28 studies | 37 studies | |
| Â | Kaplan-Meier curve | The probability of surviving in a given length of time while considering time in many small intervals [140] | 23 studies | 20 studies |
Log-rank test | Testing the null hypothesis that there is no difference between populations in the probability of an event at any time point [141] | 16 studies | 18 studies | |
C-index | The probability that, for a randomly chosen pair of patients, the one who actually experienced the event of interest has a higher predicted value than the one who has not experienced the event [85] | 11 studies | 12 studies | |
AUC | Area under the receiving operating characteristic curve is identical to C-index for a model with binary outcome [9] | 11 studies | 12 studies | |
CPE | Concordance probability estimate represents the pairwise probability of lower patient risk given longer survival time [142] | 0 study | 1 study | |
Clinical usefulness | The ability to make better decisions with a model than without it [9] | 13 studies | 1 study | |
| Â | Accuracy rate | \( =\frac{true\ negative+ true\ positive}{Total\ patients} \) [9] | 11 studies | 1 study |
Sensitivity | The fraction of true-positive classifications among the total number of patients with the outcome [9] | 9 studies | 1 study | |
Specificity | The fraction of true negative classifications among the total number of patients without the outcome [9] | 8 studies | 1 study | |
Positive predictive value (PPV) | \( =\frac{number\ of\ true\ positives}{number\ of\ positives\ calls} \) | 1 study | 0 study | |
Negative predictive value (NPV) | \( =\frac{number\ of\ true\ negative s}{number\ of\ negative\ calls} \) | 1 study | 0 study | |
Agreement | Measure the agreement when comparing two models | 0 study | 4 studies | |
|  | Kappa coefficient (κ) | Measuring the inter-rater agreement for qualitative items. | 0 study | 1 study |
Correlation coefficient (Pearson or Spearman) | Measuring how strong a pair of variables is related | 0 study | 3 studies | |
Others | Shrinkage factor | Cross-validated prognostic index [143] | 2 studies | 0 study |
| Â | Univariate analysis | Examining the distribution of cases in only one variable at a time | 2 studies | 10 studies |
Multivariate analysis | Examining more than two variables simultaneously | 3 studies | 6 studies | |