Developing a predictive nomogram for colposcopists: a retrospective, multicenter study of cervical precancer identification in China

Abstract

Background

Colposcopic examination with biopsy is the standard procedure for referrals with abnormal cervical cancer screening results; however, the decision to biopsy is controvertible. Having a predictive model may help to improve high-grade squamous intraepithelial lesion or worse (HSIL+) predictions which could reduce unnecessary testing and protecting women from unnecessary harm.

Methods

This retrospective multicenter study involved 5,854 patients identified through colposcopy databases. Cases were randomly assigned to a training set for development or to an internal validation set for performance assessment and comparability testing. Least Absolute Shrinkage and Selection Operator (LASSO) regression was used to reduce the number of candidate predictors and select statistically significant factors. Multivariable logistic regression was then used to establish a predictive model which generates risk scores for developing HSIL+. The predictive model is presented as a nomogram and was assessed for discriminability, and with calibration and decision curves. The model was externally validated with 472 consecutive patients and compared to 422 other patients from two additional hospitals.

Results

The final predictive model included age, cytology results, human papillomavirus status, transformation zone types, colposcopic impressions, and size of lesion area. The model had good overall discrimination when predicting HSIL + risk, which was internally validated (Area Under the Curve [AUC] of 0.92 (95%CI 0.90–0.94)). External validation found an AUC of 0.91 (95%CI 0.88–0.94) across the consecutive sample, and 0.88 (95%CI 0.84–0.93) across the comparative sample. Calibration suggested good coherence between predicted and observed probabilities. Decision curve analysis also suggested this model would be clinically useful.

Conclusion

We developed and validated a nomogram which incorporates multiple clinically relevant variables to better identify HSIL + cases during colposcopic examination. This model may help clinicians determining next steps and in particular, around the need to refer patients for colposcopy-guided biopsies.

Background

Approximately 600,000 women develop cervical cancer annually with a disproportionate burden of disease occurring in low- and middle-income countries (LMICs), where prevention programs are often limited [1]. With current population growth rates and demographic shifts such as aging, we can also expect this number to rise substantially [2]. The paradox here is that we can actually prevent this scenario from coming to pass, through screening and robust vaccination programs, yet many LMICs appear inhibited by their respective political economies. There are also more fundamental problems which need to be overcome in order to deliver effective cervical screening programs. These programs are necessarily multi-layered and involve cervical cytology testing e.g., Pap tests, as well as colposcopic assessments and biopsies. This means that effective cervical screening programs require experienced practitioners, who are not always readily available in LMICs [3]. Of course, new human papillomavirus (HPV) tests could prove useful in LMICs [4, 5]; however, processing all samples through cytology labs and byway of HPV-based screening actually appears to increase the number of false-negatives. This unacceptably high failure rate is likely to occur because cervical lesions related to HPV infections are probably milder and harder to identify than cytologic abnormalities [6, 7]. Therefore, we have to identify opportunities to improve screening and diagnostics, especially in cervical precancer identification.

In some ways, present colposcopic techniques lag behind HPV-related advances because the balance has rightly shifted toward prevention. At the other end of the diagnostic process, we have also witnessed the optimization of colposcopy-guided biopsy techniques. For example, the American Society of Colposcopy and Cervical Pathology (ASCCP) have proposed a low threshold biopsy protocol which involves multiple biopsies, targeting all areas. This low threshold protocol also includes acetowhite lesions and metaplasia, to improve high-grade squamous intraepithelial lesion or worse (HSIL+) detection [8, 9]. However, while this protocol will increase the reliability of both positive and negative results, we must maintain caution because biopsies are not without harm. In fact, the entire process can be traumatic and can have a negative impact on one’s self-efficacy and sexuality [10]. Unfortunately, screening and the diagnostic process generally, can also prevent women participating in other screening programs. This means, we have to improve mid-stage processes i.e. colposcopy, to avoid the twin traps in the diagnostic decision-making process. That is, we have to ensure we are not overly reliant upon less sensitive, less specific screening methods while preventing overdiagnosis.

There is no doubt that the decision-making processes are complex and decisions on biopsy, even among experienced clinicians, can vary. This means that there are subjective components which influence the decision to biopsy. These components can be debated from a number of different perspectives [11] although clinicians generally decide whether to biopsy, based upon previous experiences. Studies of this suggest the decision to biopsy is often influenced by the severity of a patient’s referral results [12, 13]. With colposcopic impressions, it may therefore be possible to quantify HSIL + risk, which clinicians can then use to develop evidence-based diagnostic decisions. This potential improvement encouraged us to develop and validate a predictive model which intercalates multiple clinically relevant variables to enhance early HSIL + case identification. The goal, of course, is to construct a simple, reliable nomogram which provides a reliable, individualized estimate for HSIL + risk.

Methods

Study design and participants

This retrospective multicenter study was conducted with six different hospitals across mainland China. Anonymized digital records were obtained to develop and validate a predictive model using colposcopy databases from four municipal or provincial hospitals. In order to maximize generalizability, this nomogram was also tested with patients enrolled from two additional hospitals in China.

Patients were assigned to one of two external validation sets. Statistically significant risk factors were identified through regression analysis. They were then intercalated into a coherent model which was converted into a nomogram for clinical practitioners. All patients with clear indications of abnormal screening results i.e., cytology and HPV testing, underwent colposcopic examination. Colposcopists assessed transformation zone (TZ) types and provided colposcopic impressions (i.e. normal/benign, (low-grade squamous intraepithelial lesion) LSIL, and HSIL+) following the international colposcopic terminology for each referred patient [14]. All colposcopic abnormalities were biopsied, and endocervical curettage was performed, if necessary. Biopsy-confirmed cases were considered eligible for this study.

Demographics and medical histories, including age, gravidity, parity, and menopause status were also obtained using digital medical records. Patients with incomplete information were excluded. We also excluded patients with a history of cervical physical therapy, surgical operations, and pelvic radiotherapy.

The Lower Anogenital Squamous Terminology (LAST) system was implemented and includes normal, LSIL, HSIL, and invasive cancers. All histology slides from punch biopsies, excision specimens and/or endocervical curettage were reviewed by histologists in the respective local hospitals. Any disagreements were resolved by a panel of independent expert histologists. The worst grade of dysplasia present was considered the final diagnosis.

This study was approved by Institutional Review Boards (IRB) in each of the participating hospitals. Research was conducted in accordance with the Declaration of Helsinki although, the requirement for written informed consent was waived by the respective IRBs because data were anonymized and retrospectively analyzed.

Potential predictive variables

All risks factors, considered potentially useful for predicting HSIL + development, were identified through key articles [8, 9, 13]. Additional factors recommended by senior clinicians were considered for inclusion in the predictive model. The following assessable predictors emerged as consistent indicators i.e., age, cytology results, HPV status, TZ types and colposcopic impressions.

Considering the nomogram was designed for general practice, we also intercalated gravidity, parity, menopause, cervix visibility, size of cervical lesion area into the prediction model after initial analysis. Classified cytology results were also included in more detail, including those negative for intraepithelial lesions or malignancy (NILM), atypical squamous cells of undetermined significance (ASC-US), LSIL, atypical squamous cells which did not exclude high grade squamous intraepithelial lesion (ASC-H), and HSIL+, and HPV status included HPV negative, non-16/18 HPV positive, and HPV16/18 positive. The coding of these variables has been provided in the supplementary materials as Table S1.

Outcome definitions

HSIL+, including HSIL and invasive cancers, were defined according to the LAST system, which are the most widely accepted, current international guidelines [15, 16]. The outcome of the predictive model was based on patient risk for developing HSIL + upon colposcopic examination. We used HSIL + as a hypothetical biopsy threshold in order to provide an additional margin of safety against misclassification. Therefore, we did not further distinguish between cervical intraepithelial neoplasia grade 2 (CIN2) or CIN3 among HSIL cases, even though many CIN2 cases are destined to regress [17]. Diagnoses based on HSIL + were determined by local histologists. If histological diagnosis or grade were not determined during the review, the case was referred to senior histologists, who were blinded to all other potential predictors.

Model development and validation

Model development and validation were performed (and reported) according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) guidelines [18]. The complete dataset was sampled randomly according to the distribution of histological results (HSIL + as opposed to < HSIL). Samples were then assigned to a training set for model development or to an internal validation set to assess performance at an approximate ratio of 7:3. This step was implemented to ensure comparability between the datasets.

Model development and internal validation were conducted in three stages. First, 10 variables were entered into the selection process. Then, least absolute shrinkage and selection operator (LASSO) was applied to reduce the number of candidate predictors and to select the strongest predictors to construct the model [19]. This LASSO approach penalizes the absolute size of model coefficients according to the value of λ. With larger penalties, the estimates of weaker factors shrink toward zero therefore, only the strongest predictors were accepted into the model.

Predictors identified using LASSO regression were entered into a multivariate logistic regression model. Those which were consistently statistically significant were used to construct the final prediction model. Multivariable logistic regression was used to calculate regression coefficients, and predictive strength was quantified as odds ratios (OR) with 95% confidence intervals (CI).

The model was calibrated to assess performance during development, and throughout internal and external validation. We also calculated discriminabilty to determine which patients were likely to develop HSIL+. Decision curves were generated to determine whether this model has clinical usefulness. Model calibration was assessed graphically using calibration plots, which intercalates predicted plots versus observed results. Model discrimination was determined by generating area under the curves (AUC). Decision curve analysis was performed for clinical use with a higher net benefit indicating enhanced clinical usefulness.

Model presentation

A nomogram is a graphical calculation device which provides a probability of a specific outcome, based on the overall effect of pertinent factors. Therefore, nomograms can potentially provide individualized, evidence-based, reliable, risk estimates. The validated prediction model which included statistically significant risk factors was presented as a regression equation which was then converted into a nomogram. Each predictor value in the nomogram was assigned a regression weight so that the total score is equivalent to a linear predictor. For this model, logistic transformation was applied to the linear predictor to produce probabilities for developing HSIL+.

Statistical analysis

Sample size was based on available data because there was no standard way to calculate sample size in advance. There were 10 candidate predictors and over 28 events per variable. This was considered sufficient to develop a stable model. Histological diagnosis was taken as golden standard. Receiver operating characteristic (ROC) curves were created by plotting the true positive rate (i.e. sensitivity) against a false positive rate (1-specificity). AUC values were generated for further comparative analysis.

Diagnostic indexes including accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated using 95% CIs with different cut-off values selected using the Clopper-Pearson method [20]. All statistical tests were two-sided and 0.05 was established as the threshold for statistical significance. Data analyses were conducted using Stata (version 15.0) and R (version 3.6.1).

Results

Basic characteristics of the study population

Throughout development and during internal validation, we identified 6,099 patients who had undergone colposcopic examination with biopsy in one of the four hospitals. Of these, 245 patients were excluded according to our predefined exclusion criteria. 5,854 patients were included, with 1,468 HSIL + cases and 4,386 < HSIL + controls. Please see Fig. 1 for a flowchart of patient selection and assignment.

Fig. 1

Flowchart for the development and validation of predictive model. (HSIL + cases included high-grade squamous intraepithelial lesion (HSIL) and invasive cancer. <HSIL controls included normal and low-grade squamous intraepithelial lesion (LSIL))

Both external validation datasets were collected from April to November, 2021, to help ensure the generalizability of findings. 472 consecutive patients and 422 patients who had undergone colposcopy with biopsy were enrolled. The prevalence of HSIL + and the distribution of observed risk factors (except for those in menopause) were significantly different between the development dataset and the internal validation dataset. Sample characteristics are provided in Table 1.

Table 1 Demographics and clinical characteristics of study population

Variable selection and model development

10 predictors measured in colposcopy clinics were included for LASSO regression modelling. Table S2 shows results from the candidate variables included in LASSO regression with corresponding coefficients for different values across the penalty parameter, λ. A lambda.min of 4.483 was observed, and all 10 predictor variables remained.

By increasing the λ value (to enhance shrinkage), we observed a lambda.1se of 32.262, with six predictors remaining. These were used to create the final model for validation. See Figure S1 for further details. The six remaining predictors were independent, statistically significant predictors for HSIL+, and were included in the logistic regression model, and for risk score calculations. The final model included age stratifications, cytologic results, HPV status, TZ types, colposcopic impression, and size of lesion area (see Table 2).

Table 2 Multivariate analysis of included predictors selected by LASSO regression procedure for detecting HSIL + in the development set

Model performance: discrimination, calibration and decision curve analysis

The discriminating performance of the predictive model in training, and under internal and external validation can be seen in Fig. 2. The AUCs based on data from the training and internal validation sets were 0.91 (95%CI 0.90–0.92) and 0.92 (95%CI 0.90–0.94), respectively. The AUCs for HSIL + risk in external validation sets 1 and 2 were 0.91 (95%CI 0.88–0.94) and 0.88 (95%CI 0.84–0.93), respectively. Overall, AUC analysis indicated good model discriminability across all external datasets.

Fig. 2

Discrimination performance of predictive model in the training, internal and external validation sets. (AUC = area under the curve)

Table S3 shows the performance of indexes i.e., sensitivity, specificity, accuracy, PPV, and NPV, for each probability as a cut-off value in identifying individuals at risk of HSIL+. The resultant model prediction scores from 0 to 1, were categorized to balance sensitivity and specificity for detection of HSIL+. We also accepted a lower cut-off point of 0.10 with 92.4% sensitivity and 60.3% specificity for cases who did not require very frequent follow-ups. A higher cut-off point of 0.55 with 95.4% specificity and 70.4% sensitivity was selected for cases which required strict and timely follow-ups.

Figure 3 provides calibration plots for observed frequencies and predicted probabilities for the model in the training, internal and external validation sets. Overall calibrations (E:O, the observed divided by expected number) in each dataset were 1.00, 1.05, 0.80, and 1.10, respectively. AUCs based on data from both the training and internal validation sets were 0.91 (95%CI 0.89–0.92) and 0.92 (95%CI 0.91–0.93), respectively. Again, good model discrimination was observed based on comparable intercepts for predictive models under external validation sets with 0.91 (95%CI 0.87–0.94) and 0.88 (95% CI 0.84–0.93), respectively. This suggests the model is highly stable and has a low level of over-fitting.

Fig. 3

Calibration plots of observed frequency and predicted probability for the predictive model in the training, internal and external validation sets. (E:O, the observed divided by expected number, with a number close to 1 showing good model fit; CITL, calibration-in-the-large; AUC, area under the curve)

Severity scores for predictive model were also analyzed with validation sets for a total of 24 invasive cancer cases. More complete data are provided in Table S4. Figure 4 provides decision curve analyses (DCA) for the training, internal and external validation sets. The horizontal and ordinate axis in this figure represents the threshold probability, and the net benefit after advantages were subtracted and counterbalanced according to disadvantages. When a patient’s risk of HSIL + reached a certain threshold, they were defined as high risk and biopsy measures were taken to confirm final diagnoses. Under training, and for internal and external validation sets, DCA found higher net benefits than biopsy for all patients, which indicates that the model developed is potentially clinically useful.

Fig. 4

Decision curve analysis depicting net benefit derived from the training, internal and external validation sets. ((A) Training set; (B) Internal validation set; (C) External validation set 1; (D) External validation set 2. The horizontal and ordinate axis of this figure represented threshold probability, and the net benefit after the advantages were subtracted by the disadvantage, respectively. When a patient’s HSIL + risk reached a certain threshold, it was defined as high risk and biopsy measures were taken to further confirm final diagnosis. Decision curve analysis showed higher net benefit than biopsy for all patients, which suggests this model developed in this study is clinical usefulness)

Nomogram presentation

Figure S2 and Table S5 provide the weighted nomogram and risk prediction scores which generates an individual’s probability of developing HSIL+, at the colposcopy stage. Each predictor was assigned a specific grading value. When all six predictors were determined, the total number of points was calculated by adding all points together. The probability of developing HSIL + was then determined using a total point scale. Several practical colposcopy examples are provided in additional file.

Discussion

In cervical cancer screening programs, colposcopic examination with biopsy has been widely accepted as a standard procedure for those with abnormal screening results. Colposcopy-guided biopsy decisions are crucial for deciding whether to follow-up, or to increase surveillance and for discussing further interventions. Taking multiple directed biopsies to improve HSIL + detection has also been widely accepted and applied in clinical practice [21, 22]. However, the decision to biopsy remains controversial among clinicians. In order to avoid missing HSIL + cases, clinicians may also prefer to biopsy, even in more unlikely instances. As has been mentioned, this can cause considerable psychological stress, which is in some instances totally unnecessary. There are also financial implications to excessive testing which increases the cost of cervical cancer screening programs [23]. Clearly, the one-size-fits-all approach is no longer appropriate. Therefore, we should make every effort to develop multivariable models which can individualize HSIL + risk estimates for colposcopists.

Here, we developed a novel predictive nomogram to identify patients who are likely to develop HSIL+. The idea was to develop and validate a model to be used with colposcopy to improve decisions around biopsy. Our model appears to have good discrimination and calibration when identifying those at high-risk of developing HSIL+. Findings were also internally and externally validated. These results were further supported by decision curve analysis, which showed a higher net benefit across almost the entire range of probability thresholds. The predictive model developed here included six statistically significant predictors, i.e., age, cytology results, HPV status, TZ types, colposcopic impression and size of lesion area. Each predictor used to calculate the risk of developing HSIL + are readily available to colposcopists, which adds to its applicability. If a patient’s estimated risk for developing HSIL + is low, clinicians may choose a wait-and-watch approach with follow-ups. For those with high-risk estimates, physicians should refer patients to colposcopic-guided biopsy for diagnosis; however, there may be opportunities to further develop this model.

More balanced cut-off values for detecting HSIL + might be selected to limit excessive biopsies although, there is a trade-off because sensitivity would likely drop. Another possibility for improving specificity while retaining high sensitivity might be to combine novel screening tests such as E6/E7mRNA, p16/Ki-67, or E6 oncoproteins. Two cut-off values have been suggested and depend upon the frequency of follow-up required. Given expert’s opinions and the findings from ROC curve analysis, we accepted a lower cut-off value of 0.10 with a 92.4% sensitivity and specificity of 60.3% for cases which did not require very frequent follow-ups. We also accepted a higher cut-off value of 0.55 with a 95.4% specificity and 70.4% sensitivity for those cases where strict and timely follow-ups were required. This would provide reassurance in negative results which can be shared with patients while reducing the number of unnecessary biopsies. Moreover, we should acknowledge that the predictive model commonly existed under-predicted risk in a population other than the population in which the model was developed. This is referred to as external validation sets. Variance in diagnostic performances in each external validation dataset may be attributed to the composition of the study sample, disease prevalence, clinical endpoints, and differences between colposcopic devices and/or approaches to biopsies. Each of these factors may cause bias which influences our ability to generalize model findings. Also, differences across populations highlights the need for more large-scale, multi-center training datasets, which could help to improve model generalizations and apply findings to different populations.

Clinical prediction models have been explored to predict cervical lesions, but few have been constructed for HSIL + predictions. Wu et al. [24] created and validated a logistic regression model for support vector machine (SVM) learning based on a multicenter cohort study of cervical cancer screening in China. Likewise, Karakitsos et al. [25] developed machine learning methods based on cytology, HPV status, E6/E7 mRNA test, and p16 immunostaining to build an algorithm to facilitate the classification of cervical intraepithelial neoplasia grade 2 or worse (CIN2+). Kahng et al. [26] developed an SVM model using age, cytology and presence of 15 HPV genotypes to identify the patient features that maximally contributed to progression to cervical precancers. Branca et al. [27] constructed comprehensive multivariate models by a panel of 13 biomarkers to predict CIN2+. However, these models have a number of different purposes and include different data sources, population characteristics, risk factors, and model performances varied, as did the extent of validation.

The aforementioned models were not developed for colposcopy, and included unattainable biomarkers such as E6 oncoprotein and p16/Ki-67, which are difficult to quantify in clinical practice. Recently, Li et al. [28] developed and validated a predictive model for endocervical curettage decision-making in cervical lesions based on colposcopic scenarios. Their prediction model intercalated screening results, TZ types, and colposcopic features which seem to be more feasible for colposcopic practice than previous studies suggest. However, their model has not been used for predicting cervical precancer risk during colposcopic examination. Other limitations of the previously published models were that they were not externally validated, and they developed models based on relatively small numbers of patients. This study was an attempt to fill this current knowledge gap by developing and externally validating a model which intercalates multiple predictors associated with colposcopy practice.

Previous studies [12, 29] have also found that combining these predictors could be used to identify patients who are likely to develop HSIL + for colposcopic biopsy. Although our study, is an attempt to determine risk at an earlier stage in the screening-diagnosis processes. We found the size of lesion area is an important predictor for detecting HSIL+. This is seemingly logical, yet it has not been described in the established guidelines, nor has it been factored into previous models. This means our model reduces the subjectivity involved in biopsy decision-making and is more appropriately evidence-based and provides a more reliable predictive tool. This study is the first to develop and validate a predictive model, which incorporates statistically significant variables as key predictors and thus advances our knowledge in the field.

Moving forward, the diagnostic accuracy of colposcopy with biopsy in the post-HPV vaccination era has become more unpredictable than ever. We must pay more attention to maintaining and optimizing the diagnostic accuracy of colposcopy with biopsy. Therefore, more scientific and technological research should be developed to improve diagnostic performances, especially in LMICs where there is a shortage of trained clinicians. In order to attain the WHO’s goal of eliminating cervical cancer worldwide by 2030 [30], we must adapt to advances and become early adopters of newly validated technologies. The findings from this study enable us to provide several recommendations for further research. Colposcopic practice needs to be further refined. Increasingly, evidence suggests that partial HPV genotyping types besides HPV16/18 carried a high risk of HSIL + that should be directly referred to colposcopy [31, 32].

We are witnessing an increasing number of innovations in colposcopic technologies. Novel colposcopy tools such as artificial intelligence (AI) guided digital colposcopy could help inexperienced clinicians standardize diagnostic procedures and to improve the accuracy of biopsy [33,34,35]. Although, these may not necessarily be readily available in LMICs and in areas where resources are unequally distributed. We are currently trying to build an interpretable cloud-based AI platform by combining our predictive model with a well-developed Colposcopic Artificial Intelligence Auxiliary Diagnostic System (CAIADS) from previous studies [36, 37]. It is hoped this will provide accessible telemedical assistance for LMICs. Although again, high-quality colposcopy training should be established to enhance cervical cancer diagnostics in LMICs. International colposcopy organizations should also provide continuous, updated, and mandatory accreditation to all colposcopy professionals, in order to provide the public with expertise and professionals with the support to fulfil their roles.

Predictors are readily available from digital clinical records in colposcopy clinics although, none have used LASSO regression analysis for predictor selection, and multicenter datasets for model development, and external validations to assess a model’s generalizability.. Therefore, this study had a number of advantages and undoubtedly adds to the evidence-base. One limitation of this study is that the model’s development was based on Chinese mainland women only, which may limit the generalizability to other populations. Other limitations include the computational complexity of the model and that our results were diagnosed by local senior clinicians. So, further research is needed to ensure this tool is easily applied and useful for less experienced clinicians. Finally, this study lacks a head-to-head comparison with clinicians due to the retrospective nature of this study design. The current model has the potential to help both patients and their providers although, further prospective clinical research would be useful to validate the effectiveness of model-assisted clinicians, generally.

Conclusion

We developed and validated a prediction model by incorporating multiple clinically relevant variables to improve HSIL + case identification during colposcopic examination. This may help clinicians making decisions around colposcopy-guided biopsy procedures; however, further global, prospective research should be conducted before adopting this tool into clinical practice.