Pathology diagnosis of intraoperative frozen thyroid lesions assisted by deep learning

Abstract

Background

Thyroid cancer is a common thyroid malignancy. The majority of thyroid lesion needs intraoperative frozen pathology diagnosis, which provides important information for precision operation. As digital whole slide images (WSIs) develop, deep learning methods for histopathological classification of the thyroid gland (paraffin sections) have achieved outstanding results. Our current study is to clarify whether deep learning assists pathology diagnosis for intraoperative frozen thyroid lesions or not.

Methods

We propose an artificial intelligence-assisted diagnostic system for frozen thyroid lesions that applies prior knowledge in tandem with a dichotomous judgment of whether the lesion is cancerous or not and a quadratic judgment of the type of cancerous lesion to categorize the frozen thyroid lesions into five categories: papillary thyroid carcinoma, medullary thyroid carcinoma, anaplastic thyroid carcinoma, follicular thyroid tumor, and non-cancerous lesion. We obtained 4409 frozen digital pathology sections (WSI) of thyroid from the First Affiliated Hospital of Sun Yat-sen University (SYSUFH) to train and test the model, and the performance was validated by a six-fold cross validation, 101 papillary microcarcinoma sections of thyroid were used to validate the system’s sensitivity, and 1388 WSIs of thyroid were used for the evaluation of the external dataset. The deep learning models were compared in terms of several metrics such as accuracy, F1 score, recall, precision and AUC (Area Under Curve).

Results

We developed the first deep learning-based frozen thyroid diagnostic classifier for histopathological WSI classification of papillary carcinoma, medullary carcinoma, follicular tumor, anaplastic carcinoma, and non-carcinoma lesion. On test slides, the system had an accuracy of 0.9459, a precision of 0.9475, and an AUC of 0.9955. In the papillary carcinoma test slides, the system was able to accurately predict even lesions as small as 2 mm in diameter. Tested with the acceleration component, the cut processing can be performed in 346.12 s and the visual inference prediction results can be obtained in 98.61 s, thus meeting the time requirements for intraoperative diagnosis. Our study employs a deep learning approach for high-precision classification of intraoperative frozen thyroid lesion distribution in the clinical setting, which has potential clinical implications for assisting pathologists and precision surgery of thyroid lesions.

Background

Pathological diagnosis provides a theoretical and practical basis for the diagnosis, treatment and prevention of diseases through the study of the causes and pathogenesis of diseases and the morphological structure of the diseased organism during the disease process. The production of hematoxylin and eosin (H&E) sections usually takes about three days to get the diagnosis result from the initial taking of specimen [1]. The number of pathologists is limited globally [2], and the lack of pathologists in hospitals has resulted in senior pathology doctors completing residents’ work. Nevertheless, the surgeon needs to know the information about the lesion as soon as possible to adjust the surgical plan. This requires rapid sectioning to obtain efficient diagnostic results from pathologists [3], and intraoperative frozen sectioning is the most widely used method [4]. It is a procedure in which the tissue cut during surgery is frozen in a frozen sectioning machine to allow for rapid cooling of the tissue through low-temperature conditions and then for production. Frozen sections are obtained during the process and are a significant guide to surgical treatment such as benign or malignancy for the patient’s lesion which might not match the original diagnostic surgical plan, and then affect the operation procedure [5]. On the contrary, many primary care hospitals have limited clinical experience in issuing frozen pathology reports because of a severe shortage of pathologists, which often makes it difficult to meet clinical needs.

Over recent years, the application of deep learning artificial intelligence approaches in digital pathology has shown excellent results, both in the broad application of different tissue diseases such as liver cancer [6, 7], breast cancer [8,9,10] and esophageal cancer [11, 12] and in the optimization and updating of cellular tissues in detection [13], segmentation [14] and classification [15] methods, all of which show great potential. For example, deep learning methods enable networks for Digital whole slide images (WSIs) lymphocyte measurement and segmentation tasks simultaneously [14]; artificial intelligence approaches for lung cancer histopathology classification by supervised or weakly supervised strategies [16]; deep learning for prediction from H&E images of follicular thyroid tumors [17]; and differential diagnosis of follicular thyroid tumors based on histopathological images using deep learning techniques [18]. However, existing deep learning methods for histopathological classification of thyroid cancer have been studied with paraffin-embedded H&E sections WSIs, but ignore the significance of intraoperative frozen sections. In this study, we attempt to develop a deep learning-based five-class classifier for identifying a broader and more detailed range of thyroid lesions. We refined the frozen sections’ analysis into five classification problem classes through differences in the histological characteristics of different cancers, including papillary thyroid carcinoma (PTC), follicular thyroid tumor (FTT), medullary thyroid carcinoma (MTC), anaplastic thyroid carcinoma (ATC), and non-cancerous thyroid lesion (NTC). ResNet [19], multi-processing, TensorRT [20], and graphics processing units (GPUs) were used to improve efficiency. To validate the performance, efficiency, and sensitivity of clinical applications, comparative model experiments, prior knowledge cascade experiments, time-cost acceleration analyses, and visualization of the effects of applications were carried out.

Due to the high resolution of digitized frozen slices, which can reach 100,000 × 100,000 pixels, direct complete WSIs processing with large-scale down-sampling loses many detailed features, which is not conducive to a classification approach regarding morphological features. Also training a convolutional neural network (CNN) on the whole slide tissue image (WSI) is computationally tricky to achieve. Cropping WSIs into tiles for analysis is a standard solution for studying histopathological WSIs by training a tiles-level classifier and later performing tiles fusion in an Expectation-Maximization (EM) manner to summarize the prediction results and obtain an effect map of the whole slice prediction [21]. Wei et al. [22] used sliding windows over the entire slide to generate small tiles and classified each tile by a neural network. They used heuristics to determine the primary and secondary histological patterns across the slide so that labels for different regions could be obtained.

In this study, the high accuracy performance and interpretability of our proposed system served as our primary goal. Against this backdrop, we ensured the accuracy of the tissue regions of candidate training tiles by double manual annotation of regions of interest (ROIs) by pathologists and pathologists. Then we segmented the localized tissue regions into small tiles and simulate real clinical diagnostic scenarios by a cascade diagnostic system to diagnose the presence of a cancerous lesion and subdivide the type of the cancerous lesion to predict the category of each tidbit separately. The final classification criterion was a fusion recovery of the entire tiles of the pathology slices, using a heuristic to identify the significant and minor histological patterns across the WSI. In addition, we confirmed the performance sensitivity of our system with test results of papillary thyroid microcarcinoma and different scale thermograms with accelerated plug-ins to speed up the diagnostic inference of the system. Our system not only allows for a more detailed classification of thyroid lesions but also allows for the system’s sensitivity to localize suspicious lesions, saving diagnostic time and cost and assisting the pathologist in making a diagnosis in copious ways. The main contributions of this paper can be summarized as follows:

• A tile-supervised WSI-label learning-based classification and diagnosis system for frozen thyroid lesions is designed. We propose a Two-step cascade diagnostic system (TSCD), which utilizes a priori knowledge cascading into a five-classification diagnostic system that fits the clinical diagnostic scenarios, and for the first time achieves fine-grained classification of lesions, realizes a high-accuracy classification task and demonstrating advanced performance.

• We conducted a systematic working test on papillary thyroid microcarcinoma alone. To ensure the accuracy of clinical application, we strictly selected cases with lesion sizes within 1 cm, and the results showed that a lesion size of 2 mm could be accurately localized.

• A diagnostic acceleration component is designed, this acceleration component accelerates the processing from multi-processing of data to TensorRT acceleration of model inference, reducing consumption time to 729.37% of the original speedup. In addition, this component can be embedded in all tiles-based classification inference models for broad applicability.

• A multi-scale thermographic visualization of the system is proposed to test the sensitivity, which is reflected by the selection of different scales of lesion areas from different cases to visualize the system’s identification results due to the high resolution of the WSI.

• For the consideration of the clinical application of the diagnostic system, a heuristic strategy of tumor prioritization based on thresholds is proposed to clarify the diagnostic process by thresholding the WSI primary and secondary tissues in a heuristic manner to promote the development of deep learning methods for the clinical application of frozen histopathological images.

• A database of frozen thyroid digital pathology sections was established, and the developability of deep learning in frozen sections was confirmed by designing a deep learning model to analyze data and assist in diagnosing pathology library data.

Methods

In this section, we will describe the preparation of data, the selection of deep learning models, composition of acceleration components, and the generation of the system for the final frozen diagnosis of thyroid lesions from several perspectives.

Slides collection and categorization

We collected frozen thyroid sections from the First Affiliated Hospital of Sun Yat-sen University to form a library of frozen thyroid digital pathology sections Four certified pathologists independently reviewed the organized frozen thyroid sections. The diagnosis was determined with the aid of the results of paraffin-embedded H&E sections and immunohistochemistry staining sections. In order to ensure the accuracy of the data, three rounds of screening were conducted sequentially through diagnostic pathology reports, four pathologists, and two pathologists in the process of selecting the data, and finally the data that were all approved were selected. Figure 1 shows the morphology of five types of typical thyroid sections, Fig. 2 shows their high magnification histopathology, and Table 5 lists the clinical management of different thyroid lesions. It can be seen that differentiating the types of thyroid cancer can facilitate efficient and targeted clinical diagnosis and help patients heal better. This study was approved by the Ethics Committee of the First Affiliated Hospital, Sun Yat-sen University.

Fig. 1

Frozen section examples WSI, A for papillary thyroid carcinoma nodule, B for medullary thyroid carcinoma nodule, C for follicular tumor nodule, D for anaplastic carcinoma nodule, and E for all non-cancerous frozen section slides WSI, from top to bottom, for nodular goiter, chronic lymphocytic thyroiditis, and subacute thyroiditis

Fig. 2

High-magnification histopathology, a listed as WSI at 20x magnification, b listed as WSI at 40x magnification, and c listed as WSI at 80x magnification, from top to bottom, PTC, MTC, FTT, ATC, NG, CLT, ST

Thyroid cancer is divided into papillary thyroid cancer, medullary thyroid cancer, anaplastic thyroid cancer and follicular thyroid cancer. However, if a follicular thyroid tumor is determined to be benign and less than 1 cm in diameter, it can be treated with an observation instead of follow-up surgery [23]. Follicular thyroid cancer which is malignant follicular thyroid tumor requires demonstration of capsular and/or vascular invasion. Ultrasound, puncture, intraoperative frozen pathology, and even postoperative paraffin-embedded pathology may only partially guarantee accuracy [24]. So there exists a tendency for various findings to be benign but may be malignant, so it is not easy to fully determine whether it is benign or malignant. Therefore, our classification tends to be divided into five parts: papillary carcinoma, follicular tumor, medullary carcinoma, anaplastic carcinoma, and non-cancer lesion, where the noncancer frozen section consists of data from nodular goiter, chronic lymphocytic thyroiditis (CLT), and subacute thyroiditis (ST) together.

WSI datasets

The database of frozen thyroid WSIs we created consisted of 4409 frozen thyroid sections from the First Hospital of Sun Yat-sen University (SYSUFH) and the First Affiliated Hospital of Sun Yat-sen University (FAH-SYSU), covering 3873 cases diagnosed as PTC, FTT, ATC, MTC or NTC, and the inclusion criteria for the dataset were that each frozen section was able to be included in the above categories. Before WSI annotation, all sections were observed by two senior pathologists at SYSUFH through microscopy, including immunohistochemically stained Sects. [25, 26] used as an aid to diagnosis. The diagnostic reports of all cases were also checked as a way to determine the accuracy of the collected data. Slides were then scanned with the Slide Scan Imaging System SQS-600P scanner at 40x magnification and digitized into SDPC format. For an unbiased evaluation, classification was performed according to the 5th edition of WHO thyroid tumor classification criteria. For the scanned digitized pathology slides, four pathologists first annotated WSI for regional ROI, and two pathologists then checked the annotated slides with at least seven years of clinical experience in the pathology department of SYSUFH to ensure the accuracy of the annotated information.

We collected three batches of frozen thyroid slices for database development, and all the data in the library were used in our system’s training validation and testing. It contained 4409 thyroid frozen sections (PTC, 2300; FTT, 225; MTC, 49; ATC, 7; NTC, 1904) for model construction, randomly divided into training and testing 11 groups in a 10:1 ratio. Of these, 2894 (PTC, 1479; FTT, 224; MTC, 49; ATC, 7; NTC, 1135) were used to construct the model, 127 (PTC, 127) were used to evaluate the sensitivity of the model to small cancers, and 1388 (PTC, 694; FTT, 1; NTC, 769) were used in the external test set to evaluate the system’s generalization ability. The number of follicular variant of papillary thyroid carcinoma (FVPTC) cases was 135, containing 163 WSIs, or 3.49% of 3873 cases, and 3.70% of 4409 WSIs, and non- invasive follicular thyroid neoplasm with papillary-like nuclear features (NIFTP) The number of cases was 11, containing 14 WSIs, representing 0.28% of 3873 cases and 0.32% of 4409 WSIs.

Based on the patch cropping method, 105622 papillary carcinoma patches, 42644 follicular tumor patches, 27491 medullary carcinoma patches, 8507 anaplastic carcinoma patches, and 215418 non-cancerous lesion patches were cropped from the training slides used for system construction. Of these, 78343 papillary carcinoma patches, 42407 follicular tumor patches, 27491 medullary carcinoma patches, 8507 anaplastic carcinoma patches, and 118895 non-cancerous lesion patches were cropped from the training slides used for model construction and the rest of the patches used for testing, as shown in Table 1.

Table 1 Statistical information of the dataset used for model development testing. The dataset was categorized into three subsets, dataset A, dataset B and dataset C, depending on the purpose

Data pre-processing

The pathology slide scanner used to obtain the WSIs was the SQS-600P slide scanning imaging system from Shenzhen Shengqiang Technology Co. (Shenzhen, China). The labeling software is also the digital pathology reading software ImageViewer provided by Shenzhen Sangqiang Technology Co., Ltd, version DPVIEW V2.0.1.0927. We use a supervised approach to train the model, which requires us to annotate the ROI region carefully.

The pathologists manually annotated the specific histopathological thyroid tissue types in each WSI by using colored irregular shapes, and the annotated WSIs were reviewed by two pathologists to ensure the accuracy of the annotation. The colored irregularly annotated regions were cut at the tile level after ROI region annotation to provide the tissue details of the cut patches, as shown in Fig. 3. Different categories of tiles obtained by cropping are shown in Fig. 4. All processing was performed at a magnification of 40x, based on a pixel scale of 0.2065 μm. We traversed all manual annotations by reading the SDPL file (the file generated by manual annotation). Use the multiplicity of the rectangle marked in the blank area as the reference coordinate system, and calculate the ratio of the reference coordinate system to the absolute coordinate system using the screen position of the colored outline, and then deflating the annotated points to the reference coordinate system, taking the center point of the annotation as the distribution of tiles is described in Table 1.

Fig. 3

Diagram of the process of cropping WSI into tiles. a shows the WSI of the original frozen thyroid nodule, and b shows the annotation of the contours by the pathologist, the green contour is the ROI area, i.e., the area with lesions and non-cancerous areas can also be present in the slides with lesions. figure (c) shows the tiles obtained by cropping from the annotated section, the number of patches obtained by cropping is proportional to the area of the annotated area

Fig. 4

Different categories of tiles obtained by cropping, A for PTC, B for FTT, C for MTC, D for ATC, and E for NTC, with a pixel size of 224 × 224

Deep neural networks

According to the research content and its applicability, what we need is a neural network framework with high accuracy while ensuring low cost. The more commonly used neural network backbones include ResNet^19, DenseNet [27], EfficientNet [28], ShuffleNet [29], etc. In addition, with the extension of the transformer to the computer vision (CV) field, the generic backbone network Swin-Transformer [30] is also showing superior performance in vision tasks such as image generation, medical image segmentation, etc. performance. We choose ResNet101 as the backbone network for histopathology classification tasks. The residual structure of ResNet is proposed to solve the problem of network degradation, and the gradient disappearance or explosion has been translated by normalized initialization, etc. The model can make learning less and more manageable, thus ensuring sure accuracy while reducing the cost. And Resnet101 is used for our histopathology classification task by replacing the last fully connected layer with five outputs representing papillary thyroid carcinoma, medullary carcinoma, follicular tumor, anaplastic carcinoma, and non-carcinoma lesion, respectively. For training, testing and optimization, we divided all the datasets into a training set, a validation set, and a test set. Although we used more than 399682 patches of data for system development, we still trained the thyroid patch model by transfer learning from models pre-trained on the ImageNet recognition task. By initializing the pre-trained model, the patch classifier can achieve better performance. At the same time, we also fine-tune models such as Swin Transformer, EfficientNet-B5, and GoogLeNet [31] using the same data and settings as Resnet101 and also test the models using the same test set to make qualitative comparisons among different kinds of model results.

A priori knowledge and classification cascade

In image classification tasks, a priori knowledge can be obtained by pre-training on large-scale image datasets [32]. Through pre-training, the model can learn the basic features and patterns of the images, and the introduction of a priori knowledge as pre-training weights can effectively improve the accuracy and efficiency of image classification. Modeled after the actual diagnostic process of clinical thyroid freezing, the TSCD system was designed as a categorization cascade of two classification cascades for the presence of cancerous lesions and a categorization cascade of four classification cascades for subdivided lesion types. However, the pre-training dataset ImageNet [33] used now has a feature extraction method that does not fit the pathology image, and the training data of the two parts of the cascade model is not as much as that of the five-classification model, which is insufficiently fine-grained for the classification task. In order to make the cascade model have similar feature extraction ability as the five-classification model and fit the pathological images of thyroid lesions, the five-classification baseline model’s is used as the pretrain model of the cascade model.

Network training and implementation details

Our training, validation and testing data are not publicly available due to strict controls on medical management and strict policies on sample privacy. Even though we have acquired several 399062 tiles for the model training process, we still apply transfer learning to our network model training to obtain better performance. By initializing the network with the default weights transferred from the ImageNet dataset, the entire network is fine-tuned to fit our data target better. The initial learning rate is set to 0.00003 and the optimizer is Adam. Momentum and attenuation are both set to 0.9. dynamic data enhancement, including horizontal flip or in the vertical direction, color dithering includes automatic adjustment of luminance in the range of (0.65, 1.35), automatic adjustment of contrast in the field of (0.5, 1.5), automatic adjustment of automatic saturation adjustment, and 0.3 size adjustment for hue to increase the data variation. To improve the learning characteristics of convergence, pixels were rescaled from 0 to 255 to 0–1 by dividing by 255, and the Z-score was normalized using the mean (0.485, 0.456, 0.406) and STD (0.229, 0.224, 0.225). The training process lasts 40 epochs, and the optimization model with the most negligible loss is saved and used. We used the deep learning framework PyTorch 1.10.2 to implement all CNN models, and an NVIDIA GeForce RTX 3090 GPU with 24 GB of memory was used for CNN model training and evaluation. Training ResNet18, ResNet34 models took about 32 h, and Resnet50 model took about 44 h, and ResNet101 model took about 48 h. GoogLeNet model took about 28 h. Swin Transformer model took about 67 h. EfficienNet-B5 took about 47 h. Vision-Transformer took about 88 h. AlexNet, DenseNet121, VGGNet-16, ShuffleNet-v2 models took about 16 h, 36 h, 43 h and 25 h.

Tile splitting and prediction

For the sake of achieving the integrity of tissue details and clarity of morphological features, we cut the labeled area into non-overlapping tiles of 224*224 pixels in size. Tiles were filtered with the criterion of 50% or more of class-labeled components to obtain a dataset with a high degree of lesion fit from both WSI and tile-level perspectives. We use our deep learning system to perform classification prediction for these tiles from manually labeled regions. After obtaining the initial tile-level class labels, we perform WSI fusion according to our tiles fusion strategy to get the WSI-level label prediction, which is the auxiliary lesion diagnosis result. Figure 5 illustrates the diagnostic flow chart of a whole frozen thyroid section.

Fig. 5

Deep learning-based diagnostic workflow for frozen thyroid lesions. The pipeline contains five modules: a WSI pre-processing section, b Resnet101 network structure, c optimal system experimental results, d WSI reconstruction visualization, and e Grad-CAM (Gradient-weighted Class Activation Mapping) heat map visualization

Whole-slide label inferencing with tiles fusion

Since the network outputs are tile-level predictions, the final diagnosis must be integrated into WSI-level forecasts. In general, the prediction results of slice-level WSI are aggregated to decide the classification based on tiles, which depend on the category to which the maximum probability belongs. Classical integration methods include averaging [34] and voting [35] procedures, which use more straightforward strategies to combine the predictions of individual learners. The majority voting method [36] is based on the classification results of a single classification model. It uses the principle of minority rule to determine the category labels predicted by the model, that is, by the number of tiles in each category and assigning the entire WSI to the one label with the highest number of corresponding types. The other is the average pooling method, where the probabilities of each category are summed and the slide labels are derived from the maximum average class probabilities [37].

In our data, tissue components of non-cancerous categories and other tumors may coexist in one WSI. For example, papillary thyroid cancer and non-cancerous parts are distributed in different regions of the same WSI. Still, the final label of one WSI is attributed to only one category. Inspired by the work of Li et al. [38], we propose a threshold-based approach to compound the results based on the majority voting method incorporating a threshold division of pathological tissue priorities. When we encounter multiple tiles class labels co-located in a single WSI, equal treatment of tumor-containing and non-cancerous WSIs will result in the neglect of microscopic cancerous lesions and serious misclassification of the patient’s diagnosis, such that the results we predict will not fit the requirements for putting into practical clinical use. Therefore, to improve the priority level of tumor-containing WSIs and enhance the sensitivity of the system to tumor regions, we propose a heuristic strategy based on threshold tumor priority, as shown in Fig. 6. Firstly, we set the threshold value according to the severity of lesions, because PTC, ATC, FTT, and MTC are all tumor types except the NTC category, so the threshold value of NTC is set to 95%. In contrast, all the rest of tumors are not differentiated by a threshold value. In addition, the pixel size of the tiles we selected was 224 × 224. Such a size is small even when compared with the lesion diameter of papillary thyroid microcarcinoma. As shown in Fig. 6, in our work for the prediction of WSI of papillary thyroid microcarcinoma with a lesion diameter ≤ 10 mm, the size of the lesion in the visualization result can be cropped to 20 tiles with continuity. Therefore, the category of this WSI was determined as NTC only when the weight of NTC exceeded a threshold value of 95% and the number of regional continuity tiles (RCT) was less than 20. In addition to the above, if there was a coexistence of NTC and other categories, all different categories were used as the main tissue component, and the same was valid for label selection. The task classification we implemented in this study is based on the visual features observable at the cellular or partial tissue level on the image tiles scale, so this strategy of fusing tiles is used.

Fig. 6

Athreshold-based tumor-first frozen section diagnostic method for the thyroid is proposed

Acceleration components

Due to the time-specific nature of intraoperative diagnosis, efficient diagnostic results need to be given in a short period of time, so we designed an acceleration component to enhance the diagnostic efficiency, and the component includes multiprocessing for cutting tiles and TensorRT to accelerate the inference.

Multiprocessing for cutting tiles

In the program of cutting tiles, multiple processes [39] can be run at the same time through the multi-process module, each process has its own independent execution space and resources, and can handle multiple tasks at the same time, here we use 9 processes, compared to the original time to get a single tile can now be obtained 9 tiles, in order to improve the concurrency and response speed of the system.

Diagnostic reasoning acceleration

PyTorch [40] is an open-source deep learning framework that can be used to build and train neural network models. ONNX (Open Neural Network Exchange) [41] is an open deep learning model exchange format that can be used to share and convert models between different deep learning frameworks. TensorRT [42] is an NVIDIA provided high-performance deep learning inference engine that can be used to accelerate the inference process of deep learning models. Our initial system was trained to obtain PyTorch models, and in model inference, to speed up the inference process, we converted Pytorch to ONNX files via the torch.onnx.export() function, after which we used the TensorRT Python API to build the TensorRT engine with ONNX models, as depicted in Fig. 7 conversion flow so that high performance inference can be achieved in TensorRT using PyTorch-trained models.

Fig. 7

TensorRT accelerates the PyTorch model inference process, which consists of three steps:(1) export the PyTorch model to an ONNX file, (2) build the TensorRT Engine, and (3) deploy the TensorRT Engine

Evaluation Metrics

In our experiments, the accuracy, precision, recall, and F1 score were used to evaluate the performance of our proposed method and the state-of-the-art techniques [43]. In the assay method, the actual category value and the prediction were the same and both were positive, then TP (True Positive); if both were negative, then TN (True Negative); the actual category value did not agree with the prediction and the prediction was positive, then false positive (FP) if the forecast was negative, then false negative (FN). Based on these basic definitions, additional evaluation metrics (Accuracy [A], Precision [P], F1-Score [F1], Recall [R], Specificity[S], TPR [True Positive Rate], FPR [False Positive Rate]) of the frozen thyroid tissue diagnostic test can be introduced as follows.

$$\:Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(1)

$$\:Precision=\frac{TP}{TP+FP}$$

(2)

$$\:\frac{2}{F1}=\frac{1}{P}+\frac{1}{R}$$

(3)

$$\:TPR=Sensitivity=Recall=\frac{TP}{TP+FN}$$

(4)

$$\:\text{S}\text{p}\text{e}\text{c}\text{i}\text{f}\text{i}\text{c}\text{i}\text{t}\text{y}\:=\frac{TN}{TN+FP}$$

(5)

$$\:\text{F}\text{P}\text{R}\:=1-\text{S}\text{p}\text{e}\text{c}\text{i}\text{f}\text{i}\text{c}\text{i}\text{t}\text{y}=\frac{FP}{TN+FP}$$

(6)

Results

Patch-based classification

In the twelve experimental ones of the Convolutional Neural Networks (CNN) model, the model’s performance is optimized by fine-tuning the model. Among them, Resnet34, Resnet50 and Resnet101 models perform close to each other in the fine-tuned training mode, but Resnet101 shows a slight lead. Table 2; Fig. 8 offer a qualitative comparison between the system backbone network Resnet and other classical deep learning models. These results compare and validate the superior results of selecting the Resnet34, Resnet50 and Resnet101 models for the classification of frozen thyroid lesions by transfer learning, and the final visualization of the whole slice prediction is in excellent agreement with the ground. The absolute consistency of the entire prophecy is also extremely high.

Table 2 Average test ACC, Precision, F1-Score, recall with standard deviation on test dataset

Fig. 8

Comparison of classification effects. It can be seen from the figure that Resnet34, Resnet50, Resnet101 are higher than other models in all four evaluation indexes

Three backbone models were selected for the cascade test since the performance differences on the five-category baseline of Resnet34, Resnet50 and Resnet101model were small. Figure 9 compares the evaluation of Accuracy [A], Precision [P], F1-score [F], and Recall [R] of the direct cascade with ImageNet as the pre-training model and the TSCD system with the five base classifications of thyroid lesions as the pre-training model under the three models metrics. The experimental results show that whether it is Resnet34, Resnet50 or Resnet101 model, the evaluation indexes from BASE classification to direct cascade to pretrain cascade system are incremental effect, as this can prove the validity of our proposed cascade system modeled on clinical diagnosis. Since the cascade of the five classifications of thyroid lesions as a pretrained model yielded more accurate results than the direct cascade using ImageNet as a pretrained model, and the overall performance of the cascade of Resnet101 outperformed that of the other models in the system, we ultimately chose Resnet101 as the model that constitutes the backbone of our TSCD system for diagnosing frozen thyroid lesions. The final test results of our TSCD system were 94.59% ± 3.55% in Accuracy, 94.75% ± 3.02% in Precision, 94.59% ± 3.44% in F1-score, 93.20% ± 2.16% in Recall, and the value of AUC was 99.55% ± 0.35% (Table 3).

Table 3 Average test ACC, Precision, F1-Score, recall with standard deviation on cascade system

Fig. 9

Better performing comparison in cascade system. The top three comprehensive performances of the above models are selected for a more visual comparison. a Comparison of the metrics performance of the three selected models on base classification and TSCD system. b Confusion matrix at the tile level predicted by the basis of the three models. c ROC curves and their AUC values for the three models on the base categorization and the TSCD system under cross-validation. It can be seen that Resnet101_pretrain_cascade is superior to other models in every index

Internal and external test sets

Internal and external test sets were selected to evaluate the generalization ability of the diagnostic system. The external test set mainly focuses on the two categories of papillary thyroid cancer and non-cancer, which is due to the imbalance in the distribution of thyroid lesions themselves, with papillary cancer accounting for a much larger proportion than other lesions. Therefore, we selected the binary classification model for the presence of cancer in the first part of the TSCD system to be used to test the effectiveness of the diagnostic system on an external test set. At the same time, we divided the internal data originally used to test the five-classification diagnostic system into two categories for the comparison test between the internal and external test sets. The test results are shown in Fig. 10. The accuracy of our model on the external test set reaches 99.37% ± 0.15%, and the accuracy of the internal test data is slightly lower than the external data set because the complexity of the internal data is greater than that of the external test set, but the overall effect is still as high as 97.62% ± 0.58%. The accuracy of our diagnostic system is 97.62%±0.58% for the binary classification of cancer or not, and 99.37%±0.15% for the external test set, which shows the strong generalization ability of our system.

Fig. 10

The first part of the TSCD system, i.e., the dichotomous classification of the presence or absence of cancer, was selected for the test. The first row shows the effects on the internal test set originally used to construct the diagnostic system, and the second row shows the effects on the external test set obtained from the East Hospital of the First Affiliated Hospital of Sun Yat-sen University. The effects on the four metrics Accuracy [A], Precision [P], F1-score [F], Recall [R], the confusion matrix, and the AUC performance are shown from left to right

Visualizing predictions with heatmaps

Grad-CAM (Gradient-weighted Class Activation Mapping) can assist in analyzing the regions of interest of the system for a given class, and we also verify whether we have learned the correct features or information through the regions of interest of the network in turn. We use Grad-CAM to draw the heat map of Fig. 11. From these heat maps, we can find that our system achieves accurate localization and recognition of the regions that we classify for attention.

Fig. 11

Grad-cam heat map visualization of tumor classification, a tile level visualization for papillary carcinoma, b tile level visualization for follicular tumor, c tile level visualization for medullary carcinoma, and d tile level visualization for anaplastic carcinoma

Slide-level performance

To obtain the predictions on the whole slide image, we mapped the system predicted tile-level results onto the original frozen sections, with different color blocks representing different categories. Figure 12 selected slices with different labels, the original WSI, the pathologist annotated WSI and the system slice-level prediction results are plotted, which visually shows the high agreement between our system classification prediction area and the pathologist annotated area.

Fig. 12

From left to right, the slide-level visualization (performance) of PTC, MTC, FTT, ATC, and NTC tissue predictions are shown. The first row shows the original slides, the second row shows the annotated WSIs with closed blue curves depicting the ROIs annotated by the pathologist, and the last row shows the corresponding WSI prediction visualization

Test on papillary thyroid microcarcinoma

To test the sensitivity of the system and for more accurate results in clinical use, we developed a sensitivity testing strategy for papillary thyroid microcarcinoma. we screened papillary thyroid microcarcinoma by lesion diameter and used them to test the system’s sensitivity. We screened 101 frozen WSIs of papillary thyroid microcarcinoma, whose lesion diameters were all within the range of 1 cm, and obtained more than 3000 tiles by cropping them and inputting them into the system for testing. And obtained visualization results displayed in Fig. 13, which can accurately identify the cancerous regions, verifying the accuracy of the system.

Fig. 13

Test of papillary thyroid microcarcinoma, the example of papillary thyroid microcarcinoma with lesion size of 2 mm in diameter, A original WSI, B manually labeled ROI, C Slide-level visualization display, the result can accurately identify the cancerous area

As can be seen from Slide-level diagnosis Fig. 13, our system can accurately identify cancerous lesions even in the face of lesions as small as 2 mm in diameter when testing papillary thyroid microcarcinoma cases.

Multi-scale Thermographic visualization

We proposed a multi-scale heat map visualization, which, due to the high resolution of WSIs, the lesion areas of different cases and different scales were selected, as in Fig. 14, we chose the WSIs of four papillary carcinoma cases and cropped the four WSIs at different scales, and the pixel sizes obtained were 224 × 224, 512 × 512, 1120 × 1120, 2240 × 2240 in order, and the results of the system in different scale tiles for grad-cam heat map visualization [44], reflecting the sensitivity of the system. Figure 14 shows the classification effect of our deep neural network system in the form of a Grad-CAM visual thermal map. Even when faced with tiles of different scales, the diagnostic system was able to pinpoint the area where the tumor was located.

Fig. 14

Grad-cam heat map visualization of different scales of tumor classification, illustration of papillary thyroid carcinoma. The pixel size of figure (a) is 224×224, figure (b) is 512×512, figure (c) is 1120×1120, and figure (d) is 2240×2240

Accelerated component results in diagnostic time cost

After the optimization, excluding the necessary time to read and store the image brought by the hardware, the average time to cut the whole WSI is reduced from 2899.89s to 346.12s, which is an 837.83% speed improvement. The average inference time of the model’s individual WSIs is reduced from 343.84s to 98.61s, which is a 348.69% speed improvement. The system’s overall inference and diagnosis time is reduced from 3243.73 s to 444.73s, an improvement of 729.37%, as shown in the details in Table 4. As intraoperative freezing has very strict time requirements, doctors need to accurately grasp the freezing time to ensure the safety and effectiveness of the procedure. The shortened time helps the effective intraoperative diagnosis, thus further improving the efficiency and success rate of the surgery.

Table 4 Time consumed by the diagnostic system under the acceleration components

The primary time cost of WSI analysis lies in the data processing and prediction results of the WSI and its visualization generation. the level down-sampling factor in the WSI pyramidal storage structure is 2, and the annotated ROI size is one-fourth of the slide height and width, Then the size of the ROI region is 10,000 × 10,000 pixels in the level 2 slide image, which corresponds to 40,000 × 40,000 in level 0 of the WSI, and the size of the tiles extracted by the CNN model is 224 × 224.A frozen slide was scanned using a slide scanning imaging system SQS-600P scanner, and the time from getting the frozen slide to scanning to obtain the WSI was within two minutes. The scanned WSI can be directly processed by our system first by cropping the whole WSI into tiles of 224 × 224 size, and the cropped tiles are directly input for the entire slide prediction module in the system for visualizing the effect, the whole process is strictly controlled within 10 min, which can well assist the pathologist’s diagnosis.

Discussion

Up to now, histopathology is still the gold standard for pathology doctors to diagnose diseases, and its development from slides under the light microscope to digital pathology images shows the significance of artificial intelligence in medical treatment. As the incidence of thyroid cancer is increasing year by year worldwide, the advancement of AI-assisted approaches to update it has received widespread attention.

Reasons for classification task selection

In recent years, there has been good development in the field of research on the artificial intelligence-assisted diagnosis of the thyroid, but almost the vast majority of the findings have focused on paraffin Sects. [45,46,47,48], and few studies have been performed on frozen sections of thyroid cancer in clinical surgery [38, 49]. The only articles that are available only address the benignity and malignancy of nodules in thyroid sections, and the management of different thyroid cancers in clinical surgery is exceptionally variable [50, 51], as shown in Table 5.

Table 5 Diagnostic modalities of different cancerous lesions of the thyroid gland

Due to time constraints in surgery, protocols for different thyroid cancers must be adjusted and determined as soon as possible. Therefore, to solve the practical clinical problem, we designed a five-class classifier for frozen thyroid lesions, through which we can cover a broader range of thyroid lesions and assist pathologists in obtaining efficient results. Since thyroid cancers are morphologically distinct, such as papillary thyroid carcinoma nuclei showing variations in size and shape, such as enlargement, lengthening, overlapping and crowding, our deep learning model centers around morphological features for the classification task.

Development of a database of frozen thyroid WSIs

We built a Frozen Thyroid WSIs Database with 3873 cases, 4409 frozen thyroid slices containing WSIs for papillary thyroid cancer, medullary carcinoma, anaplastic carcinoma, follicular tumor, nodular goiter, chronic lymphocytic thyroiditis, and subacute thyroiditis. All the data in the database were used in our diagnostic system, of which 2894 were used for system building. The creation of the Frozen Thyroid WSIs Database lays the foundation for research into rapid intraoperative diagnosis, introducing deep learning into the surgical process. In addition, the database broadens the idea of integrating artificial intelligence into clinical care and confirms the potential of deep learning approaches in the freezing field. We will also continue to collect samples by enriching the database in terms of sample types and numbers in the future.

Distinction between FVPTC and NIFTP

In the classification diagnosis, distinguishing between NIFTP and FVPTC is crucial for the improvement of model accuracy. FVPTC and NIFTP have partial similarity in pathological features, but still have obvious distinguishing features. Nuclei in FVPTC are usually larger, heteromorphic, and irregular, whereas nuclei in NIFTP are more regular, smaller, and relatively homogeneous in shape. The follicular structure in FVPTC may appear disrupted, misshapen, or atypically arranged and have more microstructural variations. The follicular structure in NIFTP usually remains relatively regular and organized and lacks significant disruption or heterogeneity [52,53,54,55]. These pathological feature differences can be effectively learned and utilized by deep learning models to achieve better results in classification tasks.

Patch size selection

Our proposed system is a neural network model based on tiles classification so the selection of tiles has a particular influence on the model results to a large extent. We use tiles of 224 × 224 pixels in size, which firstly ensure that there is no significant loss of image information, and secondly, our model is mainly based on the extraction of morphological features for classification learning, and the selected tiles are also as detailed as possible to show each morphological feature to assist in making a reliable diagnosis. By comparing the performance of the benign and malignant classification work done by Li et al. [38], the size of the tiles selected by them is 2392 × 2392, and the average accuracy of our results presents better results under the same model action, which is also reflected in the tiles-based classification model, where our tiles size selection can better capture the morphological features of the lesion. In addition, the chosen tiles size allows the model to diagnose a single thyroid slide within 5 min in most cases, satisfying the time requirement. Therefore, considering the perspective of clinical application, the size of this tile can fulfill both the model feature extraction needs and the diagnostic time limit, thus better diagnosing from the standpoint of simulated pathologists.

As the number of cases of different lesions is relatively unbalanced, this may have some impact on the generalizability of the study’s findings and the limitations of its practical application. One of the goals of the study is to generalize to a wider population or context to strengthen the validity and reliability of its practical application. We will continue to collect samples, firstly, to contain a relatively small number of instances, such as the part of anaplastic carcinoma; secondly, to collect some external data sets, and combine the two aspects with improving the robustness and generalization ability of the model.

Assistant to pathologists

Our work demonstrates that deep learning can have a fine-grained aid to the intraoperative frozen section classification diagnostic work, which helps pathologists to develop surgical strategies efficiently. On the one hand, our system can analyze digital pathology sections while the pathologist is studying frozen sections under a light microscope, visualizing the distribution of lesions and areas of cancer by a grad-cam heat map or tiles fusion. The two complement each other and work together to improve the accuracy of the pathologist’s judgmental findings. On the other hand, our system can also complement the pathologist’s diagnosis with specific details, for example, in areas easily missed, thus complementing each other. However, the deep learning approach is used only to aid the pathologist’s diagnosis and cannot directly replace the pathologist’s independent work.

Conclusion

In this paper, we propose a deep learning model classification system based on tiles to build a Frozen Thyroid WSIs Database, which is the first work on the fine classification of frozen thyroid carcinoma slices and the first digital database for frozen slices. We finally used the Resnet101 model as the final development model by comparing multiple models. We tested our data through the diagnostic system and obtained a high accuracy result of 94.59%±3.55%. Meanwhile, we proposed sensitivity tests for both systems to reflect the system’s accurate control of cancer diagnosis by pooling multiple scales. In addition, to consider the clinical application, we developed a threshold-based tumor-first heuristic strategy for dividing WSI primary and secondary tissues by double thresholds, which meets the actual clinical needs. Finally, our study can diagnose typical WSIs images within 10 min, confirming the efficient performance of our system. Our work is consistent with the application to clinical scenarios in terms of the breadth of classification coverage of frozen thyroid sections, the accuracy of model results, the efficiency of diagnosis and the consistency with experienced pathologists, and it has significant implications for clinical diagnosis. In future work, we will also enrich the number of frozen digital databases and extend them to other tissue-frozen sections to build a multi-tissue system for intraoperative rapid frozen sections.