Retrospective Study Open Access
Copyright ©The Author(s) 2022. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Gastroenterol. Oct 7, 2022; 28(37): 5469-5482
Published online Oct 7, 2022. doi: 10.3748/wjg.v28.i37.5469
Online calculator for predicting the risk of malignancy in patients with pancreatic cystic neoplasms: A multicenter, retrospective study
Dong Jiang, Zi-Xiang Chen, Fu-Xiao Ma, Yu-Yong Gong, Tian Pu, Jiang-Ming Chen, Xue-Qian Liu, Yi-Jun Zhao, Kun Xie, Hui Hou, Cheng Wang, Xiao-Ping Geng, Fu-Bao Liu
Dong Jiang, Zi-Xiang Chen, Fu-Xiao Ma, Yu-Yong Gong, Tian Pu, Jiang-Ming Chen, Xue-Qian Liu, Yi-Jun Zhao, Kun Xie, Xiao-Ping Geng, Fu-Bao Liu, Department of General Surgery, The First Affiliated Hospital of Anhui Medical University, Hefei 230000, Anhui Province, China
Hui Hou, Department of General Surgery, The Second Affiliated Hospital of Anhui Medical University, Hefei 230000, Anhui Province, China
Cheng Wang, Department of General Surgery, The First Affiliated Hospital of University of Science and Technology of China, Hefei 230000, Anhui Province, China
ORCID number: Dong Jiang (0000-0003-3372-6062); Tian Pu (0000-0002-6961-0814); Jiang-Ming Chen (0000-0003-4683-5866); Yi-Jun Zhao (0000-0001-9082-8861); Kun Xie (0000-0002-2491-4988); Hui Hou (0000-0002-5997-1229); Cheng Wang (0000-0001-7899-2079); Xiao-Ping Geng (0000-0002-6219-6366); Fu-Bao Liu (0000-0003-3431-2187).
Author contributions: Jiang D, Chen JM, Chen ZX and Ma FX contributed to the data analysis and participated in drafting the article; Liu XQ, Gong YY, and Pu T extracted the clinical data and calculated the clinical correlations; Xie K and Zhao YJ interpreted the results and revised the manuscript; Liu FB, Hou H, Wang C and Geng XP gave final approval of the version to be published; All authors contributed to the design and interpretation of the study and to further drafts and approved the final version to be published.
Supported by University Natural Science Research Project of Anhui Province, No. KJ2021ZD0021.
Institutional review board statement: The study was reviewed and approved by the Institutional ethics committees of the First Affiliated Hospital of Anhui Medical University (Approval Quick-PJ2022-06-26).
Informed consent statement: All study participants or their legal guardian provided informed written consent prior to study enrollment.
Conflict-of-interest statement: All the authors report no relevant conflicts of interest for this article.
Data sharing statement: No additional data are available.
Open-Access: This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: https://creativecommons.org/Licenses/by-nc/4.0/
Corresponding author: Fu-Bao Liu, MD, PhD, Chief Doctor, Professor, Department of General Surgery, The First Affiliated Hospital of Anhui Medical University, No. 81 Meishan Road, Hefei 230000, Anhui Province, China. lancetlfb@126.com
Received: June 16, 2022
Peer-review started: June 16, 2022
First decision: July 12, 2022
Revised: July 25, 2022
Accepted: September 8, 2022
Article in press: September 8, 2022
Published online: October 7, 2022

Abstract
BACKGROUND

Efficient and practical methods for predicting the risk of malignancy in patients with pancreatic cystic neoplasms (PCNs) are lacking.

AIM

To establish a nomogram-based online calculator for predicting the risk of malignancy in patients with PCNs.

METHODS

In this study, the clinicopathological data of target patients in three medical centers were analyzed. The independent sample t-test, Mann–Whitney U test or chi-squared test were used as appropriate for statistical analysis. After univariable and multivariable logistic regression analysis, five independent factors were screened and incorporated to develop a calculator for predicting the risk of malignancy. Finally, the concordance index (C-index), calibration, area under the curve, decision curve analysis and clinical impact curves were used to evaluate the performance of the calculator.

RESULTS

Enhanced mural nodules [odds ratio (OR): 4.314; 95% confidence interval (CI): 1.618–11.503, P = 0.003], tumor diameter ≥ 40 mm (OR: 3.514; 95%CI: 1.138–10.849, P = 0.029), main pancreatic duct dilatation (OR: 3.267; 95%CI: 1.230–8.678, P = 0.018), preoperative neutrophil-to-lymphocyte ratio ≥ 2.288 (OR: 2.702; 95%CI: 1.008–7.244, P = 0.048], and preoperative serum CA19-9 concentration ≥ 34 U/mL (OR: 3.267; 95%CI: 1.274–13.007, P = 0.018) were independent risk factors for a high risk of malignancy in patients with PCNs. In the training cohort, the nomogram achieved a C-index of 0.824 for predicting the risk of malignancy. The predictive ability of the model was then validated in an external cohort (C-index: 0.893). Compared with the risk factors identified in the relevant guidelines, the current model showed better predictive performance and clinical utility.

CONCLUSION

The calculator demonstrates optimal predictive performance for identifying the risk of malignancy, potentially yielding a personalized method for patient selection and decision-making in clinical practice.

Key Words: Pancreatic cystic neoplasms, Risk of malignancy, Nomogram, Model, Prediction, Calculator

Core Tip: A nomogram-based online calculator for predicting the risk of malignancy in patients with pancreatic cystic neoplasms was developed. The calculator demonstrates optimal predictive performance for identifying the risk of malignancy, potentially yielding a personalized method for patient selection and decision-making in clinical practice.



INTRODUCTION

Pancreatic cystic neoplasms (PCNs) are estimated to be present in 2%–45% of the general population[1-3]. With rapid advancements in medical examination technologies over the past decades, the use of these technologies in the identification of PCNs has dramatically increased. Moreover, PCNs are frequently discovered in asymptomatic patients[4]. Consequently, centers with high volumes of patients undergoing pancreatic surgery are evaluating an increasing number of PCNs cases annually. Nevertheless, how to properly manage patients with PCNs remains a topic of debate.

PCNs are usually divided into serous cystic neoplasms (SCNs), mucinous cystic neoplasms (MCNs), intraductal papillary mucinous neoplasms (IPMNs) and other rare cystic lesions, such as solid pseudopapillary tumors (SPTs) and cystic neuroendocrine tumors (cNET) which show biological behavior ranging from benign to malignant[5,6]. Moreover, the main controversy regarding the treatment of PCNs is related to the inability to precisely determine their histopathological diagnosis without surgical resection[7,8], thereby precluding the accurate identification of different subtypes of PCNs. Consequently, clinicians are challenged to maintain a difficult balance between avoiding excessive surgical treatment and keeping a malignant lesion under surveillance.

Although previous studies have focused on the identification of different types of PCNs[9-11], only a few of these studies have emphasized the judgment or prediction of malignancy risk in PCNs. Thus, identifying a method that can accurately and preoperatively predict the risk of malignancy in PCNs is of great significance. Nomograms are effective statistical tools that enable the simultaneous consideration of various factors for clinicians to visualize the malignant probability of a neoplasm, allowing the formulation of an optimal therapeutic schedule. Nomogram possess other advantages including ease of comprehension, user-friendliness and personalized evaluation[12]. Therefore, this study aimed to establish a novel clinical online nomogram-based calculator to predict the risk of malignancy in PCNs.

MATERIALS AND METHODS

The clinical and pathological data of patients with PCNs who underwent surgery at the Department of General Surgery in three medical centers between January 2015 and December 2021 were collected and analyzed. Patients from the First Affiliated Hospital of Anhui Medical University and Second Affiliated Hospital of Anhui Medical University were set as the training cohort, while those from the First Affiliated Hospital of the University of Science and Technology of China were set as the validation cohort. All hospitals were high-volume surgical centers that employed similar therapeutic approaches for PCNs. Pathological diagnosis was confirmed by postoperative specimen examinations by two experienced pathologists. According to the histological classification of PCNs proposed by the World Health Organization (2019) and the Baltimore consensus meeting, PCNs are classified as those with low- or intermediate-grade dysplasia and those with high-grade dysplasia or invasive carcinoma[13,14]. Patients from the three centers who met the following criteria were included: (1) Patients who underwent surgical treatment and were confirmed as showing SCNs, IPMNs or MCNs in postoperative pathological examinations; (2) Patients with complete clinical data, including medical history and laboratory test data; and (3) Patients who underwent preoperative imaging examinations. The following patients were excluded: (1) Patients who were pathologically confirmed as having other rare cystic lesions such as SPTs and cNET; (2) Patients who were pathologically confirmed as having other uncommon and undefined cystic tumors of the pancreas; and (3) Patients with a history of pancreatic surgery, radiotherapy or chemotherapy. All clinical data were screened and collected in a computerized database by a professional research assistant. In this study, patients were categorized as showing low-risk (low- or intermediate-grade dysplasia) or high-risk (high-grade dysplasia or invasive carcinoma) disease based on the pathological diagnosis. The appearance of high-risk disease was characterized as a study endpoint. Figure 1 depicts a flowchart of patient enrollment and the inclusion process. This study was conducted in accordance with the Declaration of Helsinki and was approved by the institutional ethics committee (Quick-PJ2022-06-26). All included patients or their relatives provided written informed consent before their data were analyzed.

Figure 1
Figure 1 Flowchart of patient recruitment and diagnosis. AHMU: Anhui Medical University; USTC: University of Science and Technology of China.
Perioperative management

In accordance with the similar preoperative evaluation protocol at all centers, all patients were routinely examined using blood tests, including routine blood counts and analysis of blood biochemistry, hemostatic function, immunological markers and tumor markers. Moreover, all patients underwent at least two preoperative imaging examinations through ultrasound (US), computed tomography (CT), magnetic resonance imaging or 2-18F-fluoro-2-deoxy-d-glucose positron emission tomography-CT, which provided information about the size, number, location and internal condition of the lesions. Definitive planning of the procedure was performed according to the examination results and the patient’s age, comorbidities and preferences.

Surgical procedures, including pancreatoduodenectomy, total pancreatectomy, partial pancreatectomy and distal pancreatectomy with or without splenic preservation, were performed depending on the tumor location and characteristics. Experienced pancreatic surgeons performed all surgical procedures. Meanwhile, the balance between the benefits of oncological clearance and the risks of inadequate parenchymal preservation was carefully considered.

Normative and meticulous postoperative care including monitoring of vital signs, maintenance of moderate tissue perfusion and support for nutritional needs was implemented for every patient at the initial stage.

Statistical analysis

All data, including demographic data, preoperative imaging data and clinical and pathological data were recorded in detail using a special recorder. Continuous variables were expressed as mean ± SD for normally distributed variables or median (interquartile range) for non-normally distributed variables and statistical tests were used as appropriate (the independent sample t-test or the Mann–Whitney U test). Categorical variables were expressed as numbers (n) and proportions (%) and compared appropriately via Fisher’s exact test or the chi-squared test. The cutoff values of continuous parameters were derived from the Youden index[15]. Univariable logistic regression analysis was performed on the training cohort to screen for risk factors associated with malignancy. On the basis of the univariable analysis, multivariable logistic regression analysis was conducted using variables that had statistical significance. A nomogram for the prediction of malignancy in patients with PCNs was developed based on the multivariable logistic regression model. The performance of the nomogram was evaluated using a concordance index (C-index) and calibration plots based on bootstrap validation method (n = 1000) to reduce overfitting bias. The area under the curve (AUC) of the receiver operating characteristic (ROC) curves and the quality indices such as sensitivity, specificity, accuracy, positive predictive value and negative predictive value with the associated 95% confidence intervals (CIs) of the model applied to the training and external validation cohorts were obtained. This allowed assessment of the predictive efficiency of the model in comparison with those of the risk factors identified in the relevant guidelines. The clinical usefulness of the nomogram was examined by determining the net benefit using decision curve analysis (DCA)[16]. DCA can assess the utility of models for decision making by plotting net benefit at a range of clinically reasonable risk thresholds. The predictive accuracy and clinical usefulness of the nomogram were demonstrated by clinical impact curves. Based on the sensitivity, specificity and predictive values, the optimal cutoff value was evaluated for accuracy. An online calculator for public use was created through shinyapps.io by RStudio to facilitate the use of the model in clinical practice. Statistical analyses were performed using R version 4.0.5 (http://www.r-project.org/, R Development Core Team) and SPSS version 23.0 (SPSS Inc, Armonk, NY, United States. In this study, the packages rms, nomogramFormula, DynNom and rmda were utilized for statistical analysis and figure drawing. A two-tailed P value < 0.05 was considered significant.

RESULTS
Patient cohorts and clinicopathologic features

The clinicopathologic data of 1764 consecutive patients who underwent pancreatic surgery at the First Affiliated Hospital of Anhui Medical University and the Second Affiliated Hospital of Anhui Medical University between January 2015 and December 2021 were collected. Among them, 246 patients (13.9%) who were pathologically confirmed as having IPMNs, MCNs or SCNs were classified as the initial target population. Of these 246 patients, 59 (24.0%) who did not fulfill the inclusion criteria were excluded, including 19 patients with incomplete clinical or imaging data, nine with a history of pancreatic surgery, 25 with mixed pathological characteristics and 6 with a history of radiotherapy or chemotherapy. Ultimately, 187 patients (76.0%) were included in the training cohort. Based on the same criteria, an independent group consisting of 103 patients from the First Affiliated Hospital of University of Science and Technology of China who were eligible for inclusion in the same period was included in the present study and served as an external validation cohort. The two cohorts showed no significant differences in baseline characteristics (P > 0.05; Tables 1 and 2).

Table 1 Preoperative clinical characteristics of patients with pancreatic cystic neoplasms.
CharacteristicsTraining cohort
Validation cohort
P value
n = 187 (64.5%)
n = 103 (35.5%)
Sex0.190
Male126 (67.4)77 (74.8)
Female61 (32.6)26 (25.2)
Age in yr0.344
< 60123 (65.8)62 (60.2)
≥ 6064 (34.2)41 (39.8)
BMI23.25 ± 3.3723.00 ± 3.530.561
Weight loss0.830
No165 (88.2)90 (87.4)
Yes22 (11.8)13 (12.6)
Alcohol0.378
No168 (89.8)89 (86.4)
Yes19 (10.2)14 (13.6)
Symptomatic0.602
No74 (39.6)44 (42.7)
Yes113 (60.4)59 (57.3)
NLR0.510
< 2.288134 (71.7)70 (68.0)
≥ 2.28853 (28.3)33 (32.0)
TB in μmol/L0.763
< 34.2181 (96.8)99 (96.1)
≥ 34.26 (3.2)4 (3.9)
CEA in ng/mL0.901
< 5175 (93.6%)96 (96.0%)
≥ 512 (6.4)7 (7.0)
CA19-9 in U/mL0.135
< 34163 (87.2)83 (80.6)
≥ 3424 (12.8)20 (19.4)
GGT in IU/L0.287
< 150174 (93.0)99 (96.1)
≥ 15013 (7.0)4 (3.9)
ALP in IU/L0.556
< 200179 (95.7)97 (94.2)
≥ 2008 (4.3)6 (5.8)
Table 2 Preoperative imaging characteristics of patients with pancreatic cystic neoplasms.
CharacteristicTraining cohort
Validation cohort
P value
n = 187 (64.5%)
n = 103 (35.5%)
Tumor involve the head of pancreas0.239
Yes65 (34.8)43 (41.7)
No122 (65.2)60 (58.3)
Bile duct dilation0.106
No174 (93.0)90 (87.4)
Yes13 (7.0)13 (12.6)
Tumor size in mm0.254
< 40 158 (84.5)92 (89.3)
≥ 40 29 (15.5)11 (10.7)
Intratumoral septum0.264
No76 (40.6)35 (34.0)
Yes111 (59.4)68 (66.0)
Cyst wall enhancement0.244
No118 (63.1)72 (69.9)
Yes69 (36.9)31 (30.1)
Cyst wall thickening0.709
No178 (95.2)97 (94.2)
Yes9 (4.8)6 (5.8)
Enhanced mural nodules0.126
No148 (79.1)89 (86.4)
Yes39 (20.9)14 (13.6)
Intracapsular calcification0.117
No162 (86.6)82 (79.6)
Yes25 (13.4)21 (20.4)
Main pancreatic duct dilatation0.419
No122 (65.2)72 (69.9)
Yes65 (34.8)31 (30.1)
Univariable and multivariable analysis in the training cohort

The results of the univariable and multivariable analyses of variables concerning malignancy in the training cohort were listed in Table 3. The optimal cutoff values of the neutrophil-to-lymphocyte ratio (NLR) was 2.288. Factors that significantly affected the risk of malignancy in the univariable analysis were then included in the multivariable analysis, which demonstrated that the presence of enhanced mural nodules [odds ratio (OR): 4.314; 95%CI: 1.618–11.503, P = 0.003], tumor diameter ≥ 40 mm (OR: 3.514; 95%CI: 1.138–10.849, P = 0.029), main pancreatic duct dilatation (OR: 3.267; 95%CI: 1.230–8.678, P = 0.018), preoperative NLR ≥ 2.288 (OR: 2.702; 95%CI: 1.008–7.244, P = 0.048) and preoperative serum CA19-9 concentration ≥ 34 U/mL (OR: 3.267; 95%CI: 1.274–13.007, P = 0.018) were independent risk factors for a high risk of malignancy in patients with PCNs.

Table 3 Univariable and multivariable logistic regression analyses of the risk factors for high risk of malignancy in patients with pancreatic cystic neoplasms.
VariableOR comparisonsUnivariable analysis
Multivariable analysis
OR (95%CI)
P value
OR (95%CI)
P value
SexFemale vs Male2.383 (1.054-5.386)0.037a0.716 (0.232-2.213)0.562
Age in yr≥ 60 vs < 602.180 (0.967-4.915)0.060
BMI0.986 (0.874-1.111)0.816
Weight lossYes vs No2.437 (0.861-6.897)0.093
AlcoholYes vs No2.252 (0.741-6.845)0.153
SymptomaticYes vs No1.774 (0.737-4.271)0.201
Tumor involve the head of pancreasYes vs No1.148 (0.487-2.705)0.753
Bile duct dilationYes vs No1.788 (0.460-6.952)0.402
Tumor diameter in mm≥ 40 vs < 404.094 (1.649-10.165)0.002a3.514 (1.138-10.849)0.029a
Intratumoral septumYes vs No0.539 (0.240-1.211)0.135
Cyst wall enhancementYes vs No1.127 (0.494-2.569)0.777
Cyst wall thickeningYes vs No1.670 (0.329-8.488)0.536
Enhanced mural nodulesYes vs No6.490 (2.746-15.342)< 0.001a4.314 (1.618-11.503)0.003a
Intracapsular calcificationYes vs No0.208 (0.027-1.606)0.132
Main pancreatic duct dilatationYes vs No3.574 (1.558-8.201)0.003a3.267 (1.230-8.678)0.018a
NLR≥ 2.288 vs < 2.2883.077 (1.350-7.015)0.008a2.702 (1.008-7.244)0.048a
TB in μmol/L≥ 34.2 vs < 34.26.240 (1.192-32.657)0.030a0.870 (0.050-15.020)0.924
ALP in IU/L≥ 200 vs < 2009.420 (1.499-59.213)0.017a0.632 (0.064-6.285)0.695
GGT in IU/L≥ 150 vs < 1502.925 (0.931-9.189)0.066
CEA in ng/mL≥ 5 vs < 56.955 (2.060-23.477)0.002a3.798 (0.825-17.482)0.087
CA199 in U/mL≥ 34 vs < 344.547 (1.750-11.816)0.002a3.267 (1.274-13.007)0.018a
Development and evaluation of the predictive model

The factors that were proven to be related to a high risk of malignancy were utilized to establish a nomogram for evaluating the probability of malignancy (Figure 2). The model obtained using this approach showed good predictive capacity for estimating the risk of malignancy in patients with PCNs (training cohort: C-index, 0.824; 95%CI: 0.735–0.914, vs validation cohort: C-index, 0.893; 95%CI: 0.823–0.963). The calibration plots demonstrated good consistency between the observed and predicted probabilities in both the internal and external validations. Both predicted and reference curves were approximately aligned which indicates good performance of the nomogram (Figure 3). Overall performance was scored at 0.091 using Brier's score which measured the difference between observed and predicted values (values closer to 0 indicate better predictive ability). A calibration slope of 1.0 was obtained to assess the agreement between observed and predicted values (values closer to 1.0 indicate better performance)[17]. An online calculator for predicting the risk of malignancy was created and was freely available at https://ahmuptt.shinyapps.io/JDYX/.

Figure 2
Figure 2 Nomogram for predicting risk of malignancy in the training cohort. NLR: Neutrophil-to-lymphocyte ratio.
Figure 3
Figure 3 Calibration curves for predicting the risk of malignancy. The nomogram had concordance index values of 0.824 and 0.893 in the training and validation cohort, respectively. A: Training cohort; B: Validation cohort.
Comparison of the performance of the nomogram and the risk factors identified in the relevant guidelines

Using ROC analyses, the diagnostic performance of the nomogram established in this study was compared with those of the risk factors identified in the relevant guidelines[6,18], including tumor diameter ≥ 40 mm, enhanced mural nodules and main pancreatic duct dilatation (Figure 4). In the training cohort, the AUC values and 95%CIs of the nomogram and the three factors (tumor diameter ≥ 40 mm, enhanced mural nodules and main pancreatic duct dilatation) were 0.824 (0.735–0.914), 0.619 (0.496–0.742), 0.692 (0.575–0.810) and 0.653 (0.540–0.766), respectively. In the validation cohort, these values were 0.893 (0.823–0.963), 0.718 (0.560–0.876), 0.665 (0.506–0.824) and 0.672 (0.524–0.820), respectively. In both cohorts, significant statistical differences were found between the nomogram and the three factors (P < 0.05). The training cohort had an accuracy, sensitivity, specificity, positive predictive value and negative predictive value of 0.829, 0.643, 0.862, 0.451 and 0.932, respectively. In the validation cohort, these values were 0.925, 0.882, 0.826, 0.501 and 0.973, respectively. Thus, the nomogram showed high accuracy in predicting the risk of malignancy in patients with PCNs. When considering the maximum Youden index value, the optimal cutoff value of the nomogram was 160.8 in the ROC curve and the sensitivity and specificity for differentiating between high and low risk were 64.3% and 86.2%, respectively. This cut off value is used to categorize patients with total nomogram score of < 160.8 points or ≥ 160.8 points as either low- or high-risk group, respectively. In the training cohort, DCA showed that using the nomogram to predict the risk of malignancy had a greater net benefit than the other three factors when the threshold probability ranged from 0.2 to 1.0. In the validation cohort, DCA showed that the nomogram had a greater net benefit when the threshold probability ranged from 0.0 to 0.4 (Figure 5). Figure 6 depicts the number of high-risk patients (solid red line) and the number of high-risk patients with the outcome (black dotted line) at different threshold probabilities within a given population. In both the training and validation cohorts, the solid red and black dotted lines show a great fit indicating that the model has remarkable predictive ability.

Figure 4
Figure 4 Receiver operating characteristic curve of the nomogram and other three risk factors identified in guidelines. A: Training cohort; B: Validation cohort. AUC: Area under the curve.
Figure 5
Figure 5 Decision curve analysis of the nomogram and other three risk factors identified in guidelines. The x-axis represents the threshold probability and the y-axis represents the net benefit. In the training cohort, the nomogram adds more net benefit than the other three factors when the threshold probability ranged from 0.2 to 1.0. In the validation cohort, the nomogram adds more net benefit than the other three factors when the threshold probability ranged from 0.0 to 0.4. A: In the training cohort; B: In the validation cohort.
Figure 6
Figure 6 Clinical impact curves of the nomogram. At different threshold probabilities within a given population, the number of high-risk patients (solid red line) and the number of high-risk patients with the outcome (black dotted line) are shown. In both training and validation cohort, the solid red line and black dotted line show a great fit. A: In the training cohort; B: In the validation cohort.
DISCUSSION

The increasing use of high-resolution cross-sectional imaging techniques has resulted in more frequent detection of PCNs in recent years[19]. However, accurate prediction of the risk of malignancy of PCNs as early as possible is of great value in developing more appropriate individualized treatment strategies. In the current study, we developed and validated a preoperative clinical model that strongly predicted the risk of high-grade dysplasia or invasive cancer in patients with PCNs.

As shown in Tables 1 and 2, our study analyzed dozens of objective factors, and the findings indicated that for patients with PCNs, lesions with enhanced mural nodules, tumor diameter ≥ 40 mm, main pancreatic duct dilatation, preoperative NLR ≥ 2.288 and preoperative serum CA19-9 levels ≥ 34 U/mL were significant independent predictors. These factors were combined with patients’ preoperative imaging and serological data to quantify the risk of malignancy concisely and spontaneously. The sufficient statistical power of the model for predicting the risk of malignancy was verified based on multiple validation methods. The C-index of the model in the validation cohort was 0.893 (0.823–0.963), highlighting the ability of the model to distinguish low-risk disease from high-risk disease in a large group of patients. In addition, we used an independent validation dataset that effectively decreased the risk of overfitting the model to an individual dataset. The similarity between the C-index of the training [0.824 (0.735–0.914)] and validation [0.893 (0.823–0.963)] cohorts suggested that this model was widely applicable. In addition, a mobile-friendly online version of the nomogram was programmed to eliminate the inconvenience of traditional nomograms in clinical use.

The latest guidelines, including the European evidence-based guidelines on PCNs and the International Association of Pancreatology guidelines of 2017, have described the management strategies for different subtypes of PCNs in detail[6,18]. In addition, many published nomogram models for predicting malignancy in patients with a specific subtype of PCNs, such as IPMNs, have been established to facilitate the preoperative prediction of malignant lesions[20-22]. Nevertheless, real-world preoperative diagnosis of the different PCN subtypes remains difficult due to their similarities in both clinical and imaging features. In particular, accurate diagnosis of pathological neoplasms and preoperative assessments of the benign or malignant status of some lesions with atypical manifestations are even more difficult[23]. Previous studies have reported differing probabilities for the accurate preoperative identification of each subtype. Salvia et al[24] found that in 476 patients with PCNs, 78% showed postoperative pathological examination results consistent with the preoperative diagnosis[24]. Furthermore, a multicenter study of 2251 patients in China showed that the preoperative diagnosis was not always accurate for a specific subtype with a correct diagnosis rate of only 33%[25]. Meanwhile, the risk factors proposed to be associated with malignant lesions in the guidelines lack universality, efficiency and accuracy in clinical applications to a certain extent. In this study, a model with strong practicability and high predictive efficiency was constructed by analyzing PCNs showing similar clinical and imaging manifestations across subtypes. Therefore, this model can help clinicians predict the risk of lesion malignancy when they are unable to accurately determine the PCN subtype or have difficulties in confirming the nature of the tumor.

NLR was also included as a laboratory indicator in the nomogram. Researches have shown that inflammation contributes to the occurrence and progression of tumors by releasing cytokines and other inflammatory mediators[26,27]. NLR measurement can serve as a simple and convenient way to assess the systemic inflammatory response and many recent studies have confirmed that inflammatory markers play an important role in predicting benign and malignant PCNs[28-30]. In the current study, a significant correlation was found between NLR and high-risk groups by multivariable analysis and the cutoff value of NLR was calculated as 2.43 which was rarely studied and reported in previous studies. We believe that neutrophils can recruit and activate inflammatory cells by producing cytokines and chemokines thus acting on the tumor microenvironment. In addition, reactive oxygen species and proteases produced by neutrophils can regulate tumor cell proliferation, angiogenesis and metastasis[31]. Thus, an increase in NLR may indicate a macroscopic withdrawal of inflammatory and immune responses in the local tumor microenvironment.

The nomogram was chosen to establish the model mainly because of its ability to assign risk probabilities on a continuous scale as an individualized risk score, rather than simply dividing patients into malignant and benign groups. This allows for additional risk stratification and may help patients and doctors tailor treatment decisions based on patients’ individual risks. In addition, as shown in Figure 2, the nomogram showed that the model score for cases with serum CA19-9 levels ≥ 34 U/mL (100 points) was significantly higher than that for cases with an NLR of ≥ 2.288 (61 points), suggesting that serum CA19-9 levels ≥ 34 U/mL increase the possibility of a malignant lesion significantly. Moreover, the additive effect of several risk factors is important clinically. In this context, the patient-specific quantitative estimation of risk is valuable. Therefore, in cases where benign and malignant PCNs cannot be easily distinguished, clinicians can calculate the total nomogram scores using this predictive model. We propose that surgical treatment should be considered as the first choice for patients with a total nomogram score ≥ 160.

In recent years, machine learning has rapidly developed as a tool with promising results and improved usability[32]. Machine learning addresses many preexisting and novel medical challenges and is widely discussed by researchers and practitioners alike. Machine learning algorithms have many advantages, such as improved flexibility and scalability when compared with traditional statistical methods. This makes machine learning advantageous for many tasks, including risk stratification, diagnosis, classification and survival predictions. Another advantage of machine learning is the ability to analyze diverse data types (e.g., demographic data, laboratory findings, imaging data and doctors' free-text notes) and to incorporate them into predictions for disease risk, diagnosis, prognosis and appropriate treatment[33]. Despite these advantages, the application of machine learning in healthcare delivery also presents unique challenges, including the need for data pre-processing, model training and refinement of the system with respect to the actual clinical problem. Also crucial are ethical considerations which include medico-legal implications, doctors' understanding of machine learning tools, and data privacy and security[34]. In summary, attempts can be made to refine the model further using advanced machine learning in the future.

The present study had several limitations that merit discussion. First, endoscopic US (EUS) helps identify PCNs with features that may indicate the need for surgical resection[6]. However, in our study, only less than 10% of the patients underwent EUS. This may be attributable to the considerable interobserver variation in EUS-based diagnoses. On the other hand, data on EUS-based differentiation between benign and malignant PCNs are inconsistent[35,36]. Moreover, economic affordability is an important consideration, especially for most patients from rural areas. Second, heterogeneity in pathological diagnosis to determine the grade of dysplasia or malignancy and heterogeneity between different imaging examinations may exist. Third, this was a retrospective study considering only patients who underwent surgery which meant that it had inherent limitations resulting from potential selection biases; prospective validation is therefore required to confirm the value of the findings.

CONCLUSION

In conclusion, we developed and validated a novel online calculator using a nomogram based on widely available data to predict the risk of malignancy in patients with PCNs dynamically. The calculator is user-friendly, highly accurate and well validated. Clinicians can use it to alert patients at high risk of malignancy at early stages and to design individual therapy for them.

ARTICLE HIGHLIGHTS
Research background

Efficient and practical methods for predicting the risk of malignancy in patients with pancreatic cystic neoplasms (PCNs) are currently lacking.

Research motivation

Currently, there is no effective clinical prediction model for patients with PCNs and no large study has been conducted to predict malignant risk.

Research objectives

The aim of this study was to identify the risk factors influencing the malignant risk of PCNs and develop a prediction model that is useful for clinical surgeons when making decisions regarding surgical interventions.

Research methods

Data collected in three major medical centers were analyzed to identify independent risk parameters and propose a calculator for patients with PCNs. A number of statistical indices, such as concordance index, calibration curves, area under the curve, decision curve analysis, CIC and others were used to evaluate the performance of the nomogram.

Research results

Five factors, including enhanced mural nodules, tumor diameter ≥ 40 mm, main pancreatic duct dilatation, preoperative neutrophil-to-lymphocyte ratio ≥ 2.288 and preoperative serum CA19-9 concentration ≥ 34 U/mL were found to independently influencing the risk of malignancy. As a result, the model we constructed has a greater predictive value than the factors identified in relevant guidelines.

Research conclusions

For the first time, a model was developed to predict the malignant risk of PCNs and an online calculator was further established to guide decision-making.

Research perspectives

More medical centers included, more data collection and application of “Artificial Intelligence”.

ACKNOWLEDGEMENTS

We would like to thank Prof. Faming Pan (Department of Epidemiology and Biostatistics, School of Public Health, Anhui Medical University), who has made valuable support to the data quality and data analysis which ensured the reliability of the study.

Footnotes

Provenance and peer review: Unsolicited article; Externally peer reviewed.

Peer-review model: Single blind

Specialty type: Gastroenterology and hepatology

Country/Territory of origin: China

Peer-review report’s scientific quality classification

Grade A (Excellent): 0

Grade B (Very good): B, B, B

Grade C (Good): C

Grade D (Fair): D

Grade E (Poor): 0

P-Reviewer: Cheng H, China; Ghareeb WM, China; Luo J, China; Tsujinaka S, Japan S-Editor: Fan JR L-Editor: Filipodia P-Editor: Yu HG

References
1.  de Jong K, Nio CY, Hermans JJ, Dijkgraaf MG, Gouma DJ, van Eijck CH, van Heel E, Klass G, Fockens P, Bruno MJ. High prevalence of pancreatic cysts detected by screening magnetic resonance imaging examinations. Clin Gastroenterol Hepatol. 2010;8:806-811.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 358]  [Cited by in F6Publishing: 273]  [Article Influence: 29.8]  [Reference Citation Analysis (0)]
2.  Girometti R, Intini S, Brondani G, Como G, Londero F, Bresadola F, Zuiani C, Bazzocchi M. Incidental pancreatic cysts on 3D turbo spin echo magnetic resonance cholangiopancreatography: prevalence and relation with clinical and imaging features. Abdom Imaging. 2011;36:196-205.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 80]  [Cited by in F6Publishing: 82]  [Article Influence: 7.3]  [Reference Citation Analysis (0)]
3.  Ip IK, Mortele KJ, Prevedello LM, Khorasani R. Focal cystic pancreatic lesions: assessing variation in radiologists' management recommendations. Radiology. 2011;259:136-141.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 61]  [Cited by in F6Publishing: 62]  [Article Influence: 5.5]  [Reference Citation Analysis (0)]
4.  Laffan TA, Horton KM, Klein AP, Berlanstein B, Siegelman SS, Kawamoto S, Johnson PT, Fishman EK, Hruban RH. Prevalence of unsuspected pancreatic cysts on MDCT. AJR Am J Roentgenol. 2008;191:802-807.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 616]  [Cited by in F6Publishing: 665]  [Article Influence: 44.0]  [Reference Citation Analysis (0)]
5.  Kern SE, Hruban RH, Hidalgo M, Yeo CJ. An introduction to pancreatic adenocarcinoma genetics, pathology and therapy. Cancer Biol Ther. 2002;1:607-613.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 35]  [Cited by in F6Publishing: 40]  [Article Influence: 1.8]  [Reference Citation Analysis (0)]
6.  European Study Group on Cystic Tumours of the Pancreas. European evidence-based guidelines on pancreatic cystic neoplasms. Gut. 2018;67:789-804.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 589]  [Cited by in F6Publishing: 513]  [Article Influence: 147.3]  [Reference Citation Analysis (0)]
7.  Kim SY, Lee JM, Kim SH, Shin KS, Kim YJ, An SK, Han CJ, Han JK, Choi BI. Macrocystic neoplasms of the pancreas: CT differentiation of serous oligocystic adenoma from mucinous cystadenoma and intraductal papillary mucinous tumor. AJR Am J Roentgenol. 2006;187:1192-1198.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 121]  [Cited by in F6Publishing: 126]  [Article Influence: 7.6]  [Reference Citation Analysis (0)]
8.  Wang GX, Wang ZP, Chen HL, Zhang D, Wen L. Discrimination of serous cystadenoma from mucinous cystic neoplasm and branch duct intraductal papillary mucinous neoplasm in the pancreas with CT. Abdom Radiol (NY). 2020;45:2772-2778.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 5]  [Cited by in F6Publishing: 4]  [Article Influence: 2.5]  [Reference Citation Analysis (0)]
9.  Chen S, Ren S, Guo K, Daniels MJ, Wang Z, Chen R. Preoperative differentiation of serous cystic neoplasms from mucin-producing pancreatic cystic neoplasms using a CT-based radiomics nomogram. Abdom Radiol (NY). 2021;46:2637-2646.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 4]  [Cited by in F6Publishing: 2]  [Article Influence: 4.0]  [Reference Citation Analysis (0)]
10.  Yang J, Guo X, Ou X, Zhang W, Ma X. Discrimination of Pancreatic Serous Cystadenomas From Mucinous Cystadenomas With CT Textural Features: Based on Machine Learning. Front Oncol. 2019;9:494.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 38]  [Cited by in F6Publishing: 40]  [Article Influence: 12.7]  [Reference Citation Analysis (0)]
11.  Dmitriev K, Kaufman AE, Javed AA, Hruban RH, Fishman EK, Lennon AM, Saltz JH. Classification of Pancreatic Cysts in Computed Tomography Images Using a Random Forest and Convolutional Neural Network Ensemble. Med Image Comput Comput Assist Interv. 2017;10435:150-158.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 24]  [Cited by in F6Publishing: 10]  [Article Influence: 4.8]  [Reference Citation Analysis (0)]
12.  Balachandran VP, Gonen M, Smith JJ, DeMatteo RP. Nomograms in oncology: more than meets the eye. Lancet Oncol. 2015;16:e173-e180.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 1119]  [Cited by in F6Publishing: 1283]  [Article Influence: 159.9]  [Reference Citation Analysis (0)]
13.  Nagtegaal ID, Odze RD, Klimstra D, Paradis V, Rugge M, Schirmacher P, Washington KM, Carneiro F, Cree IA; WHO Classification of Tumours Editorial Board. The 2019 WHO classification of tumours of the digestive system. Histopathology. 2020;76:182-188.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 924]  [Cited by in F6Publishing: 1031]  [Article Influence: 308.0]  [Reference Citation Analysis (0)]
14.  Basturk O, Hong SM, Wood LD, Adsay NV, Albores-Saavedra J, Biankin AV, Brosens LA, Fukushima N, Goggins M, Hruban RH, Kato Y, Klimstra DS, Klöppel G, Krasinskas A, Longnecker DS, Matthaei H, Offerhaus GJ, Shimizu M, Takaori K, Terris B, Yachida S, Esposito I, Furukawa T; Baltimore Consensus Meeting. A Revised Classification System and Recommendations From the Baltimore Consensus Meeting for Neoplastic Precursor Lesions in the Pancreas. Am J Surg Pathol. 2015;39:1730-1741.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 469]  [Cited by in F6Publishing: 491]  [Article Influence: 78.2]  [Reference Citation Analysis (0)]
15.  Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3:32-35.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in F6Publishing: 40]  [Reference Citation Analysis (0)]
16.  Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak. 2008;8:53.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 596]  [Cited by in F6Publishing: 673]  [Article Influence: 42.6]  [Reference Citation Analysis (0)]
17.  Semenkovich TR, Yan Y, Subramanian M, Meyers BF, Kozower BD, Nava R, Patterson GA, Kreisel D, Puri V. A Clinical Nomogram for Predicting Node-positive Disease in Esophageal Cancer. Ann Surg. 2021;273:e214-e221.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 20]  [Cited by in F6Publishing: 26]  [Article Influence: 20.0]  [Reference Citation Analysis (0)]
18.  Tanaka M, Fernández-Del Castillo C, Kamisawa T, Jang JY, Levy P, Ohtsuka T, Salvia R, Shimizu Y, Tada M, Wolfgang CL. Revisions of international consensus Fukuoka guidelines for the management of IPMN of the pancreas. Pancreatology. 2017;17:738-753.  [PubMed]  [DOI]  [Cited in This Article: ]
19.  Gaujoux S, Brennan MF, Gonen M, D'Angelica MI, DeMatteo R, Fong Y, Schattner M, DiMaio C, Janakos M, Jarnagin WR, Allen PJ. Cystic lesions of the pancreas: changes in the presentation and management of 1,424 patients at a single institution over a 15-year time period. J Am Coll Surg. 2011;212:590-600; discussion 600.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 166]  [Cited by in F6Publishing: 178]  [Article Influence: 15.1]  [Reference Citation Analysis (0)]
20.  Attiyeh MA, Fernández-Del Castillo C, Al Efishat M, Eaton AA, Gönen M, Batts R, Pergolini I, Rezaee N, Lillemoe KD, Ferrone CR, Mino-Kenudson M, Weiss MJ, Cameron JL, Hruban RH, D'Angelica MI, DeMatteo RP, Kingham TP, Jarnagin WR, Wolfgang CL, Allen PJ. Development and Validation of a Multi-institutional Preoperative Nomogram for Predicting Grade of Dysplasia in Intraductal Papillary Mucinous Neoplasms (IPMNs) of the Pancreas: A Report from The Pancreatic Surgery Consortium. Ann Surg. 2018;267:157-163.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 69]  [Cited by in F6Publishing: 77]  [Article Influence: 13.8]  [Reference Citation Analysis (0)]
21.  Jang JY, Park T, Lee S, Kim Y, Lee SY, Kim SW, Kim SC, Song KB, Yamamoto M, Hatori T, Hirono S, Satoi S, Fujii T, Hirano S, Hashimoto Y, Shimizu Y, Choi DW, Choi SH, Heo JS, Motoi F, Matsumoto I, Lee WJ, Kang CM, Han HS, Yoon YS, Sho M, Nagano H, Honda G, Kim SG, Yu HC, Chung JC, Nagakawa Y, Seo HI, Yamaue H. Proposed Nomogram Predicting the Individual Risk of Malignancy in the Patients With Branch Duct Type Intraductal Papillary Mucinous Neoplasms of the Pancreas. Ann Surg. 2017;266:1062-1068.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 81]  [Cited by in F6Publishing: 88]  [Article Influence: 16.2]  [Reference Citation Analysis (0)]
22.  Shimizu Y, Hijioka S, Hirono S, Kin T, Ohtsuka T, Kanno A, Koshita S, Hanada K, Kitano M, Inoue H, Itoi T, Ueki T, Matsuo K, Yanagisawa A, Yamaue H, Sugiyama M, Okazaki K. New Model for Predicting Malignancy in Patients With Intraductal Papillary Mucinous Neoplasm. Ann Surg. 2020;272:155-162.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 27]  [Cited by in F6Publishing: 31]  [Article Influence: 13.5]  [Reference Citation Analysis (0)]
23.  Sahani DV, Kadavigere R, Saokar A, Fernandez-del Castillo C, Brugge WR, Hahn PF. Cystic pancreatic lesions: a simple imaging-based classification system for guiding management. Radiographics. 2005;25:1471-1484.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 257]  [Cited by in F6Publishing: 285]  [Article Influence: 16.1]  [Reference Citation Analysis (0)]
24.  Salvia R, Malleo G, Marchegiani G, Pennacchio S, Paiella S, Paini M, Pea A, Butturini G, Pederzoli P, Bassi C. Pancreatic resections for cystic neoplasms: from the surgeon's presumption to the pathologist's reality. Surgery. 2012;152:S135-S142.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 107]  [Cited by in F6Publishing: 89]  [Article Influence: 10.7]  [Reference Citation Analysis (0)]
25.  Pancreatic Surgery of Chinese Academic Society of Young Surgeons. [The current status of diagnosis and treatment of pancreatic cystic neoplasm in China: a report of 2 251 cases]. Zhonghua Wai Ke Za Zhi. 2018;56:24-29.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in F6Publishing: 6]  [Reference Citation Analysis (0)]
26.  Bausch D, Pausch T, Krauss T, Hopt UT, Fernandez-del-Castillo C, Warshaw AL, Thayer SP, Keck T. Neutrophil granulocyte derived MMP-9 is a VEGF independent functional component of the angiogenic switch in pancreatic ductal adenocarcinoma. Angiogenesis. 2011;14:235-243.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 88]  [Cited by in F6Publishing: 98]  [Article Influence: 8.0]  [Reference Citation Analysis (0)]
27.  Gong L, Cumpian AM, Caetano MS, Ochoa CE, De la Garza MM, Lapid DJ, Mirabolfathinejad SG, Dickey BF, Zhou Q, Moghaddam SJ. Promoting effect of neutrophils on lung tumorigenesis is mediated by CXCR2 and neutrophil elastase. Mol Cancer. 2013;12:154.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 94]  [Cited by in F6Publishing: 103]  [Article Influence: 10.4]  [Reference Citation Analysis (0)]
28.  Zhou W, Rong Y, Kuang T, Xu Y, Shen X, Ji Y, Lou W, Wang D. The value of systemic inflammatory markers in identifying malignancy in mucinous pancreatic cystic neoplasms. Oncotarget. 2017;8:115561-115569.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 5]  [Cited by in F6Publishing: 5]  [Article Influence: 1.0]  [Reference Citation Analysis (0)]
29.  Gemenetzis G, Bagante F, Griffin JF, Rezaee N, Javed AA, Manos LL, Lennon AM, Wood LD, Hruban RH, Zheng L, Zaheer A, Fishman EK, Ahuja N, Cameron JL, Weiss MJ, He J, Wolfgang CL. Neutrophil-to-lymphocyte Ratio is a Predictive Marker for Invasive Malignancy in Intraductal Papillary Mucinous Neoplasms of the Pancreas. Ann Surg. 2017;266:339-345.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 70]  [Cited by in F6Publishing: 79]  [Article Influence: 14.0]  [Reference Citation Analysis (0)]
30.  Goh BK, Tan DM, Chan CY, Lee SY, Lee VT, Thng CH, Low AS, Tai DW, Cheow PC, Chow PK, Ooi LL, Chung AY. Are preoperative blood neutrophil-to-lymphocyte and platelet-to-lymphocyte ratios useful in predicting malignancy in surgically-treated mucin-producing pancreatic cystic neoplasms? J Surg Oncol. 2015;112:366-371.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 29]  [Cited by in F6Publishing: 31]  [Article Influence: 4.1]  [Reference Citation Analysis (0)]
31.  Gregory AD, Houghton AM. Tumor-associated neutrophils: new targets for cancer therapy. Cancer Res. 2011;71:2411-2416.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 456]  [Cited by in F6Publishing: 496]  [Article Influence: 41.5]  [Reference Citation Analysis (0)]
32.  Harish KB, Price WN, Aphinyanaphongs Y. Open-Source Clinical Machine Learning Models: Critical Appraisal of Feasibility, Advantages, and Challenges. JMIR Form Res. 2022;6:e33970.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 1]  [Cited by in F6Publishing: 1]  [Article Influence: 1.0]  [Reference Citation Analysis (0)]
33.  Sendak M, Gao M, Nichols M, Lin A, Balu S. Machine Learning in Health Care: A Critical Appraisal of Challenges and Opportunities. EGEMS (Wash DC). 2019;7:1.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 24]  [Cited by in F6Publishing: 30]  [Article Influence: 8.0]  [Reference Citation Analysis (0)]
34.  Decherchi S, Pedrini E, Mordenti M, Cavalli A, Sangiorgi L. Opportunities and Challenges for Machine Learning in Rare Diseases. Front Med (Lausanne). 2021;8:747612.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 7]  [Cited by in F6Publishing: 8]  [Article Influence: 7.0]  [Reference Citation Analysis (0)]
35.  Donahue TR, Hines OJ, Farrell JJ, Tomlinson JS, Eibl G, Reber HA. Cystic neoplasms of the pancreas: results of 114 cases. Pancreas. 2010;39:1271-1276.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 17]  [Cited by in F6Publishing: 13]  [Article Influence: 1.4]  [Reference Citation Analysis (0)]
36.  Kim JH, Eun HW, Park HJ, Hong SS, Kim YJ. Diagnostic performance of MRI and EUS in the differentiation of benign from malignant pancreatic cyst and cyst communication with the main duct. Eur J Radiol. 2012;81:2927-2935.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 39]  [Cited by in F6Publishing: 41]  [Article Influence: 3.9]  [Reference Citation Analysis (0)]