Non-contrast computed tomography radiomics model to predict benign and malignant thyroid nodules with lobe segmentation: A dual-center study

Table 1 Multi-cohort comparison of thyroid lobes characteristics, n (%)

Characteristics	Training cohort	Validation cohort	Temporal test cohort	External test cohort	P valuea	P valueb	P valuec
Grouped	n = 264	n = 112	n = 97	n = 81	0.203	0.605	0.053
Benign	80 (30)	26 (23)	26 (27)	15 (19)
Malignant	184 (70)	86 (77)	71 (73)	66 (81)
Gender					0.355	0.415	0.463
Female	211 (80)	84 (75)	73 (75)	61 (75)
Male	53 (20)	28 (25)	24 (25)	20 (25)
Age (years), median (P25-P75)	51.00 (38.75-60.00)	48.50 (35.75-58.00)	50.00 (41.00-57.00)	52.00 (41.00-60.00)	0.100	0.944	0.413
BMI, median (P25-P75)	24.46 (22.04-26.30)	24.73 (22.18-27.05)	24.77 (23.11-26.9)	25.01 (24.29-25.78)	0.288	0.161	0.042
Multiple nodules					0.517	0.519	0.036
No	148 (56)	58 (52)	50 (52)	34 (42)
Yes	116 (44)	54 (48)	47 (48)	47 (58)
Tumor size group					0.095	0.001	0.007
Small (≤ 5 mm)	43 (16)	29 (26)	21 (22)	24 (30)
Medium (5-10 mm)	99 (38)	38 (34)	52 (54)	33 (41)
Large (> 10 mm)	122 (46)	45 (40)	24 (25)	24 (30)
Calcify					0.629	0.344	< 0.001
No	192 (73)	78 (70)	76 (78)	42 (52)
Yes	72 (27)	34 (30)	21 (22)	39 (48)
Cystic					0.741	0.399	< 0.001
No	229 (87)	95 (85)	88 (91)	38 (47)
Yes	35 (13)	17 (15)	9 (9)	43 (53)
FT3 (pmol/L), median (P25-P75)	4.76 (4.39-5.09)	4.76 (4.44-4.99)	4.71 (4.46-5.03)	4.79 (4.32-5.45)	0.891	0.918	0.299
FT4 (pmol/L), median (P25-P75)	16.20 (14.88-17.90)	16.20 (14.57-18.30)	17.40 (15.70-19.00)	16.80 (14.50-19.60)	0.976	0.002	0.204
TSH (mU/L), median (P25-P75)	1.73 (1.37-2.44)	1.62 (1.13-2.59)	2.16 (1.38-2.67)	1.68 (1.04-3.01)	0.098	0.153	0.665
TGAb (IU/mL), median (P25-P75)	16.70 (15.4-21.45)	16.70 (15.3-17.72)	17.00 (15.81-17.90)	17.80 (17.30-18.10)	0.173	0.789	0.001
TPOAb (IU/mL), median (P25-P75)	13.00 (12.35-13.22)	13.00 (11.35-15.10)	15.00 (14.20-15.74)	12.60 (12.10-13.50)	0.707	< 0.001	< 0.001

^aThe significant difference between the training cohort and the validation cohort.

^bThe significant difference between the training cohort and the temporal test cohort.

^cThe significant difference between the training cohort and the external test cohort.

Biased data are expressed as median (interquartile range) for continuous variables and as n (%) for categorical variables. The Mann-Whitney U test was applied to compare continuous variables, and the χ²d test was used for categorical variables. BMI: Body mass index; FT3: Free triiodothyronine; FT4: Free thyroxine; TSH: Thyroid-stimulating hormone; TGAb: Thyroglobulin antibody; TPOAb: Thyroid peroxidase antibody.

Table 2 Comparison between the benign and malignant thyroid lobes in the training cohort, n (%)

Characteristics	Benign (n = 80)	Malignant (n = 184)	P value
Gender			0.234
Female	68 (85)	143 (78)
Male	12 (15)	41 (22)
Age (years), median (P25-P75)	54.50 (49.00-64.25)	49.00 (35.00-57.25)	< 0.001
BMI, median (P25-P75)	24.16 (21.93-25.98)	24.80 (22.19-26.35)	0.296
Multiple nodules			0.716
No	43 (54)	105 (57)
Yes	37 (46)	79 (43)
Tumor size group			< 0.001
Small (≤ 5 mm)	4 (5)	39 (21)
Medium (5-10 mm)	21 (26)	78 (42)
Large (> 10 mm)	55 (69)	67 (36)
Calcify			0.028
No	66 (82)	126 (68)
Yes	14 (18)	58 (32)
Cystic			< 0.001
No	57 (71)	172 (93)
Yes	23 (29)	12 (7)
FT3 (pmol/L), median (P25-P75)	4.76 (4.43-5.01)	4.76 (4.39-5.14)	0.421
FT4 (pmol/L), median (P25-P75)	16.20 (14.88-16.92)	16.20 (14.95-18.05)	0.142
TSH (mU/L), median (P25-P75)	1.73 (1.40-2.03)	1.73 (1.37-2.63)	0.227
TGAb (IU/mL), median (P25-P75)	16.70 (15.62-18.05)	16.70 (15.4-25.85)	0.164
TPOAb (IU/mL), median (P25-P75)	13.00 (12.80-13.82)	13.00 (12.35-13.00)	0.671

Biased data are expressed as median (interquartile range) for continuous variables and as n (%) for categorical variables. The Mann-Whitney U test was applied to compare continuous variables, and the χ²d test was used for categorical variables. BMI: Body mass index; FT3: Free triiodothyronine; FT4: Free thyroxine; TSH: Thyroid-stimulating hormone; TGAb: Thyroglobulin antibody; TPOAb: Thyroid peroxidase antibody.

Table 3 Weights of least absolute shrinkage and selection operator selected features and training set Z-score parameters

Feature names	Weight	Average	Variance
Intercept	0.250252	-	-
Original_shape_Elongation	0.311096	0.524	0.104
Original_firstorder_10Percentile	0.213349	50.029	15.014
Original_firstorder_interquartilerange	0.157092	29.216	10.044
Original_glcm_maximumprobability	-0.10344	0.304	0.092
Original_gldm_dependenceentropy	0.192143	6.133	0.351
Original_ngtdm_Contrast	-0.03429	0.009	0.006
Original_ngtdm_Strength	-0.00931	0.472	1.371
Log-sigma-1-mm-3D_glcm_InverseVariance	-0.44471	0.310	0.026
Log-sigma-1-mm-3D_glszm_LargeAreaHighGrayLevelEmphasis	0.326205	68429444.000	388518307.356
log-sigma-1-mm-3D_glszm_LargeAreaLowGrayLevelEmphasis	-0.02438	10850.300	46169.980
Log -sigma-2-mm-3D_firstorder_Maximum	0.173416	56.164	27.585
Log -sigma-2-mm-3D_glrlm_RunLengthNonUniformity	-0.11635	1356.938	450.535
Log -sigma-2-mm-3D_glszm_ZoneEntropy	-0.18766	5.256	0.371
Log -sigma-3-mm-3D_firstorder_TotalEnergy	-0.26768	57352644.000	20352814.000
Log -sigma-3-mm-3D_glcm_ClusterProminence	0.123895	1173.183	544.414
Wavelet-LLH_firstorder_Mean	0.15131	9.850	2.798
Wavelet-LLH_firstorder_Skewness	0.088985	-1.869	1.851
Wavelet-LHH_glcm_InverseVariance	-0.38886	0.503	0.005
Wavelet-LHH_gldm_LargeDependenceHighGrayLevelEmphasis	-0.25005	3423.815	3331.008
wavelet-HLL_glszm_GrayLevelNonUniformityNormalized	0.104264	0.159	0.094
Wavelet-HLL_glszm_LargeAreaLowGrayLevelEmphasis	-0.05978	10742.590	83666.610
Wavelet-HLH_glcm_InverseVariance	-0.14526	0.501	0.005
Wavelet-HHL_glszm_GrayLevelNonUniformity	0.087039	18.166	12.923
Wavelet-HHH_firstorder_InterquartileRange	0.284918	6.497	1.234
Wavelet-LLL_glcm_InverseVariance	-0.45374	0.448	0.023
Wavelet-LLL_gldm_DependenceEntropy	-0.27325	7.039	0.328
Wavelet-LLL_glszm_GrayLevelNonUniformity	-0.23569	39.394	14.681

Table 4 Multi-cohort predictive performance of various models

Model	AUC (95%CI)	Accuracy	Sensitivity	Specificity	PPV	NPV	Precision	Recall	F1	Brier
Training cohort (benign = 80, malignant = 184), SMOTE (benign = 104)
LR	0.845 (0.794-0.896)	0.799	0.832	0.725	0.874	0.652	0.874	0.832	0.852	0.140
DT	0.806 (0.745-0.866)	0.826	0.829	0.815	0.946	0.550	0.946	0.829	0.883	0.136
RF	1.000 (1.000-1.000)	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	0.027
XGB	0.899 (0.845-0.932)	0.845	0.783	0.875	0.935	0.636	0.935	0.783	0.852	0.125
SVM	0.844 (0.791-0.896)	0.814	0.864	0.700	0.869	0.691	0.869	0.864	0.866	0.141
KNN	1.000 (1.000-1.000)	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	0.031
LGBM	0.692 (0.631-0.752)	0.561	0.755	0.600	0.813	0.516	0.813	0.755	0.783	0.208
Senior radiologist	0.596 (0.505-0.633)	0.542	0.500	0.638	0.760	0.357	0.760	0.500	0.603	0.209
Junior radiologist	0.529 (0.464-0.594)	0.496	0.446	0.613	0.736	0.324	0.726	0.446	0.552	0.211
Validation cohort (benign = 26, malignant = 86)
LR	0.834 (0.750-0.917)	0.768	0.837	0.538	0.857	0.500	0.857	0.837	0.847	0.128
DT	0.646 (0.529-0.762)	0.696	0.802	0.346	0.802	0.346	0.802	0.802	0.802	0.218
RF	0.729 (0.620-0.838)	0.759	0.860	0.423	0.831	0.478	0.831	0.860	0.846	0.166
XGB	0.803 (0.715-0.890)	0.696	0.744	0.538	0.842	0.389	0.842	0.744	0.790	0.144
SVM	0.820 (0.728-0.912)	0.795	0.860	0.577	0.871	0.556	0.871	0.860	0.865	0.130
KNN	0.793 (0.697-0.890)	0.741	0.779	0.615	0.870	0.457	0.870	0.779	0.822	0.180
LGBM	0.724 (0.628-0.820)	0.545	0.430	0.923	0.949	0.329	0.949	0.430	0.592	0.182
Senior radiologist	0.558 (0.449-0.667)	0.527	0.500	0.615	0.811	0.271	0.811	0.500	0.619	0.182
Junior radiologist	0.538 (0.428-0.649)	0.518	0.500	0.577	0.796	0.259	0.796	0.500	0.614	0.182
Temporal test cohort (benign = 26, malignant = 71)
LR	0.814 (0.717-0.912)	0.825	0.930	0.538	0.846	0.737	0.846	0.930	0.886	0.137
DT	0.757 (0.652-0.851)	0.742	0.780	0.533	0.901	0.308	0.901	0.780	0.837	0.178
RF	0.795 (0.696-0.894)	0.763	0.901	0.385	0.800	0.588	0.800	0.901	0.848	0.152
XGB	0.855 (0.775-0.935)	0.773	0.831	0.615	0.855	0.571	0.855	0.831	0.843	0.139
SVM	0.816 (0.719-0.913)	0.845	0.972	0.500	0.841	0.867	0.841	0.972	0.902	0.141
KNN	0.719 (0.607-0.832)	0.711	0.803	0.462	0.803	0.462	0.803	0.803	0.803	0.208
LGBM	0.800 (0.697-0.904)	0.835	0.873	0.731	0.899	0.679	0.899	0.873	0.886	0.193
External test cohort (benign = 15, malignant = 66)
LR	0.782 (0.607-0.907)	0.765	0.848	0.400	0.862	0.375	0.862	0.848	0.855	0.127
DT	0.715 (0.566-0.863)	0.778	0.887	0.421	0.833	0.533	0.833	0.887	0.859	0.166
RF	0.751 (0.589-0.913)	0.815	0.879	0.533	0.892	0.500	0.892	0.879	0.885	0.143
XGB	0.802 (0.644-0.939)	0.728	0.727	0.733	0.923	0.379	0.923	0.727	0.814	0.121
SVM	0.800 (0.674-0.926)	0.815	0.909	0.400	0.870	0.500	0.870	0.909	0.889	0.120
KNN	0.813 (0.697-0.928)	0.803	0.848	0.600	0.903	0.474	0.903	0.848	0.875	0.158
LGBM	0.728 (0.668-0.789)	0.753	0.848	0.333	0.848	0.333	0.848	0.848	0.848	0.163

CI: Confidence interval; SMOTE: Synthetic minority over-sampling technique; DT: Decision trees; KNM: K-nearest neighbors; LR: Logistic regression; RF: Random forests; LGBM: Light gradient boosting machines; SVM: Support vector machines; XGB: Extreme gradient boosting; AUC: Area under the receiver operating characteristic curve; NPV: Negative predictive value; PPV: Positive predictive value.

Table 5 DeLong test for different models and human radiologists in the validation cohort

Models	DT	LR	RF	XGB	SVM	KNN	LGBM	Senior radiologist	Junior radiologist
DT	-	0.011	0.118	< 0.001	< 0.001	0.022	0.309	0.235	0.179
LR	0.011	-	0.137	0.615	0.833	0.537	0.093	< 0.001	< 0.001
RF	0.118	0.137	-	0.038	0.016	0.100	0.950	0.031	0.017
XGB	< 0.001	0.615	0.038	-	0.469	0.825	0.239	< 0.001	< 0.001
SVM	< 0.001	0.833	0.016	0.469	-	0.535	0.159	< 0.001	< 0.001
KNN	0.022	0.537	0.100	0.825	0.535	-	0.321	0.002	< 0.001
LGBM	0.309	0.093	0.950	0.239	0.159	0.321	-	0.017	0.008
Senior radiologist	0.235	< 0.001	0.031	< 0.001	< 0.001	0.002	0.017	-	0.810
Junior radiologist	0.179	< 0.001	0.017	< 0.001	< 0.001	< 0.001	0.008	0.810	-

DT: Decision trees; KNM: K-nearest neighbors; LR: Logistic regression; RF: Random forests; LGBM: Light gradient boosting machines; SVM: Support vector machines; XGB: Extreme gradient boosting.