Integrating ultrasound and serum indicators for evaluating outcomes of targeted immunotherapy in advanced liver cancer

Table 1 Basic information of all patients, n (%)

Variable	All patients	Group 1	Group 2	Group 3	Test statistic (F/H/χ²)	P value
Diameter	8.46 (7.09, 9.31)	8.46 (7.27, 9.32)	8.43 (7.32, 9.24)	8.28 (7.01, 9.31)	1.4	0.49
Age	51.00 (46.00, 57.00)	50.00 (45.00, 56.00)	52.00 (47.25, 62.00)	50.00 (45.00, 55.00)	4.41	0.11
PV	1.10 (1.00, 1.18)	1.06 (1.00, 1.15)	1.10 (1.00, 1.17)	1.10 (1.00, 1.20)	2.05	0.35
Height	160.00 (155.00, 162.00)	159.00 (155.00, 162.00)	159.50 (155.00, 162.00)	160.00 (156.00, 162.00)	2.13	0.34
Weight	57.00 (52.00, 62.00)	56.50 (52.00, 62.00)	56.50 (51.25, 62.00)	57.50 (52.00, 62.00)	1.11	0.57
BMI	22.46 (21.00, 24.22)	22.49 (21.04, 24.13)	22.10 (21.04, 24.17)	22.43 (21.00, 24.61)	0.51	0.77
WBC	5.29 (4.56, 6.55)	5.29 (4.58, 6.68)	5.42 (4.60, 6.09)	5.28 (4.56, 6.50)	0.32	0.85
Hemoglobin	123.00 (110.25, 131.00)	122.00 (110.50, 131.00)	126.50 (116.50, 131.75)	120.00 (110.00, 130.00)	2.19	0.33
PLT	238.50 (201.25, 280.00)	241.00 (202.00, 279.00)	231.50 (191.00, 255.00)	244.00 (205.00, 284.00)	2.94	0.23
Alanine aminotransferase	59.00 (34.00, 108.00)	65.00 (41.00, 123.50)	50.00 (31.25, 82.00)	56.00 (33.00, 88.00)	6.52	0.03
Aspartate aminotransferase	68.00 (41.00, 94.00)	73.00 (43.00, 105.00)	57.00 (34.75, 88.25)	68.00 (40.00, 93.00)	2.91	0.23
Alkaline phosphatase	235.00 (132.25, 377.50)	243.00 (150.50, 382.50)	192.50 (132.25, 300.00)	248.00 (111.00, 482.00)	1.9	0.38
γ-glutamyl transpeptidase	241.00 (108.00, 442.75)	247.00 (125.50, 448.00)	166.50 (108.00, 373.25)	204.00 (103.00, 448.00)	1.78	0.41
Albumin	39.00 (35.00, 42.00)	38.00 (35.00, 42.00)	39.00 (35.00, 43.00)	38.00 (35.00, 41.00)	1.01	0.60
Total bilirubin	16.75 (11.25, 28.80)	17.30 (11.70, 31.30)	14.80 (11.00, 20.10)	15.60 (11.00, 24.80)	2.92	0.23
Direct bilirubin	3.65 (2.40, 9.20)	4.20 (2.40, 13.05)	3.10 (2.23, 5.47)	3.70 (2.40, 8.10)	3.81	0.14
Indirect bilirubin	12.10 (8.60, 17.30)	12.50 (9.35, 18.00)	11.70 (8.60, 16.73)	11.10 (8.30, 17.00)	1.97	0.37
Cholesterol	5.25 (4.26, 6.90)	5.26 (4.26, 6.90)	5.33 (4.30, 6.59)	5.16 (4.12, 7.50)	0.06	0.96
Prothrombin time	12.10 (11.60, 12.50)	12.00 (11.60, 12.50)	12.20 (11.60, 12.50)	12.20 (11.70, 12.60)	0.69	0.70
Prothrombin time activity	100.00 (100.00, 125.00)	100.00 (100.00, 128.50)	100.00 (100.00, 112.00)	100.00 (100.00, 132.00)	2.09	0.35
International normalized ratio	1.00 (0.94, 1.00)	1.00 (0.96, 1.00)	1.00 (0.94, 1.00)	1.00 (0.94, 1.00)	0.05	0.97
TIN	79 (25.82)	40 (27.97)	11 (17.74)	28 (27.72)	2.65	0.26
TIE	227 (74.18)	103 (72.03)	51 (82.26)	73 (72.28)	2.65	0.26
NLR > 3	140 (45.75)	65 (45.45)	28 (45.16)	47 (46.53)	0.04	0.98
NLR ≤ 3	166 (54.25)	78 (54.55)	34 (54.84)	54 (53.47)	0.04	0.98
ΔT non-increasement	136 (44.44)	65 (45.45)	22 (35.48)	49 (48.51)	2.75	0.25
ΔT increasement	170 (55.56)	78 (54.55)	40 (64.52)	52 (51.49)	2.75	0.25
ΔP non-increasement	245 (80.07)	114 (79.72)	51 (82.26)	80 (79.21)	0.24	0.88
ΔP increasement	61 (19.93)	29 (20.28)	11 (17.74)	21 (20.79)	0.24	0.88
ECOG (0)	213 (69.61)	95 (66.43)	42 (67.74)	76 (75.25)	2.3	0.31
ECOG (1)	93 (30.39)	48 (33.57)	20 (32.26)	25 (24.75)	2.3	0.31
Male	268 (87.58)	125 (87.41)	55 (88.71)	88 (87.13)	0.1	0.95
Female	38 (12.42)	18 (12.59)	7 (11.29)	13 (12.87)	0.1	0.95
Uneven echo	108 (35.29)	57 (39.86)	19 (30.65)	32 (31.68)	2.47	0.29
Homogeneous echo	198 (64.71)	86 (60.14)	43 (69.35)	69 (68.32)	2.47	0.29
Smooth envelope	37 (12.09)	19 (13.29)	5 (8.06)	13 (12.87)	1.2	0.54
Rough envelope	269 (87.91)	124 (86.71)	57 (91.94)	88 (87.13)	1.2	0.54
Drinking history (Yes)	78 (25.49)	40 (27.97)	10 (16.13)	28 (27.72)	3.59	0.16
Drinking history (No)	228 (74.51)	103 (72.03)	52 (83.87)	73 (72.28)	3.59	0.16
AFP > 100 ng/mL	195 (63.73)	89 (62.24)	38 (61.29)	68 (67.33)	0.86	0.64
AFP ≤ 100 ng/mL	111 (36.27)	54 (37.76)	24 (38.71)	33 (32.67)	0.86	0.64
CRP > 1 mg/dL	176 (57.52)	86 (60.14)	33 (53.23)	57 (56.44)	0.92	0.63
CRP ≤ 1 mg/dL	130 (42.48)	57 (39.86)	29 (46.77)	44 (43.56)	0.92	0.63
Rough gallbladder	213 (69.61)	95 (66.43)	42 (67.74)	76 (75.25)	2.3	0.31
Smooth gallbladder	93 (30.39)	48 (33.57)	20 (32.26)	25 (24.75)	2.3	0.31
Macrovascular invasion (Yes)	192 (62.75)	85 (59.44)	39 (62.90)	68 (67.33)	1.58	0.45
Macrovascular invasion (No)	114 (37.25)	58 (40.56)	23 (37.10)	33 (32.67)	1.58	0.45
Viral infection (Yes)	269 (87.91)	124 (86.71)	59 (95.16)	86 (85.15)	3.98	0.13
Viral infection (No)	37 (12.09)	19 (13.29)	3 (4.84)	15 (14.85)	3.98	0.13
TNM (IIIA)	46 (15.03)	21 (14.69)	8 (12.90)	17 (16.83)	8.79	0.18
TNM (IIIB)	61 (19.93)	23 (16.08)	12 (19.35)	26 (25.74)	8.79	0.18
TNM (IIIC)	92 (30.07)	47 (32.87)	24 (38.71)	21 (20.79)	8.79	0.18
TNM (IV)	107 (34.97)	52 (36.36)	18 (29.03)	37 (36.63)	8.79	0.18
BCLC (B)	61 (19.93)	28 (19.58)	10 (16.13)	23 (22.77)	1.67	0.79
BCLC (C)	153 (50.00)	69 (48.25)	34 (54.84)	50 (49.50)	1.67	0.79
BCLC (D)	92 (30.07)	46 (32.17)	18 (29.03)	28 (27.72)	1.67	0.79
CNLC (II)	61 (19.93)	35 (24.48)	10 (16.13)	16 (15.84)	4.75	0.31
CNLC (III)	183 (59.80)	81 (56.64)	36 (58.06)	66 (65.35)	4.75	0.31
CNLC (IV)	62 (20.26)	27 (18.88)	16 (25.81)	19 (18.81)	4.75	0.31
HKLC (II)	62 (20.26)	27 (18.88)	12 (19.35)	23 (22.77)	3.59	0.73
HKLC (III)	92 (30.07)	43 (30.07)	15 (24.19)	34 (33.66)	3.59	0.73
HKLC (IV)	122 (39.87)	57 (39.86)	28 (45.16)	37 (36.63)	3.59	0.73
HKLC (V)	30 (9.80)	16 (11.19)	7 (11.29)	7 (6.93)	3.59	0.73
CRAFITY (0)	48 (15.69)	23 (16.08)	12 (19.35)	13 (12.87)	1.63	0.80
CRAFITY (1)	145 (47.39)	65 (45.45)	29 (46.77)	51 (50.50)	1.63	0.80
CRAFITY (2)	113 (36.93)	55 (38.46)	21 (33.87)	37 (36.63)	1.63	0.80

PV: Portal vein width; BMI: Body mass index; WBC: White blood cell; PLT: Platelet count; TIN: Target and immunotherapy noneffective; TIE: Target and immunotherapy effective; NLR: Neutrophil to lymphocyte ratio; ECOG: Eastern Cooperative Oncology Group; AFP: Alpha fetoprotein; CRP: C-reactive protein; TNM: Tumor-node-metastasis; BCLC: Barcelona Clinic Liver Cancer; CNLC: China Liver Cancer; HKLC: Hong Kong Liver Cancer; CRAFITY: C-reactive protein and alpha-fetoprotein in immunotherapy; ΔT: The difference in stiffness of the tumor six weeks after treatment minus the stiffness before treatment; ΔP: The difference in stiffness of the surrounding tissue six weeks after treatment minus the stiffness before treatment.

Table 2 Performance of various machine learning algorithms in the internal validation group

Model	AUC	AUC 95%CI	Accuracy	Sensitivity	Specificity	PPV	NPV
Random forest	0.975	0.924-0.998	0.951	0.727	0.981	0.957	0.944
Gradient boosting	0.864	0.798-0.902	0.983	0.909	0.852	0.942	0.980
LightGBM	0.834	0.801-0.895	0.967	0.818	0.891	0.944	0.962
Voting classifier	0.902	0.881-0.972	0.983	0.909	0.948	0.879	0.980
Support vector classifier	0.822	0.728-0.915	0.967	0.818	0.927	0.943	0.962
Logistic regression	0.728	0.708-0.857	0.967	0.909	0.980	0.909	0.980
XGBoost	0.851	0.799-0.924	0.983	0.909	0.893	0.915	0.980
Extra trees	0.901	0.854-0.95	0.935	0.727	0.980	0.888	0.943
K-nearest neighbors	0.798	0.705-0.867	0.887	0.636	0.941	0.701	0.923
Decision tree	0.837	0.757-0.903	0.919	0.95	0.901	0.687	0.913
Naive Bayes	0.878	0.797-0.960	0.838	0.818	0.843	0.529	0.955
AdaBoost	0.868	0.783-0.952	0.919	0.818	0.941	0.751	0.962
Ridge classifier	0.842	0.798-0.895	0.967	0.909	0.980	0.909	0.980

AUC: Area under the curve; CI: Confidence interval; PPV: Positive predict value; NPV: Negative predict value; LightGBM: Light gradient boosting machine; XGBoost: Extreme gradient boosting.

Table 3 Prediction performance of different models in different groups

	Model	AUC	AUC 95%CI	Sensitivity	Specificity	PPV	NPV	Accuracy
Internal training cohort	TIPM	0.895	0.845-0.945	0.65	0.961	0.866	0.876	0.874
	TNM	0.541	0.460-0.623	0.672	0.721	0.735	0.655	0.624
	BCLC	0.446	0.364-0.527	0.685	0.734	0.724	0.751	0.675
	CNLC	0.500	0.419-0.582	0.648	0.654	0.812	0.802	0.741
	HKLC	0.526	0.444-0.608	0.724	0.801	0.751	0.754	0.657
	CRAFITY	0.571	0.490-0.652	0.85	0.721	0.654	0.793	0.751
Internal validation cohort	TIPM	0.907	0.835-0.979	0.636	0.980	0.875	0.925	0.919
	TNM	0.515	0.375-0.624	0.624	0.655	0.561	0.688	0.675
	BCLC	0.614	0.492-0.735	0.715	0.457	0.558	0.724	0.742
	CNLC	0.405	0.283-0.527	0.537	0.621	0.597	0.739	0.567
	HKLC	0.432	0.308-0.555	0.522	0.439	0.621	0.751	0.744
	CRAFITY	0.524	0.400-0.649	0.727	0.702	0.701	0.755	0.685
External validation cohort	TIPM	0.899	0.840-0.957	0.642	0.972	0.901	0.876	0.881
	TNM	0.518	0.420-0.615	0.577	0.715	0.544	0.674	0.655
	BCLC	0.474	0.376-0.571	0.622	0.702	0.658	0.587	0.724
	CNLC	0.584	0.488-0.680	0.478	0.658	0.728	0.653	0.726
	HKLC	0.505	0.408-0.603	0.685	0.653	0.648	0.724	0.658
	CRAFITY	0.546	0.449-0.644	0.892	0.654	0.722	0.769	0.685

AUC: Area under the curve; CI: Confidence interval; PPV: Positive predict value; NPV: Negative predict value; TIPM: Target Immunotherapy Predictive Model; TNM: Tumor-node-metastasis; BCLC: Barcelona Clinic Liver Cancer; CNLC: China Liver Cancer; HKLC: Hong Kong Liver Cancer; CRAFITY: C-reactive protein and alpha-fetoprotein in immunotherapy.

Table 4 Comparison of calibration performance of different models in different groups

	Model	Brier score	HL statistic	HL P value	ECE	MCE
Internal training cohort	TIPM	0.111	3.945	0.861	0.111	0.251
	TNM	0.429	25000	< 0.01	0.353	0.692
	BCLC	0.435	81000	< 0.01	0.433	0.782
	CNLC	0.344	14400	< 0.01	0.416	0.629
	HKLC	0.317	16000	< 0.01	0.318	0.625
	CRAFITY	0.400	36000	< 0.01	0.393	0.636
Internal validation cohort	TIPM	0.086	5.456	0.707	0.221	0.647
	TNM	0.446	40000	< 0.01	0.447	0.833
	BCLC	0.362	10000	< 0.01	0.391	0.722
	CNLC	0.419	40000	< 0.01	0.471	0.937
	HKLC	0.367	90000	< 0.01	0.475	0.925
	CRAFITY	0.423	90000	< 0.01	0.469	0.761
External validation cohort	TIPM	0.098	10.11	0.256	0.102	0.287
	TNM	0.427	25000	< 0.01	0.541	0.702
	BCLC	0.400	49000	< 0.01	0.304	0.75
	CNLC	0.301	10000	< 0.01	0.062	0.684
	HKLC	0.309	49000	< 0.01	0.548	0.714
	CRAFITY	0.403	90000	< 0.01	0.230	0.675

HL: Hosmer-Lemeshow; ECE: Expected calibration error; MCE: Maximum calibration error; TIPM: Target Immunotherapy Predictive Model; TNM: Tumor-node-metastasis; BCLC: Barcelona Clinic Liver Cancer; CNLC: China Liver Cancer; HKLC: Hong Kong Liver Cancer; CRAFITY: C-reactive protein and alpha-fetoprotein in immunotherapy.

Table 5 Comparison of net reclassification index and integrated discrimination improvement between the Target Immunotherapy Predictive Model and other classical models

Model	Internal training cohort, NRI	Internal training cohort, IDI	Internal validation cohort, NRI	Internal validation cohort, IDI	External validation cohort, NRI	External validation cohort, IDI
TIPM vs TNM	0.175	0.374	0.258	0.481	0.418	0.435
TIPM vs BCLC	0.330	0.497	0.238	0.330	0.605	0.492
TIPM vs CNLC	0.387	0.423	0.440	0.587	0.534	0.355
TIPM vs HKLC	0.278	0.385	0.602	0.558	0.754	0.454
TIPM vs CRAFITY	0.225	0.340	0.329	0.451	0.484	0.401

NRI: Net reclassification index; IDI: Integrated discrimination improvement; TIPM: Target Immunotherapy Predictive Model; TNM: Tumor-node-metastasis; BCLC: Barcelona Clinic Liver Cancer; CNLC: China Liver Cancer; HKLC: Hong Kong Liver Cancer; CRAFITY: C-reactive protein and alpha-fetoprotein in immunotherapy.