Retrospective Cohort Study Open Access
Copyright ©The Author(s) 2020. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Gastroenterol. Jul 14, 2020; 26(26): 3800-3813
Published online Jul 14, 2020. doi: 10.3748/wjg.v26.i26.3800
Nomogram for predicting transmural bowel infarction in patients with acute superior mesenteric venous thrombosis
Meng Jiang, Xin-Wu Cui, Sino-German Tongji-Caritas Research Center of Ultrasound in Medicine, Department of Medical Ultrasound, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, Hubei Province, China
Chang-Li Li, Department of Geratology, Hubei Provincial Hospital of Integrated Chinese and Western Medicine, Wuhan 430015, Hubei Province, China
Chun-Qiu Pan, Department of Emergency Medicine, Nanfang Hospital, Southern Medical University, Guangzhou 510515, Guangdong Province, China
Wen-Zhi Lv, Department of Artificial Intelligence, Julei Technology Company, Wuhan 430030, Hubei Province, China
Yu-Fei Ren, Department of Computer Center, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430030, Hubei Province, China
Christoph F Dietrich, Department of Internal Medicine, Hirslanden Clinic, Berne 27804, Switzerland
ORCID number: Meng Jiang (0000-0002-6669-547X); Chang-Li Li (0000-0001-8756-1473); Chun-Qiu Pan (0000-0003-3464-6363); Wen-Zhi Lv (0000-0002-4083-1779); Yu-Fei Ren (0000-0003-3749-8036); Xin-Wu Cui (0000-0003-3890-6660); Christoph F Dietrich (0000-0001-6382-6377).
Author contributions: Jiang M, Cui XW and Li CL designed the study; Ren YF and Cui XW supervised the study; Jiang M, Pan CQ and Lv WZ collected and analyzed the data; Ren YF performed the R Script; Jiang M prepared the figures and tables, and drafted the manuscript; Dietrich CF revised the manuscript for important intellectual content; Ren YF and Cui XW contributed equally as co-corresponding authors.
Supported by Wuhan Tongji Hospital, No. 2017A002; and Wuhan Science and Technology Bureau, No. 2017060201010181.
Institutional review board statement: This study was reviewed and approved by the Ethics Committee of the Tongji Medical college, Huazhong University of Science and Technology.
Informed consent statement: Patients were not required to give informed consent to the study because the analysis used anonymous clinical data that were obtained after each patient agreed to treatment by written consent.
Conflict-of-interest statement: We have no financial relationships to disclose.
Data sharing statement: No additional data are available.
STROBE statement: The authors have read the STROBE Statement-checklist of items, and the manuscript was prepared and revised according to the STROBE Statement-checklist of items.
Open-Access: This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Corresponding author: Xin-Wu Cui, MD, PhD, Professor, Deputy Director, Sino-German Tongji-Caritas Research Center of Ultrasound in Medicine, Department of Medical Ultrasound, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, No. 1095, Jiefang Avenue, Wuhan 430030, Hubei Province, China. cuixinwu@live.cn
Received: February 25, 2020
Peer-review started: February 25, 2020
First decision: May 22, 2020
Revised: May 23, 2020
Accepted: June 5, 2020
Article in press: June 5, 2020
Published online: July 14, 2020

Abstract
BACKGROUND

The prognosis of acute mesenteric ischemia (AMI) caused by superior mesenteric venous thrombosis (SMVT) remains undetermined and early detection of transmural bowel infarction (TBI) is crucial. The predisposition to develop TBI is of clinical concern, which can lead to fatal sepsis with hemodynamic instability and multi-organ failure. Early resection of necrotic bowel could improve the prognosis of AMI, however, accurate prediction of TBI remains a challenge for clinicians. When determining the eligibility for explorative laparotomy, the underlying risk factors for bowel infarction should be fully evaluated.

AIM

To develop and externally validate a nomogram for prediction of TBI in patients with acute SMVT.

METHODS

Consecutive data from 207 acute SMVT patients at the Wuhan Tongji Hospital and 89 patients at the Guangzhou Nanfang Hospital between July 2005 and December 2018 were included in this study. They were grouped as training and external validation cohort. The 207 cases (training cohort) from Tongji Hospital were divided into TBI and reversible intestinal ischemia groups based on the final therapeutic outcomes. Univariate and multivariate logistic regression analyses were conducted to identify independent risk factors for TBI using the training data, and a nomogram was subsequently developed. The performance of the nomogram was evaluated with respect to discrimination, calibration, and clinical usefulness in the training and external validation cohort.

RESULTS

Univariate and multivariate logistic regression analyses identified the following independent prognostic factors associated with TBI in the training cohort: The decreased bowel wall enhancement (OR = 6.37, P < 0.001), rebound tenderness (OR = 7.14, P < 0.001), serum lactate levels > 2 mmol/L (OR = 3.14, P = 0.009) and previous history of deep venous thrombosis (OR = 6.37, P < 0.001). Incorporating these four factors, the nomogram achieved good calibration in the training set [area under the receiver operator characteristic curve (AUC) 0.860; 95%CI: 0.771-0.925] and the external validation set (AUC 0.851; 95%CI: 0.796-0.897). The positive and negative predictive values (95%CIs) of the nomogram were calculated, resulting in positive predictive values of 54.55% (40.07%-68.29%) and 53.85% (43.66%-63.72%) and negative predictive values of 93.33% (82.14%-97.71%) and 92.24% (85.91%-95.86%) for the training and validation cohorts, respectively. Based on the nomogram, patients who had a Nomo-score of more than 90 were considered to have high risk for TBI. Decision curve analysis indicated that the nomogram was clinically useful.

CONCLUSION

The nomogram achieved an optimal prediction of TBI in patients with AMI. Using the model, the risk for an individual patient inclined to TBI can be assessed, thus providing a rational therapeutic choice.

Key Words: Superior mesenteric venous thrombosis, Acute mesenteric ischemia, Transmural bowel infarction, Reversible intestinal ischemia, Predictors, Nomogram

Core tip: The high mortality rate of acute superior mesenteric venous thrombosis is closely associated with the occurrence of transmural bowel infarction (TBI). Early detection and subsequent resection of irreversible necrotic intestine before sepsis and multi-organ failure could improve the functional outcome of the small bowel and patient prognosis. We found that the decreased bowel wall enhancement, rebound tenderness, serum lactate levels > 2 mmol/L and previous history of deep venous thrombosis independently predicted TBI. A nomogram that incorporated these four risk factors achieved an area under the receiver operator characteristic curve of 0.860 and 0.851 in the training and validation cohort, respectively, with good calibration. The nomogram can be conveniently used to facilitate the individualized prediction of TBI in patients with acute mesenteric ischemia.



INTRODUCTION

As a rare insidious condition, acute mesenteric venous thrombosis (MVT) has an incidence of 1 in 1000 emergency laparotomies for acute abdomen[1]. The prevalence of MVT has increased over the last two decades, likely as a result of the wide use of abdominal contrast-enhanced computed tomography (CT). Superior MVT (SMVT) is the most common type of MVT that accounts for approximately 5% - 10% of all mesenteric ischemia[2]. Despite advances in managing thromboembolic diseases over the past 40 years, the average 30-d mortality of acute SMVT is still as high as 32.1% in severe cases[3]. The predisposition to develop transmural bowel infarction (TBI) is of clinical concern, which can lead to fatal sepsis with hemodynamic instability and multi-organ failure[4].

Advances in imaging modalities, especially CT angiography, have greatly enabled early detection of SMVT in the setting of acute abdominal pain. Unfortunately, TBI is not rare, and intestinal resection is still mandatory for some patients. TBI often leads to a complex clinical situation and increases the patients' physiological burden, which poses a major challenge for clinicians. Thus, accurate and reliable prediction of bowel infarction is critical for decision-making in an emergency setting. Preventing the progression from reversible intestinal ischemia to TBI should be a primary goal in the management of SMVT.

Nomograms can provide individualized and highly accurate risk estimation, which are easy to use and can facilitate clinical decision-making. We undertook the present study to develop and externally validate a nomogram to predict TBI in patients with acute SMVT.

MATERIALS AND METHODS
Patients

Consecutive patients with SMVT from Tongji Hospital of Huazhong University of Science and Technology, Wuhan, China (as training cohort) and from Nanfang Hospital of Southern Medical University, Guangzhou, China (as validation cohort) were included. This retrospective study (clinical trial number: ChiCTR1900026320) was approved by the institutional review board of the two hospitals. Since the study was retrospectively designed and did not cause any harm to the patients, the informed consent was waived by the board.

We retrospectively reviewed the electronic medical record system using the key word “mesenteric venous thrombosis” to identify SMVT patients with acute mesenteric ischemia (AMI) who underwent contrast-enhanced CT at the two institutions. SMVT-related AMI was defined as the association of acute abdominal symptoms, CT features of bowel injury, as well as vascular insufficiency of the superior mesenteric vein. Two senior radiologists reviewed all CT images and confirmed the presence of thrombosis and bowel injury based on consensus. K statistics was used to evaluate the concordance between the two radiologists, and any disagreements were resolved by discussion. Diagnosis was confirmed pathologically in cases that intestinal resection was performed. To explore the effect of SMVT on bowel infarction without confounding risk factors, we excluded patients who had coexisting mesenteric artery thrombosis. In addition, cases in that acute SMVT was secondary to mechanical small bowel obstruction were also excluded.

Treatment protocol

At the two tertiary referral centers in China, we have employed a standard protocol for the treatment of acute SMVT. The treatment protocol included bowel rest, nasogastric tube decompression for patients with abdominal distention, intravenous fluids, prophylactic antibiotics, prompt anticoagulation or surgical exploration if necessary. For all patients, once acute SMVT was diagnosed, intravenous administration of unfractionated heparin (3000-5000 IU/d) or subcutaneous administration of low-molecular-weight heparin (LMWH or enoxaparin, 1 mg/kg per day) was applied at first. Thereafter, for patients with anticipated surgery, intravenous unfractionated heparin was administered according to the active partial thrombin time. Patients who received conservative treatment were injected with LMWH (1 mg/kg; BID) subcutaneously and monitored closely. Transmural bowel necrosis from resected specimens was confirmed histologically.

Outcomes

To determine the indicators of TBI, patients were divided into two groups according to the final therapeutic outcome. The TBI group was defined as: (1) Pathology assessment as extensive, transmural intestinal necrosis; (2) Definite imaging of bowel perforation on CT; or (3) Unresected patients with extensive bowel necrosis assessed during open-close laparotomy procedures. All resected specimens were retrospectively reviewed by a senior pathologist. The patients who did not progress to transmural bowel necrosis but recovered from AMI or superficial ischemic lesions after systemic anticoagulation or thrombectomy were categorized as reversible intestinal ischemia group. This was confirmed by explorative laparotomy, repeated CT scan or clinical follow-up.

Statistical analysis

Continuous variables were presented as median (interquartile range), and were compared between the training and validation cohort using Mann-Whitney U test or ttest as appropriate. Categorical variables were reported as whole numbers and proportions and were compared by the χ2 test or Fisher’s exact test where appropriate. Statistical analyses were conducted using R software (version 3.6.1) and SPSS 20.0 software (SPSS Inc., Chicago, IL, United States). All the statistical significance levels were two-sided, with P value less than 0.05. The detailed description of the decision curve analysis (DCA) algorithm is provided in the Supplementary text.

Data collection

Demographic and clinical data were extracted from case records, including age, gender, coexisting medical conditions (e.g., tobacco use, malignant disease, or previous history of deep venous thrombosis), clinical manifestations, physical findings and laboratory test results. All serum biochemical parameters were collected at the onset of symptoms. Radiologic features including extent of thrombus and associated conditions (e.g., decreased bowel wall enhancement, bowel wall thickening and pneumatosis intestinalis) were also recorded. We also extracted the suspected risk factors for acute SMVT, such as recent surgery, intraperitoneal inflammation, and liver cirrhosis.

Construction of the nomogram

Continuous variables were transformed into dichotomous variables using the upper value of normal as the cutoff level (e.g., white blood cell count > 10 × 109/L, percentage of neutrophil granulocyte > 75%, creatinine > 106 µmol/L, or venous lactate levels > 2 mmol/L), and C-reactive protein (50 mg/L as cutoff value in line with previous study[5].

Univariate logistic regression analysis was conducted to assess each variable in the training cohort for investigating the independent risk factors associated with TBI. Then, a multivariate logistic regression analysis incorporating all the significant risk variables was performed, using backward step-down selection procedure with a liberal P < 0.05 as the retention criteria to select the final indicators of TBI. A nomogram was developed based on the results of multivariate logistic regression analysis.

Performance of the nomogram

Calibration of the nomogram was evaluated using calibration curve and Hosmer-Lemeshow test (non-significance of the Hosmer-Lemeshow test indicates good agreement)[6]. The discrimination performance of the nomogram was assessed using the area under the receiver operator characteristic curve (AUC). The nomogram was subjected to bootstrapping validation (1000 bootstrap resamples) to calculate a relatively corrected AUC. Then the performance of the nomogram was tested in the external validation cohort by calibration curve and AUC.

Clinical use assessment

DCA was performed to estimate the clinical usefulness of the prediction model by quantifying the net benefits at different threshold probabilities[7,8]. For clinical use, the total scores (defined as Nomo-score in this study) of each case were calculated according to the nomogram algorithm. Then the optimal cutoff value of the Nomo-score was determined by maximizing the Youden index. Performance of the optimal cutoff value of the Nomo-score was assessed by the sensitivity, specificity, as well as positive and negative predictive values.

RESULTS
Patient selection

After excluding 12 patients with chronic SMVT, chart review yielded 230 consecutive patients who had acute SMVT in the training cohort between July 2005 and June 2018 (Wuhan cohort). Of the 230 patients, 13 cases were excluded from the analysis as the acute SMVT was secondary to mechanical small bowel obstruction. Acute SMVT concomitant with mesenteric artery thrombosis was found in 10 patients, who were also excluded, leaving a final sample of 207 patients. Explorative laparotomy was performed in 78 (38%) patients , and bowel resection was performed in 65 (83%) of these patients based on assessment of bowel viability with respect to color, dilatation and peristaltic motion of the bowel, pulsations of the mesenteric arcade arteries, as well as bleeding from cut surfaces. Thrombectomy was performed in 11 (14%) patients, and 2 (3%) received open-close procedure due to extensive bowel necrosis, who refused further treatment and died at last. The algorithm of patient screening in the training cohort is shown in Figure 1A. Pathological analysis of the surgical specimens confirmed TBI in 56 (86%) of the 65 patients, while superficial ischemic lesions were seen in 9 (14%) patients. The mean time between admission and surgical exploration for patients with and without TBI was 41.6 ± 30.5 (8-192) h and 32.4 ± 23.7 (5-96) h, respectively.

Figure 1
Figure 1 Flow diagram of study populations. A: Training cohort; B: Validation cohort. SMVT: Superior mesenteric venous thrombosis; AMI: Acute mesenteric ischemia.

In this study, patients with superficial ischemic lesions confirmed by pathological examination and clinically recovered through thrombectomy were classified as reversible intestinal ischemia group, while patients who underwent open-close procedure due to extensive bowel necrosis were deemed as TBI. One hundred twenty-nine patients (62%) who did not progress to surgery and recovered after conservative therapy were considered as having reversible intestinal ischemia. Eventually, reversible intestinal ischemia and TBI were the final diagnosis in 149 (72%) and 58 (28%) patients, respectively.

In the external validation cohort (Guangzhou cohort), 89 eligible cases were retrieved from August 2007 to December 2018 using the same criteria (Figure 1B). TBI was confirmed in 27 (30%) of these patients.

Characteristics of the study population

The patients’ clinical characteristics in the training and validation cohorts are summarized in Table 1. There were no differences in the clinicopathological characteristics between the two cohorts in most of the comparisons. In the training cohort, 82 patients had a clinical history of liver cirrhosis, and 20.7% (17/82) cases developed TBI. In the validation cohort, 18.9% (7/37) of the patients with liver cirrhosis progressed to TBI finally.

Table 1 Participant characteristics.
VariableCohort, n (%)
Training (n = 207)Validation (n = 89)P value
Age, yrs48.5 ± 13.251.3 ± 8.20.07
Gender0.52
Male145 (70.0)59 (66.3)
Female62 (30.0)30 (33.7)
Tobacco use40 (19.3)25 (28.1)0.09
Coexisting medical conditions
Hypertension19 (9.2)13 (14.6)0.17
Diabetes20 (9.7)13 (14.6)0.22
Ischemic heart disease8 (3.9)9 (10.1)0.05
Previous history of DVT39 (18.8)17 (19.1)0.81
Malignant disease24 (2.4)5 (5.6)0.11
Clinical manifestations
Diarrhea22 (10.6)14 (15.7)0.22
Hematochezia or melena62 (30.0)19 (21.3)0.13
Fever26 (12.6)13 (14.6)0.63
Physical findings
Abdominal distention109 (52.7)37 (41.6)0.16
Abdominal tenderness39 (18.8)24 (27.0)0.12
Rebound tenderness38 (18.4)21 (23.6)0.3
Laboratory findings
White blood cells, 109/L7.0 (4.7-11.4)11.0 (7.5-14.4)0.05
Hemoglobin, g/L109 (82-122)113 (92-130)0.11
Platelets, 109/L191 (100-288)168 (133-214)0.06
C-reactive protein, mg/L51.3 (7.7-67.5)53.9 (18.2-75.1)0.23
Serum lactate levels, mmol/L4.3 (2.7-6.5)3.9 (1.3-5.7)0.31
Creatinine, µmol/L72 (60-91)76 (65-108)0.07
D-dimer, mg/L16.9 (3.0-179.6)14.1 (6.9-60.2)0.61
Procalcitonin, µg/L0.5 (0.4-0.6)0.4 (0.1-1.8)0.17
Radiologic features
Distal thrombosis63 (30.4)25 (28.1%)0.69
Portal vein or/and splenic vein extension143 (69.1)54 (60.7)0.16
Decreased bowel wall enhancement70 (33.8)36 (40.4)0.28
Bowel wall thickening42 (21.3)19 (21.3)0.84
Bowel loop dilation41 (19.8)16 (18.0)0.71
Pneumatosis intestinalis52 (25.1)27 (30.3)0.13
Ascites91 (44.0)31 (34.8)0.14
Other risk factors
Recent surgery46 (22.2)15 (16.9)0.3
Intraperitoneal inflammation73 (35.3)31 (34.8)0.94
Liver cirrhosis82 (39.6)37 (41.6)0.75
Development and validation of a TBI-predicting nomogram

The results of univariate logistic analysis are presented in Table 2. Stepwise multivariate logistic regression indicated that the decreased bowel wall enhancement (OR = 6.37, P < 0.001), rebound tenderness (OR = 7.14, P < 0.001), serum lactate levels > 2 mmol/L (OR = 3.14, P = 0.009) and previous history of deep venous thrombosis (DVT) (OR = 6.37, P < 0.001) all independently predicted TBI (Table 3). These independently associated risk factors were used to construct a TBI risk estimation nomogram (Figure 2A). The scoring system is shown in Supplementary Table 1, which can be used for a more accurate calculation of predictions than drawing lines on the nomogram. Figure 2B shows the calibration curve of the nomogram. The calibration curve and Hosmer-Lemeshow test statistics (P = 0.316) showed good calibration in the training cohort. An AUC of 0.860 (95%CI: 0.771-0.925) also showed good discrimination by the nomogram (Figure 2D). The favorable calibration of the nomogram was also confirmed in the external validation set (Figure 2C). The Hosmer-Lemeshow test yielded a P value of 0.203, and the AUC of the validation cohort was 0.851 (95%CI: 0.796-0.897) (Figure 2E). Thus, our nomogram performed well in both the training and external validation sets.

Table 2 Univariate analysis of risk factors associated with transmural bowel infarction in the training cohort.
VariableOR (95%CI)P value
Age, yr0.97 (0.95-0.99)0.01
Gender, male vs female2.22 (1.06-4.65)0.03
Tobacco use1.03 (0.48-2.23)0.94
Hypertension1.10 (0.38-3.20)0.86
Diabetes3.85 (0.86-17.14)0.08
Ischemic heart disease2.81 (0.34-23.36)0.34
Previous history of DVT3.14 (1.53-6.47)0.002
Malignant disease0.41 (0.21-1.78)0.99
Diarrhea1.04 (0.39-2.81)0.93
Hematochezia2.53 (1.33-4.79)0.004
Fever1.16 (0.47-2.83)0.75
Abdominal distention3.56 (1.82-6.97)< 0.001
Abdominal tenderness7.25 (3.40-15.46)< 0.001
Rebound tenderness9.14 (4.17-20.04)< 0.001
White blood cells, > 10 × 109/L vs ≤ 10 × 109/L1.12 (1.06-1.17)< 0.001
Hemoglobin, > 120 g/L vs ≤ 120 g/L1.01 (1.00-1.02)0.35
Platelets, > 100 × 109/L vs ≤ 100 × 109/L1.00 (0.99-1.01)0.96
C-reactive protein, > 50 mg/L vs ≤ 50 mg/L1.02 (1.01-1.02)0.05
Serum lactate levels, > 2 mmol/L vs ≤ 2 mmol/L6.53 (3.27-13.04)< 0.001
Creatinine, > 106 µmol/L vs ≤ 106 µmol/L2.05 (0.96-4.41)0.07
D-dimer, > 0.5 mg/L vs ≤ 0.5 mg/L1.00 (1.00-1.01)0.50
Procalcitonin, > 0.5 µg/L vs ≤ 0.5 µg/L1.02 (0.92-1.12)0.72
Distal thrombosis2.13 (1.07-4.36)0.02
Portal vein or/and splenic vein extension1.01 (0.52-1.94)0.98
Decreased bowel wall enhancement9.00 (4.24-19.12)< 0.001
Bowel wall thickening1.99 (0.98-2.01)0.24
Bowel loop dilation2.21 (1.67-3.54)0.03
Pneumatosis intestinalis1.36 (1.23-3.15)0.04
Ascites1.40 (0.76-2.58)0.28
Recent surgery1.13 (0.54-2.38)0.74
Intraperitoneal inflammation1.63 (0.84-3.16)0.15
Liver cirrhosis2.44 (1.28-4.68)0.007
Figure 2
Figure 2 Nomogram for evaluation of transmural bowel infarction risk and its predictive performance. A: Nomogram for estimating the risk of transmural bowel infarction (TBI) in patients with acute mesenteric ischemia (AMI). The nomogram is used to find the position of each variable on the corresponding axis, draw a line to the points axis for the number of points, add the points from all of the variables, and draw a line from the total points axis to determine the risk of TBI at the lower line of the nomogram. For example, a 30-year-old male patient (have transmural bowel infarction finally) with a previous history of deep venous thrombosis and serum lactate levels > 2 mmol/L, no rebound tenderness and no decreased bowel wall enhancement, has a risk of 59% of TBI calculated by the nomogram; B and C: Calibration curves of the nomogram in the training and validation sets, respectively. Calibration curves depict the calibration of the model in terms of the agreement between the predicted probabilities of TBI and observed outcomes of TBI. The dotted blue line represents an ideal prediction, and the dotted red line represents the predictive ability of the nomogram. The closer the dotted red line fit is to the dotted blue line, the better the predictive accuracy of the nomogram is; D and E: The ROC curves of the nomogram in the training and validation sets, respectively. TBI: Transmural bowel infarction; AMI: Acute mesenteric ischemia; DVT: Deep venous thrombosis; AUC: Area under the receiver operator characteristic curve; ROC: Receiver operator characteristic curve.
Table 3 Multivariate logistic regression analysis of risk factors associated with transmural bowel infarction in the training cohort.
VariableβOR95%CIP value
Decreased bowel wall enhancement1.856.372.55-15.90< 0.001
Rebound tenderness1.977.142.73-18.65< 0.001
Serum lactate levels > 2 mmol/L1.443.141.33-7.410.009
DVT history1.816.372.55-15.90< 0.001

Decision curve analysis DCA was used to facilitate the assessment of the nomogram. Figure 3A shows the basic plot of model performance for the nomogram. The DCA graphically demonstrated the clinical value of the model based on a continuum of threshold for TBI risk (X axis) and the net benefit of using the model to stratify the risk of the patients (Y axis) relative to the hypothesis that no patient will have a TBI. The decision curve indicated that when the threshold probability for a patient or a doctor is within a range from 0 to 1.0, the nomogram adds more net benefit than the “treat-all” or “treat-none” schemes. Figure 3B shows the estimated number of patients who would be at high risk for each potential risk threshold and visually demonstrates the proportion of the patients who are truly positive cases. For instance, if a 40% risk threshold was used, of 1000 patients screened, about 200 patients would be deemed at high risk, with about 180 of these patients being true TBI cases.

Figure 3
Figure 3 Decision curve analysis for the nomogram. A: The Y-axis shows the net benefit. The X-axis shows the corresponding risk threshold. The green line represents the assumption that all patients have transmural bowel infarction (TBI). The thin black line represents the assumption that no patients have TBI. The red line represents the nomogram. The decision curve in the validation cohort indicated that if the threshold probability is between 0 and 1.0, then using the nomogram to predict TBI adds more benefit than the treat-all-patients scheme or treat-none scheme; B: Clinical impact curve for the risk model. Of 1000 patients, the red solid line shows the total number who would be deemed at high risk for each risk threshold. The green dashed line shows how many of those would be true positives cases. DCA: Decision curve analysis; TBI: Transmural bowel infarction.
Predicting TBI based on the Nomo-score

The optimal cutoff value of the Nomo-score was determined to be 90. The sensitivity, specificity, positive predictive value, and negative predictive value when used in predicting TBI were 88.89%, 67.74%, 54.55%, and 93.33% in the training cohort, and 84.84%, 71.81%, 53.85%, and 92.24% in the external validation cohort, respectively (Table 4).

Table 4 Performance of the nomogram for estimating the risk of transmural bowel infarction.
VariableValue (95%CI)
Training cohort (207)Validation cohort (89)
Cutoff value9090
AUC0.860 (0.771-0.925)0.851 (0.796-0.897)
Sensitivity, %88.89 (71.94-96.15)84.48 (73.07-91.62)
Specificity, %67.74 (55.37-78.05)71.81 (64.11-78.42)
Positive predictive value, %54.55 (40.07-68.29)53.85 (43.66-63.72)
Negative predictive value, %93.33 (82.14-97.71)92.24 (85.91-95.86)
Positive likelihood ratio2.76 (2.43-3.07)3.00 (2.84-3.16)
Negative likelihood ratio0.16 (0.08-0.32)0.22 (0.17-0.27)
Diagnostic accuracy, %74.16 (64.20-82.12)75.36 (69.07-80.73)
DISCUSSION

Acute SMVT is a rare but serious condition due to its intestinal ischemic complications. The widespread use of contrast-enhanced CT has made early diagnosis possible by a noninvasive approach, which can provide incremental information as evidence of ischemia warranting a change in treatment strategy. However, acute SMVT still carries a high risk of extensive intestinal infarction and surgical exploration with bowel resection is still mandatory for some patients. Recently, Kim et al[9] conducted a study involving 66 patients with acute SMVT, of whom 15 (23%) patients underwent bowel resection due to progressive intestinal ischemia and bowel infarction, and 3 (5%) patients died at last, despite adequate intravenous or subcutaneous anticoagulation were applied immediately at diagnosis. Another research reported the application of multidisciplinary stepwise management strategy for acute SMVT, and 18 of the 43 (42%) subjects underwent bowel resection due to TBI[10]. The reported rates of bowel infarction in existing literatures ranged from 6% to 42%, consistent with our rate of 29%[2,10-16]. Therefore, preventing the progression from reversible to irreversible ischemic bowel injury should be a primary goal in the management of acute SMVT[5]. However, the early detection of TBI remains a challenge and further investigation is required to identify the most important prognostic factors.

In this retrospective study, we developed and externally validated a nomogram to predict TBI in patients with acute SMVT. The decreased bowel wall enhancement, rebound tenderness, serum lactate levels > 2 mmol/L and previous history of DVT independently predicted this event. Incorporating these clinical, biological, and radiological factors into an easy-to-use prediction model can facilitate the individualized prediction of TBI. The nomogram also consistently predicted TBI with a high accuracy and provided good clinical usefulness throughout the range of bowel infarction risk as assessed by DCA.

Based on our findings, the previous history of DVT could increase the chances of bowel infarction in the setting of acute SMVT. Venous thrombosis often results from a combination of endothelial injury, hypercoagulability or stasis. The patients with a previous history of DVT may have had several episodes of undetected (possibly asymptomatic) MVT prior to the index event and thus have acute-on-chronic thrombosis with greater compromise of venous flow.

The decreased bowel wall enhancement on CT angiography is a strong established risk indictor for bowel infarction[9,17]. In our study, CT scan detected this feature in 70 patients, and among them, 34 patients had confirmed TBI finally. Previous studies demonstrated that the decreased bowel wall enhancement in patients with acute SMVT was significantly associated with surgical exploration and bowel resection[9,10].

Several other reports on AMI implied that the extent of venous thrombus was related to TBI[9,18]. However, Grisham et al[19] found no association between the presence of multiple vein thromboses and increased mortality rates. Additionally, Kumar et al[20] showed that patients with isolated SMVT were even at a greater risk of development of bowel infarction, and more likely to require surgery. In view of that the extent of venous thrombus did not show enough predictive strength on the basis of univariate association with bowel infarction, we excluded this variable for model construction.

Peritoneal signs, including involuntary guarding, rebound tenderness, and abdominal wall rigidity often present in the case of TBI. According to our results, the rebound tenderness could be a strong predictor for TBI. In parallel with our outcomes, Kim et al[9] found that rebound tenderness was observed more frequently in patients who underwent bowel resection than those who did not (33% vs 4%).

Lactate is an important parameter that closely related to necrosis, inflammation and hypoxia. Our results demonstrated that serum lactate levels > 2 mmol/L was significantly associated with the occurrence of TBI. In line with our findings, Higashizono and Nuzzo revealed that serum lactate levels tend to increase significantly after bowel infarction[5,21]. Additionally, Leone et al[22] indicated that serum lactate levels decreased significantly after resection of the necrotic bowel, further validating its predictive value for TBI.

Accurate identification of patients that will need surgery due to bowel infarction is crucial, because of the high morbidity and mortality associated with unidentified and unresected necrotic intestine[3,23,24]. Earlier resection of necrotic bowel before the development of multi-organ failure could improve the functional outcome of the small bowel and patient prognosis. However, this issue remains a challenge. In a study the authors presented 767 surgeons with a clinical vignette of AMI, and showed that the surgeons’ decision on whether a surgery is required or not varies markedly[25].

In a previous study, Nuzzo et al[5] developed a risk score incorporating organ failure, serum lactate levels > 2 mmol/L and bowel loop dilation on CT scan to predict irreversible transmural bowel necrosis for AMI. The risk factors associated with bowel necrosis identified in this study was partly different from ours, probably because their cohort comprised venous and arterial occlusion, but we just included acute SMVT patients. In addition, we suggested that organ failure might be of late onset during the disease course. Furthermore, we think that the risk score based on the number of predictive factors could not reflect the weight of each parameter.

Although several studies have investigated the risk factors of TBI for AMI, to our knowledge, none have presented the data in the form of a validated nomogram[5,9,26,27]. Nomograms have advantages as they provide quantified individual risk assessment in a dynamic manner. For clinical use of our nomogram, we proposed the sensitivity, specificity, negative and positive predictive value in assessing the risk of TBI using 90 as the cut-off value (Table 4). We show that patients with a Nomo-score less than 90 are the subgroup of low-risk inclined to TBI (negative predictive value, 93.33% for training and 92.24% for validation). The AUC of our model was 0.860 and 0.851 in the training and external validation cohort, respectively. The calibration curves presented a good agreement between the actual probability and predicted probability of TBI. Thus, we believe that our nomogram could be a reliable and objective tool that will provide clinicians favorable evidence for decision making.

Cautions should be exercised when interpreting the findings due to several limitations. First, some bias may inevitably exist and affect our analysis because it was a retrospective study. And thus, treatment strategy might not be entirely consistent among clinicians. Second, although we have validated the nomogram in an external cohort, the number of variables evaluated in respect to the number of primary outcome events may have led to an overfitting of the accuracy of the model, thus prospective multicenter validation using a larger group of patients is still necessary to acquire high-level evidence for further clinical application. Third, pathological evidence could not be obtained for patients that did not progress to surgery but confirmed by recovery from the specific vascular therapy. Fourth, evaluation of decreased bowel wall enhancement in the nomogram was related to the reading of CT images, and interpretation may vary among radiologists. In addition, since it is hard to assess the length of decreased bowel wall enhancement quantitatively on CT imaging, we did not examine the association between this factor and TBI. Further high-level evidence is needed to clarify this issue. Finally, data on the long-term outcomes of patients were unavailable, however, this will not affect our ability to identify risk factors of TBI associated with AMI.

In conclusion, our nomogram is an individual predictive tool that incorporates four risk factors, shows favorable predictive accuracy for assessing TBI risk in patients with AMI. The model might facilitate timely recognition and effective management of high-risk patients.

ARTICLE HIGHLIGHTS
Research background

The prognosis of acute mesenteric ischemia (AMI) caused by superior mesenteric venous thrombosis (SMVT) remains obscure and early detection of transmural bowel infarction (TBI) is crucial. The predisposition to develop TBI is of clinical concern, which can lead to fatal sepsis with hemodynamic instability and multiorgan failure. Early resection of necrotic bowel could improve the prognosis of AMI, however, accurate prediction of TBI remains a challenge for clinicians. When determining eligibility for explorative laparotomy, the underlying risk factors for bowel infarction should be fully evaluated.

Research motivation

Nomograms can provide individualized and highly accurate risk estimation, which are easy to use and can facilitate clinical decision-making. We undertook the present study to develop and externally validate a nomogram to predict TBI in patients with acute SMVT.

Research methods

Consecutive data from 207 acute SMVT patients at the Wuhan Tongji Hospital and 89 patients at the Guangzhou Nanfang Hospital between July 2005 and December 2018 were included in this study. They were grouped as training and external validation cohort. The 207 cases (training cohort) from Tongji hospital were divided into TBI and reversible intestinal ischemia groups based on the final therapeutic outcomes. Then univariate and multivariate logistic regression analyses were conducted to identify independent risk factors for TBI using the training data, and a nomogram was subsequently developed. The performance of the nomogram was evaluated with respect to discrimination, calibration, and clinical usefulness in the training and external validation cohort.

Research results

Univariate and multivariate logistic regression analyses identified the following independent prognostic factors associated with TBI in the training cohort: The decreased bowel wall enhancement (OR = 6.37, P < 0.001), rebound tenderness (OR = 7.14, P < 0.001), serum lactate levels > 2 mmol/L (OR = 3.14, P = 0.009) and previous history of deep venous thrombosis (OR = 6.37, P < 0.001). Incorporating these four factors, the nomogram achieved good calibration in the training set (AUC 0.860; 95%CI: 0.771-0.925) and the external validation set (AUC 0.851; 95%CI: 0.796-0.897). The positive and negative predictive values (95%CIs) of the nomogram were calculated, resulting in positive predictive values of 54.55% (40.07%-68.29%) and 53.85% (43.66%-63.72%) and negative predictive values of 93.33% (82.14%-97.71%) and 92.24% (85.91%-95.86%) for the training and validation cohorts, respectively. Based on the nomogram, patients who had a Nomo-score of more than 90 were considered to have high risk for TBI. Decision curve analysis indicated that the nomogram was clinically useful.

Research conclusions

The nomogram achieved an optimal prediction of TBI in patients with AMI. Using the model, the risk for an individual patient inclined to TBI can be assessed, thus providing a rational therapeutic choice.

Research perspectives

Although we have validated the nomogram in an external cohort, the number of variables evaluated in respect to the number of primary outcome events may have led to an overfitting of the accuracy of the model, thus prospective multicenter validation using a larger group of patients is still necessary to acquire high-level evidence for further clinical application.

Footnotes

Manuscript source: Invited manuscript

Specialty type: Gastroenterology and hepatology

Country/Territory of origin: China

Peer-review report’s scientific quality classification

Grade A (Excellent): 0

Grade B (Very good): B, B, B, B

Grade C (Good): C

Grade D (Fair): 0

Grade E (Poor): 0

P-Reviewer: Isaji S, Kimura Y, Leal RF, Richardson WS, Soresi M S-Editor: Dou Y L-Editor: MedE-Ma JY E-Editor: Zhang YL

References
1.  Blumberg SN, Maldonado TS. Mesenteric venous thrombosis. J Vasc Surg Venous Lymphat Disord. 2016;4:501-507.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 16]  [Cited by in F6Publishing: 9]  [Article Influence: 1.1]  [Reference Citation Analysis (0)]
2.  Brunaud L, Antunes L, Collinet-Adler S, Marchal F, Ayav A, Bresler L, Boissel P. Acute mesenteric venous thrombosis: case for nonoperative management. J Vasc Surg. 2001;34:673-679.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 129]  [Cited by in F6Publishing: 112]  [Article Influence: 4.9]  [Reference Citation Analysis (0)]
3.  Schoots IG, Koffeman GI, Legemate DA, Levi M, van Gulik TM. Systematic review of survival after acute mesenteric ischaemia according to disease aetiology. Br J Surg. 2004;91:17-27.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 297]  [Cited by in F6Publishing: 312]  [Article Influence: 15.6]  [Reference Citation Analysis (0)]
4.  Klar E, Rahmanian PB, Bücker A, Hauenstein K, Jauch KW, Luther B. Acute mesenteric ischemia: a vascular emergency. Dtsch Arztebl Int. 2012;109:249-256.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 30]  [Cited by in F6Publishing: 60]  [Article Influence: 5.0]  [Reference Citation Analysis (0)]
5.  Nuzzo A, Maggiori L, Ronot M, Becq A, Plessier A, Gault N, Joly F, Castier Y, Vilgrain V, Paugam C, Panis Y, Bouhnik Y, Cazals-Hatem D, Corcos O. Predictive Factors of Intestinal Necrosis in Acute Mesenteric Ischemia: Prospective Study from an Intestinal Stroke Center. Am J Gastroenterol. 2017;112:597-605.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 119]  [Cited by in F6Publishing: 116]  [Article Influence: 16.6]  [Reference Citation Analysis (0)]
6.  Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited. Crit Care Med. 2007;35:2052-2056.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 476]  [Cited by in F6Publishing: 584]  [Article Influence: 34.4]  [Reference Citation Analysis (0)]
7.  Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26:565-574.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 2176]  [Cited by in F6Publishing: 2913]  [Article Influence: 171.4]  [Reference Citation Analysis (0)]
8.  Kerr KF, Brown MD, Zhu K, Janes H. Assessing the Clinical Impact of Risk Prediction Models With Decision Curves: Guidance for Correct Interpretation and Appropriate Use. J Clin Oncol. 2016;34:2534-2540.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 224]  [Cited by in F6Publishing: 338]  [Article Influence: 42.3]  [Reference Citation Analysis (0)]
9.  Kim HK, Hwang D, Park S, Lee JM, Huh S. Treatment outcomes and risk factors for bowel infarction in patients with acute superior mesenteric venous thrombosis. J Vasc Surg Venous Lymphat Disord. 2017;5:638-646.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 13]  [Cited by in F6Publishing: 14]  [Article Influence: 2.0]  [Reference Citation Analysis (0)]
10.  Yang S, Fan X, Ding W, Liu B, Meng J, Xu D, He C, Yu W, Wu X, Li J. Multidisciplinary stepwise management strategy for acute superior mesenteric venous thrombosis: an intestinal stroke center experience. Thromb Res. 2015;135:36-45.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 27]  [Cited by in F6Publishing: 31]  [Article Influence: 3.1]  [Reference Citation Analysis (0)]
11.  Zhang J, Duan ZQ, Song QB, Luo YW, Xin SJ, Zhang Q. Acute mesenteric venous thrombosis: a better outcome achieved through improved imaging techniques and a changed policy of clinical management. Eur J Vasc Endovasc Surg. 2004;28:329-334.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 45]  [Cited by in F6Publishing: 48]  [Article Influence: 2.4]  [Reference Citation Analysis (0)]
12.  Amitrano L, Guardascione MA, Scaglione M, Pezzullo L, Sangiuliano N, Armellino MF, Manguso F, Margaglione M, Ames PR, Iannaccone L, Grandone E, Romano L, Balzano A. Prognostic factors in noncirrhotic patients with splanchnic vein thromboses. Am J Gastroenterol. 2007;102:2464-2470.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 200]  [Cited by in F6Publishing: 180]  [Article Influence: 10.6]  [Reference Citation Analysis (0)]
13.  Hedayati N, Riha GM, Kougias P, Huynh TT, Cheng C, Bechara C, Bismuth J, Dardik A, Lin PH. Prognostic factors and treatment outcome in mesenteric vein thrombosis. Vasc Endovascular Surg. 2008;42:217-224.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 36]  [Cited by in F6Publishing: 40]  [Article Influence: 2.5]  [Reference Citation Analysis (0)]
14.  Acosta S, Alhadad A, Svensson P, Ekberg O. Epidemiology, risk and prognostic factors in mesenteric venous thrombosis. Br J Surg. 2008;95:1245-1251.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 183]  [Cited by in F6Publishing: 181]  [Article Influence: 11.3]  [Reference Citation Analysis (0)]
15.  Zeng Q, Fu QN, Li FH, Wang XH, Liu H, Zhao Y. Early initiation of argatroban therapy in the management of acute superior mesenteric venous thrombosis. Exp Ther Med. 2017;13:1526-1534.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 8]  [Cited by in F6Publishing: 8]  [Article Influence: 1.1]  [Reference Citation Analysis (0)]
16.  Yang S, Zhang L, Liu K, Fan X, Ding W, He C, Wu X, Li J. Postoperative Catheter-Directed Thrombolysis Versus Systemic Anticoagulation for Acute Superior Mesenteric Venous Thrombosis. Ann Vasc Surg. 2016;35:88-97.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 12]  [Cited by in F6Publishing: 20]  [Article Influence: 2.5]  [Reference Citation Analysis (0)]
17.  Russell CE, Wadhera RK, Piazza G. Mesenteric venous thrombosis. Circulation. 2015;131:1599-1603.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 40]  [Cited by in F6Publishing: 39]  [Article Influence: 4.3]  [Reference Citation Analysis (0)]
18.  Gertsch P, Matthews J, Lerut J, Luder P, Blumgart LH. Acute thrombosis of the splanchnic veins. Arch Surg. 1993;128:341-345.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 64]  [Cited by in F6Publishing: 66]  [Article Influence: 2.1]  [Reference Citation Analysis (0)]
19.  Grisham A, Lohr J, Guenther JM, Engel AM. Deciphering mesenteric venous thrombosis: imaging and treatment. Vasc Endovascular Surg. 2005;39:473-479.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 42]  [Cited by in F6Publishing: 45]  [Article Influence: 2.5]  [Reference Citation Analysis (0)]
20.  Kumar S, Kamath PS. Acute superior mesenteric venous thrombosis: one disease or two? Am J Gastroenterol. 2003;98:1299-1304.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 90]  [Cited by in F6Publishing: 90]  [Article Influence: 4.3]  [Reference Citation Analysis (0)]
21.  Higashizono K, Yano H, Miyake O, Yamasawa K, Hashimoto M. Postoperative pneumatosis intestinalis (PI) and portal venous gas (PVG) may indicate bowel necrosis: a 52-case study. BMC Surg. 2016;16:42.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 15]  [Cited by in F6Publishing: 16]  [Article Influence: 2.0]  [Reference Citation Analysis (0)]
22.  Leone M, Bechis C, Baumstarck K, Ouattara A, Collange O, Augustin P, Annane D, Arbelot C, Asehnoune K, Baldési O, Bourcier S, Delapierre L, Demory D, Hengy B, Ichai C, Kipnis E, Brasdefer E, Lasocki S, Legrand M, Mimoz O, Rimmelé T, Aliane J, Bertrand PM, Bruder N, Klasen F, Friou E, Lévy B, Martinez O, Peytel E, Piton A, Richter E, Toufik K, Vogler MC, Wallet F, Boufi M, Allaouchiche B, Constantin JM, Martin C, Jaber S, Lefrant JY. Outcome of acute mesenteric ischemia in the intensive care unit: a retrospective, multicenter study of 780 cases. Intensive Care Med. 2015;41:667-676.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 100]  [Cited by in F6Publishing: 82]  [Article Influence: 9.1]  [Reference Citation Analysis (0)]
23.  Turner JR. Intestinal mucosal barrier function in health and disease. Nat Rev Immunol. 2009;9:799-809.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 2121]  [Cited by in F6Publishing: 2385]  [Article Influence: 159.0]  [Reference Citation Analysis (0)]
24.  Piton G, Belon F, Cypriani B, Regnard J, Puyraveau M, Manzon C, Navellou JC, Capellier G. Enterocyte damage in critically ill patients is associated with shock condition and 28-day mortality. Crit Care Med. 2013;41:2169-2176.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 72]  [Cited by in F6Publishing: 74]  [Article Influence: 6.7]  [Reference Citation Analysis (0)]
25.  Sacks GD, Dawes AJ, Ettner SL, Brook RH, Fox CR, Maggard-Gibbons M, Ko CY, Russell MM. Surgeon Perception of Risk and Benefit in the Decision to Operate. Ann Surg. 2016;264:896-903.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 47]  [Cited by in F6Publishing: 50]  [Article Influence: 7.1]  [Reference Citation Analysis (0)]
26.  Wang Y, Zhao R, Xia L, Cui YP, Zhou Y, Wu XT. Predictive Risk Factors of Intestinal Necrosis in Patients with Mesenteric Venous Thrombosis: Retrospective Study from a Single Center. Can J Gastroenterol Hepatol. 2019;2019:8906803.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 8]  [Cited by in F6Publishing: 8]  [Article Influence: 1.6]  [Reference Citation Analysis (0)]
27.  Elkrief L, Corcos O, Bruno O, Larroque B, Rautou PE, Zekrini K, Bretagnol F, Joly F, Francoz C, Bondjemah V, Cazals-Hatem D, Boudaoud L, De Raucourt E, Panis Y, Goria O, Hillaire S, Valla D, Plessier A. Type 2 diabetes mellitus as a risk factor for intestinal resection in patients with superior mesenteric vein thrombosis. Liver Int. 2014;34:1314-1321.  [PubMed]  [DOI]  [Cited in This Article: ]  [Cited by in Crossref: 34]  [Cited by in F6Publishing: 33]  [Article Influence: 3.3]  [Reference Citation Analysis (0)]