Published online Apr 28, 2014. doi: 10.3748/wjg.v20.i16.4787
Revised: February 9, 2014
Accepted: March 8, 2014
Published online: April 28, 2014
AIM: To estimate the validity of the point shear-wave elastography method by evaluating its reproducibility and accuracy for assessing liver stiffness.
METHODS: This was a single-center, cross-sectional study. Consecutive patients with chronic viral hepatitis scheduled for liver biopsy (LB) (Group 1) and healthy volunteers (Group 2) were studied. In each subject 10 consecutive point shear-wave elastography (PSWE) measurements were performed using the iU22 ultrasound system (Philips Medical Systems, Bothell, WA, United States). Patients in Group 1 underwent PSWE, transient elastography (TE) using FibroScan (Echosens, Paris, France) and ultrasound-assisted LB. For the assessment of PSWE reproducibility two expert raters (rater 1 and rater 2) independently performed the examinations. The performance of PSWE was compared to that of TE using LB as a reference standard. Fibrosis was staged according to the METAVIR scoring system. Receiver operating characteristic curve analyses were performed to calculate the area under the receiver operating characteristic curve (AUC) for F≥ 2, F≥ 3 and F = 4. The intraobserver and interobserver reproducibility of PSWE were assessed by calculating Lin’s concordance correlation coefficient.
RESULTS: To assess the performance of PSWE, 134 consecutive patients in Group 1 were studied. The median values of PSWE and TE (in kilopascals) were 4.7 (IQR = 3.8-5.4) and 5.5 (IQR = 4.7-6.5), respectively, in patients at the F0-F1 stage and 3.5 (IQR = 3.2-4.0) and 4.4 (IQR = 3.5-4.9), respectively, in the healthy volunteers in Group 2 (P < 10-5). In the univariate analysis, the PSWE and TE values showed a high correlation with the fibrosis stage; low correlations with the degree of necroinflammation, aspartate aminotransferase and gamma-glutamyl transferase (GGT); and a moderate negative correlation with the platelet count. A multiple regression analysis confirmed the correlations of both PSWE and TE with fibrosis stage and GGT but not with any other variables. The following AUC values were found: 0.80 (0.71-0.87) for PSWE and 0.82 (0.73-0.89) for TE (P = 0.42); 0.88 (0.80-0.94) for PSWE and 0.95 (0.88-0.98) for TE (P = 0.06); and 0.95 (0.89-0.99) for PSWE and 0.92 (0.85-0.97) for TE (P = 0.30) for F≥ 2, F≥ 3 and F = 4, respectively. To assess PSWE reproducibility, 116 subjects were studied, including 47 consecutive patients scheduled for LB (Group 1) and 69 consecutive healthy volunteers (Group 2). The intraobserver agreement ranged from 0.83 (95%CI: 0.79-0.88) to 0.96 (95%CI: 0.95-0.97) for rater 1 and from 0.84 (95%CI: 0.79-0.88) to 0.96 (95%CI: 0.95-0.97) for rater 2. The interobserver agreement yielded values from 0.83 (95%CI: 0.78-0.88) to 0.93 (95%CI: 0.91-0.95).
CONCLUSION: PSWE is a reproducible method for assessing liver stiffness, and it compares with TE. Compared with patients with nonsignificant fibrosis, healthy volunteers showed significantly lower values.
Core tip: The results of this study show that point shear-wave elastography (PSWE) is a highly reproducible method for assessing liver stiffness that is characterized by high levels of intraobserver and interobserver agreement, both overall and for single measurements. The PSWE performance compares with that of transient elastography (TE), the most widely accepted method for the noninvasive assessment of liver fibrosis. Compared with TE, routine ultrasound with elastography is advantageous because it allows the evaluation of other parameters that are complementary to stiffness, is highly accurate for the diagnosis of cirrhosis and can be used to screen for focal liver lesions.
- Citation: Ferraioli G, Tinelli C, Lissandrin R, Zicchetti M, Bello BD, Filice G, Filice C. Point shear wave elastography method for assessing liver stiffness. World J Gastroenterol 2014; 20(16): 4787-4796
- URL: https://www.wjgnet.com/1007-9327/full/v20/i16/4787.htm
- DOI: https://dx.doi.org/10.3748/wjg.v20.i16.4787
The prognosis and management of chronic viral hepatitis depend on the extent and progression of liver fibrosis, which constitute the most important predictor of disease outcome and influence the indication for antiviral treatment[1].
In the last decade, methods to noninvasively quantify liver fibrosis have been developed. The first available method was transient elastography (TE)[2-6]. Several studies have demonstrated a high accuracy of TE in identifying significant fibrosis (F > 2) and cirrhosis (F = 4) in patients with chronic hepatitis C[7-11].
The recent guidelines for the management of hepatitis C infection from the European Association for the Study of the Liver allow the use of TE, instead of liver biopsy (LB), in patients with chronic hepatitis C for assessing liver disease severity prior to therapy at a safe level of predictability[12]. TE has been approved by the French National Health Authority for the evaluation of fibrosis in treatment-naïve patients with chronic hepatitis C and no comorbidities[13].
Shear wave elastography techniques have been implemented in conventional real-time ultrasound systems, and several studies have shown their accuracy in the assessment of liver fibrosis[14-22]. Compared with TE, these techniques have the advantage of B-mode image guidance; thus, they can allow the user to choose the best acoustic window for correctly performing an examination in real time.
The aim of this study was to estimate the validity of a new point shear wave elastography (PSWE) technique by evaluating the reproducibility of measurements and the accuracy of this method in the assessment of liver fibrosis. The performance of PSWE was compared to that of TE using liver histology as a reference standard.
This was a single-center, cross-sectional study. All consecutive patients with chronic viral hepatitis who were scheduled for liver biopsy at the Infectious Diseases Department of Policlinico San Matteo were enrolled in the study (Group 1). Consecutive healthy volunteers were also enrolled (Group 2).
The accuracy of PSWE in the assessment of liver fibrosis was prospectively estimated in consecutive patients in Group 1. LB was performed on the same day as the PSWE and TE measurements, as day-case procedures. Examinations were performed in the morning after an overnight fast. The patients’ characteristics, epidemiological data and biochemical test results were recorded.
The reproducibility of PSWE measurements was prospectively assessed in consecutive subjects in Group 1 and Group 2. Two expert raters (rater 1 and rater 2) independently performed 10 consecutive measurements in each subject. All the subjects were asked to fast for at least six hours prior to the examination. The intraobserver agreement was assessed by comparing the median values of all the measurements and by comparing several combinations of measurements or single measurements. The interobserver agreement was assessed by comparing the median value of the 10 measurements performed in the same subject by each rater and by comparing combinations of measurements or single measurements.
Moreover, the results of liver stiffness measurements performed in patients with nonsignificant fibrosis (F0-F1) were compared to the values obtained in the healthy volunteers in Group 2.
Three physicians, each of whom was blinded to the other’s results, independently performed the measurements. The PSWE measurements were performed by G.F. and M.Z., and the TE measurements were performed by M.Z. and R.L.
The study protocol was approved by the institutional Ethics Committee. The participants provided written informed consent.
LB was performed by three experienced physicians (C.F., G.M. and E.B.) using a 17-gauge modified Menghini needle (Hepafix; Braun, Melsungen, Germany). The same intercostal space used for the TE and PSWE measurements was chosen for LB. The specimens were assessed on site by a single expert liver pathologist (B.D.B.) who was blind to both the TE and PSWE results. Liver fibrosis and necroinflammatory activity were evaluated semiquantitatively according to the METAVIR system[23]. Steatosis was graded according to the method of Kleiner et al[24] as S0, steatosis in fewer than 5% of hepatocytes; S1, 5%-33%; S2, 34%-66%; and S3, more than 66%.
TE measurements were performed using the M probe of the FibroScan® device by two physicians (M.Z. and R.L.) with experience performing at least 50 TE procedures. During the acquisition, the patients lay in the dorsal decubitus position with the right arm in maximum abduction. The results were expressed in kilopascals (kPa). Only examinations with 10 valid measurements and an interquartile range/mean (IQR/M) < 30% for values greater than 7.1 kPa were considered reliable[2,4,25].
The examinations were performed using the iU22 ultrasound system (Philips Healthcare, Bothell, WA, United States) with a convex broadband probe and the ElastPQ® technique. As with other shear wave elastography methods, this technique generates shear waves inside the liver using radiation force from a focused ultrasound beam. The ultrasound machine monitors the shear wave propagation using a Doppler-like ultrasound technique and measures the velocity of the shear wave. The shear wave velocity is displayed in meters per second (m/s) or in kPa through Young’s modulus E = 3 (vS2.ρ), where E is Young’s modulus, vS is the shear wave velocity and ρ is the density of the tissue. If the amount of non-shear wave motion exceeds a threshold, the system does not display a calculation.
The two raters performing the PSWE measurements (G.F. and M.Z.) had seven years and two years, respectively, of experience in real-time elastography studies. They received training in PSWE measurements for two days before the study began. The examinations were performed in the right lobe of the liver through intercostal spaces, with the subject lying supine with the right arm in maximal abduction. Using a real-time B-mode image, the rater selected a vessel-free area, at least 1.5 cm below Glisson’s capsule, where a fixed region of interest of 0.5 cm × 1.5 cm was placed by moving a trackball. The patients were instructed to hold their breath while the rater pressed a button that launched the data acquisition. Each rater performed 10 valid measurements, which were expressed in kPa. Measurements < 1 kPa were rejected by the raters.
Sample size considerations for the accuracy of PSWE: A total sample size of 130 subjects, which included 65 subjects with the disease, i.e., a prevalence of approximately 50%, was estimated to achieve 88% power to detect changes in sensitivity and in specificity from 0.75 to 0.90 using a two-sided binomial test. The target significance level was 0.05.
Sample size considerations for reproducibility of PSWE: A sample size of 100 subjects, with two observations per subject, was estimated to achieve 97% power to detect a concordance correlation of 0.95 under the alternative hypothesis when the concordance correlation under the null hypothesis was 0.90 using an F-test with a significance level of 0.05.
Descriptive statistics were produced for the demographic, clinical and laboratory characteristics of this study sample of patients. The Shapiro-Wilk test was used to test the normal distribution of quantitative variables. For quantitative variables that were normally distributed, the results were expressed as mean ± SD; otherwise, medians and interquartile ranges (IQR; 25th-75th percentile) were reported. Qualitative variables were summarized as counts and percentages. A one-way ANOVA or the Kruskal-Wallis analysis of variance by ranks, with a Bonferroni correction, was used to analyze differences among patients undergoing liver biopsy. Pearson’s or Spearman’s rank coefficient was used to identify correlations between two study variables.
Linear regression was used for the multivariate model. A frequency distribution was obtained to choose optimal cut-off values of PSWE and to maximize the sum of the sensitivity and specificity for different fibrosis thresholds: F0-F1 vs F2-F4 (F≥ 2), F0-F2 vs F3-F4 (F≥ 3) and F0-F3 vs F4 (F = 4). For TE, we used cut-off values determined in a previous study[26]. The diagnostic performance of PSWE, TE and their combinations was assessed using receiver operating characteristic (ROC) curves and an area under the ROC (AUC) curve analysis. Comparisons of AUCs were performed using the method described by DeLong et al[27] for correlated data. The Obuchowski measure was used to take into account all the pairwise comparisons between stages to minimize the spectrum effect and the risk of multiple testing[28].
Interobserver reproducibility was assessed by calculating Lin’s concordance correlation coefficient (CCC)[29]. The CCC combines measures of both precision and accuracy to determine the degree of deviation of the observed data from the line of perfect concordance (i.e., the line at 45 degrees on a square scatterplot). The CCC increases in value as a function of the proximity of the data’s reduced major axis to the line of perfect concordance (the accuracy of the data) and as a function of the tightness of the data about its reduced major axis (the precision of the data). CCC values range from 0 to +1. As CCC values approach 1, the measurement differences between the different raters become negligible and more consistent. The interobserver agreement was classified as poor (CCC = 0.00-0.20), fair to good (CCC = 0.40-0.75) or excellent (CCC > 0.75)[30]. The CCCs were reported with 95% confidence intervals (CIs).
The data analysis was performed with the STATA statistical package (release 11.1, 2010, Stata Corporation, College Station, Texas, United States) and MedCalc (version 11.2, 2011 MedCalc Software bvba, Ostend, Belgium).
From August 2011 through April 2013, one hundred and thirty-four consecutive subjects in Group 1 and sixty-nine subjects in Group 2 were prospectively studied. Data from some patients in Group 1 have been reported in previous studies[18,26].
One hundred forty patients were eligible during the recruitment period. Six patients were excluded because they were undergoing antiviral therapy. Due to patient recruitment from our referring physicians, there were no patients with overt cirrhosis or ascites in this series of patients. One hundred thirty-four patients met the inclusion criteria. LB was performed in all the patients on the same day as the PSWE and TE measurements, and no complications were observed. The specimen length was adequate for liver histology in all but one patient. TE was feasible in all but one patient, whereas the PSWE measurements failed in five patients. The PSWE measurement failures were due to narrow intercostal spaces in four cases and obesity in one.
The characteristics of the 134 patients are summarized in Table 1. The mean length of the LB specimens was 2.5 (0.78) cm. The results of the statistical analysis performed on the data from the 102 patients with chronic hepatitis C are provided hereafter.
| Characteristics | All patients(n = 134) | Patients with chronic hepatitis C (n = 102) |
| Sex, females | 29 (23.4) | 20 (21.3) |
| Age, yr (SD) | 43.70 (11.4) | 45.2 (11.0) |
| BMI, kg/m2 (SD) | 25.1 (4.5) | 25.2 (4.9) |
| AST, IU/L (IQR) | 46 (26-78) | 46 (30-83) |
| ALT, IU/L (IQR) | 69 (41-122) | 70 (43-127) |
| Alkaline phosphatase, IU/L (IQR) | 69 (59-87) | 70 (59-95) |
| GGT, IU/L (IQR) | 49 (26-80) | 50 (36-88) |
| Total bilirubin, M/L (IQR) | 0.62 (0.46-0.94) | 0.60 (0.44-0.89) |
| Platelet count, 103/mm3 (SD) | 227.8 (76.0) | 227.7 (70.1) |
| Prothrombin time, % (SD) | 94.0 (15.5) | 94.4 (17.0) |
| HCV infection | 102 (76.1) | - |
| HBV infection | 28 (20.9) | - |
| Other1 | 4 (3.0) | - |
| Fibrosis score (METAVIR)2 | ||
| F0 | 14 (10.5) | 6 (5.9) |
| F1 | 56 (42.1) | 44 (43.6) |
| F2 | 29 (21.8) | 24 (23.8) |
| F3 | 20 (15.0) | 17 (16.8) |
| F4 | 14 (10.5) | 10 (9.9) |
| Activity grade (METAVIR) | ||
| A0 | 12 (9.0) | 5 (4.9) |
| A1 | 77 (58.0) | 57 (56.4) |
| A2 | 28 (21.0) | 27 (26.7) |
| A3 | 16 (12.0) | 12 (11.9) |
| Steatosis grade | ||
| S0 | 83 (62.4) | 62 (61.3) |
| S1 | 30 (22.6) | 23 (22.8) |
| S2 | 16 (12.0) | 12 (11.9) |
| S3 | 4 (3.0) | 4 (4.0) |
| LSM, kPa (IQR) | 6.6 (5.0-8.9) | 6.5 (5.0-8.9) |
| PSWE, kPa (IQR) | 5.3 (3.9-6.6) | 5.2 (3.9-6.6) |
In the univariate analysis, the PSWE and TE values showed a high correlation with the fibrosis stage and low correlations with the degree of necroinflammation, aspartate aminotransferase (AST) and gamma-glutamyl transferase (GGT). The following values were obtained for PSWE and TE: (1) for liver fibrosis: r = 0.61 (P < 10-5) and r = 0.68 (P < 10-5); (2) for the degree of necroinflammation: r = 0.39 (P < 10-5) and r = 0.40 (P < 10-5); (3) for AST: r = 0.37 (P = 0.0002) and r = 0.32 (P = 0.001); and (4) for GGT: r = 0.48 (P < 10-5) and r = 0.43 (P < 10-5), respectively. The PSWE and TE values showed a moderate negative correlation with the platelet count (r = -0.34, P < 0.0002; r = -0.36, P < 10-5). No correlations with other variables, including steatosis, were found.
A multiple regression analysis that included METAVIR stage, METAVIR grade, AST, GGT and platelet count confirmed the correlations, for both PSWE and TE, with fibrosis stage and GGT, but not with any other variables. The corresponding coefficients for METAVIR stage and GGT were 1.66 (95%CI: 0.85-2.46; P < 10-5) and 0.007 (95%CI: 0.003-0.011; P = 0.002), respectively, for PSWE; and 3.05 (95%CI: 1.96-4.14; P < 10-5) and 0.007 (95%CI: 0.001-0.060; P = 0.002), respectively, for TE.
After corrections for gender and age, both the PSWE and TE values differed significantly between the patients with chronic hepatitis C at the F0-F1 stage (n=50) and the healthy volunteers (n = 69) (Figure 1).
Liver stiffness assessment: Comparison of PSWE and TE: The median values, interquartile ranges, ranges, numbers of outliers and P values of the measurements obtained for each fibrosis stage using PSWE and TE are shown in Figure 2.
The optimal cut-off values for different levels of fibrosis were determined by analyzing the ROCs for PSWE. For TE, we used cut-off values obtained in a previous study[26]. The cut-off values of PSWE and TE for each METAVIR stage, along with the AUCs, sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio and negative likelihood ratio, are presented in Table 2. Figure 3 shows the ROC curves for significant (F≥ 2) and severe fibrosis (F≥ 3), as well as for cirrhosis (F = 4). For staging advanced fibrosis (F≥ 3), TE had higher accuracy than PSWE, but this difference did not reach statistical significance. The Obuchowski measures were good for both PSWE [0.80 (95%CI: 0.73-0.86)] and TE [0.83 (95%CI: 0.77-0.90)].
| Parameter | Method | F≥2 | F≥3 | F= 4 |
| Cut-off in kPa | PSWE | 5.7 | 5.8 | 7.2 |
| TE | 6.9 | 7.3 | 9.3 | |
| AUC | PSWE | 0.80 (0.71-0.87) | 0.88 (0.80-0.94) | 0.95 (0.89-0.99) |
| TE | 0.82 (0.73-0.89) | 0.95 (0.88-0.98) | 0.92 (0.85-0.97) | |
| Sensitivity % | PSWE | 62.0 (47.2-75.3) | 85.2 (66.3-95.8) | 90.0 (55.5-99.7) |
| TE | 62.7 (48.1-75.9) | 89.9 (70.8-97.6) | 90.0 (55.5-99.7) | |
| Specificity % | PSWE | 91.7 (80.0-97.7) | 84.5 (74.0-92.0) | 88.6 (80.1-94.4) |
| TE | 83.7 (70.3-92.7) | 80.8 (69.9-89.1) | 87.8 (79.2-93.7) | |
| PPV % | PSWE | 88.6 (73.3-96.8) | 67.6 (49.5-82.6) | 47.4 (24.4-71.1) |
| TE | 80.0 (64.1-91.1) | 63.2 (45.7-78.4) | 45.0 (23.1-78.5) | |
| NPV % | PSWE | 69.8 (57.0-80.8) | 93.7 (84.7-98.3) | 98.7 (93.1-100) |
| TE | 68.3 (55.0-79.7) | 95.2 (86.5-99.0) | 98.7 (93.2-100) | |
| +LR | PSWE | 7.4 (2.8-19.5) | 5.5 (3.1-9.7) | 7.9 (4.3-14.7) |
| TE | 3.8 (2.0-7.5) | 4.6 (2.8-7.6) | 7.4 (4.1-13.3) | |
| -LR | PSWE | 0.4 (0.3-0.6) | 0.2 (0.07-0.4) | 0.1 (0.02-0.7) |
| TE | 0.4 (0.3-0.6) | 0.1 (0.05-0.4) | 0.1 (0.02-0.7) |
In total, 116 subjects were studied, including 47 consecutive patients scheduled for liver biopsy (Group 1) and 69 consecutive healthy volunteers (Group 2). The characteristics of the 116 subjects are summarized in Table 3. TE and PSWE were feasible in all the subjects.
| Characteristic | Total | Group 1 | Group 2 |
| n = 116 | n = 47 | n = 69 | |
| Sex, females (%) | 44 (37.9) | 10 (21.3)b | 34 (49.3) |
| Age, yr (SD) | 41.9 (13.5) | 44.1 (12.5)a | 38.9 (13.8) |
| BMI, kg/m2 (SD) | 23.2 (4.1) | 23.7 (2.7) | 23.0 (4.9) |
| PSWE measurement, rater 1 (IQR) | 3.95 (3.3-5.2) | 5.34 (4.1-7.3)b | 3.53 (3.2-4.1) |
| PSWE measurement, rater 2 (IQR) | 3.92 (3.3-5.2) | 5.34 (4.1-7.1)b | 3.52 (3.1-4.0) |
Intraobserver agreement: For the 116 subjects in Group 1 and Group 2, the intraobserver agreement ranged from 0.83 (95%CI: 0.79-0.88) to 0.96 (95%CI: 0.95-0.97) for rater 1 and from 0.84 (95%CI: 0.79-0.88) to 0.96 (95%CI: 0.95-0.97) for rater 2 (Table 4).
| Measurements | Rater 1 | Rater 2 | Rater 1 vs Rater 2 |
| CCC (95%CI) | CCC (95%CI) | CCC (95%CI) | |
| 1, 2, 3, 4, 5 vs 6, 7, 8, 9, 10 | 0.96 (0.95-0.97) | 0.96 (0.95-0.97) | - |
| 1, 3, 5, 7, 9 vs 2, 4, 6, 8, 10 | 0.95 (0.93-0.96) | 0.96 (0.94-0.97) | - |
| 1, 2, 3, 4, 5 | - | - | 0.91 (0.88-0.94) |
| 6, 7, 8, 9, 10 | - | - | 0.93 (0.91-0.95) |
| 1, 3, 5, 7, 9 | - | - | 0.93 (0.91-0.95) |
| 2, 4, 6, 8, 10 | - | - | 0.92 (0.90-0.94) |
| All | - | - | 0.93 (0.90-0.95) |
| 1 vs 5 | 0.93 (0.91-0.95) | 0.85 (0.80-0.90) | - |
| 5 vs 10 | 0.83 (0.79-0.88) | 0.88 (0.84-0.92) | - |
| 2 vs 5 | 0.87 (0.84-0.91) | 0.84 (0.79-0.89) | - |
| 3 vs 8 | 0.93 (0.91-0.95) | 0.84 (0.79-0.88) | - |
| 1 | - | - | 0.89 (0.85-0.93) |
| 2 | - | - | 0.83 (0.78-0.88) |
| 5 | - | - | 0.85 (0.80-0.90) |
| 8 | - | - | 0.84 (0.79-0.90) |
When each group was considered separately, the CCC ranged from 0.77 (95%CI: 0.68-0.87) to 0.96 (95%CI: 0.94-0.97) for both raters in Group 1, from 0.80 (95%CI: 0.66-0.94) to 0.81 (95%CI: 0.67-0.94) for rater 1 and from 0.82 (95%CI: 0.69-0.94) to 0.83 (95%CI: 0.72-0.95) for rater 2 in Group 2.
Interobserver agreement: For the 116 subjects in Group 1 and Group 2, the interobserver agreement yielded CCC values ranging from 0.83 (95%CI: 0.78-0.88) to 0.93 (95%CI: 0.91-0.95) (Table 4).
When each group was considered separately, the interobserver agreement ranged from 0.76 (95%CI: 0.66-0.86) to 0.93 (95%CI: 0.88-0.96) in Group 1 and from 0.75 (95%CI: 0.58-0.95) to 0.83 (95%CI: 0.70-0.95) in Group 2.
This study was undertaken to assess the validity of PSWE, i.e., the repeatability of measurements and the performance of this method. The results show that PSWE is a highly reproducible method for assessing liver stiffness because it was characterized by very high levels of intraobserver and interobserver agreement, both overall and for single measurements. Moreover, the reproducibility of the method was similar in healthy subjects and in patients with chronic viral hepatitis. Ultrasound imaging techniques are subject to user dependency; nonetheless, we observed a high interobserver agreement rate that was similar to that reported for TE[9]. Nevertheless, good interobserver agreement rates have been reported for other shear wave elastography ultrasound-based techniques, suggesting that the method itself has low variability and requires only a short period of training to be performed reliably[31-33]. Indeed, the benefits of image guidance will likely reduce the learning curve and the variations between measurements[33].
The results of this study show that TE and PSWE results are directly and linearly correlated with the stages of fibrosis determined using histology. Furthermore, the performance of PSWE compares with that of TE, the first available technique and the most widely accepted method for noninvasive assessment of liver fibrosis. In our series, liver stiffness did not correlate with liver steatosis in either the univariate or multivariate analysis; thus, steatosis was not a confounding variable. This result is similar to those observed in other studies using shear wave elastography techniques integrated into ultrasound systems, and this result appears to indicate that the value obtained was a true estimate of the stiffness of the liver[14,16,18,19,34]. The influence of necroinflammation on liver stiffness is controversial; some studies have found an influence[9,11,14,19,34], and others have not[2-4,16,18,22]. In our series, we found no correlation between liver stiffness and necroinflammation. A positive correlation with GGT was found, which is in agreement with the results of a study by Forns et al[35], which identified this variable as an independent predictor of liver fibrosis.
The diagnostic accuracy of PSWE was similar to that reported by some other studies, which used a different point shear wave elastography method that also included acoustic radiation force impulse (ARFI); those studies reported no improvement in accuracy relative to TE for staging liver fibrosis[14,20,21,36]. Rizzo et al[16] found that ARFI was more accurate than TE for the staging of both significant and severe liver fibrosis. However, those results were not confirmed by a recent meta-analysis that compared ARFI with TE and found comparable diagnostic accuracies of both methods for the diagnosis of severe fibrosis and a slightly but significantly higher diagnostic accuracy of TE for the diagnosis of significant fibrosis and cirrhosis[17].
PSWE is a recently developed method that is part of the second generation of ultrasound elastography methods. These methods differ from the first-generation TE in several aspects, including the generation of shear waves within the organ by a focused ultrasound beam and the capability of focusing the beam at different locations within the organ under ultrasound image guidance. These properties should improve the feasibility of stiffness measurements in obese patients and patients with ascites; they may also improve the accuracy of PSWE relative to TE. However, the current study demonstrated that neither the feasibility nor the accuracy of PSWE was higher than that of TE. This finding could be attributable to the fact that the patients in our series had a body mass index within the normal range and the absence of patients with ascites. On the other hand, compared with TE, routine ultrasound systems with an elastography technique are advantageous in that they also allow the evaluation of other parameters that are complementary to stiffness, they are highly accurate for the diagnosis of cirrhosis and they could be used to screen for focal liver lesions[20,37].
The optimal cut-offs identified for each fibrosis stage, which were based on the maximal sensitivity and specificity, were close to each other. However, the diagnostic accuracy, assessed with AUCs, was high, suggesting that the PSWE method is acceptable for staging liver fibrosis but needs to be refined. On the other hand, liver histology could be an imperfect gold standard because it is affected by intraobserver and interobserver variabilities in fibrosis assessment and represents only 1/50000 of the entire liver mass[38]. Moreover, a recent study showed that liver biopsy exhibited a relative lower level of performance compared with FibroTest and TE when evaluated similarly for the diagnosis of advanced fibrosis[39]. As was very recently reported for TE[25], in our study, both TE and PSWE showed excellent negative predictive value for cirrhosis and very good positive predictive value for significant fibrosis. On the contrary, both techniques showed insufficient positive predictive value for cirrhosis and only fair negative predictive value for significant fibrosis.
The values of stiffness obtained using PSWE in healthy subjects were significantly lower than those obtained in patients with nonsignificant fibrosis (F0-F1) based on liver histology. This result indicates that the PSWE technique, which is noninvasive and readily available in ultrasound systems, could be a useful adjunct tool when performing ultrasound examinations of the liver because this method may allow physicians to select patients who need to be further evaluated for chronic liver disease.
Our study has limitations. First, the different stages of fibrosis, particularly advanced fibrosis and cirrhosis, were not equally represented among the patients in our series; almost half of the patients were at the F0-F1 stage, which may have affected the optimal cut-off values obtained with the ROC curves. This uneven distribution of fibrosis stages among consecutive patients reflects what is normally observed in clinical settings. On the other hand, the Obuchowski measure, which was used to minimize the spectrum bias, was good for both PSWE and TE. Second, our study population had a low prevalence of obesity, which could be a technical limitation; thus, the applicability of these results is limited. Third, the analysis was performed in a relatively small number of patients; thus, these results need to be validated in larger studies.
In conclusion, PSWE is a highly reproducible method for assessing liver stiffness. For staging liver fibrosis, PSWE compares favorably with TE. Healthy volunteers show significantly lower values compared with patients with nonsignificant fibrosis. Further studies in larger series of patients are needed to confirm these results.
The authors would like to thank all the collaborators in the Liver Fibrosis Study Group: Elisabetta Above, MD; Giorgio Barbarini, MD; Raffaele Bruno, MD; Silvia Corona, MSc; Carolina Dellafiore, MD; Marta Di Gregorio, MD; Roberto Gulminetti, MD; Paolo Lanzarini, MD; Serena Ludovisi, MD; Laura Maiocchi, MD; Antonello Malfitano, MD; Giuseppe Michelone, MD; Lorenzo Minoli, MD; Mario Mondelli, MD; Stefano Novati, MD; Savino FA Patruno, MD; Alessandro Perretti, MD; Gianluigi Poma, MD; Paolo Sacchi, MD; Domenico Zanaboni, MD; and Marco Zaramella, MD. The authors would also like to thank the following for their valuable help in complying with the study protocol: Ms. Livia Astroni, Ms. Natali Calabrese, Mr. Filippo Cuda, Mr. Lorenzo Guioli, Ms. Maura Marchisoni, Ms. Giampiera Nava, Ms. Loredana Pavesi and Ms. Barbara Ricci, who are nurses in the outpatient ward of the Infectious Diseases Department, and Ms. Nadia Locatelli, the secretary of the Ultrasound Unit. The authors are very grateful to Enrico Brunetti, MD, for performing some of the liver biopsies.
In the last decade, methods to noninvasively quantify liver fibrosis have been developed. Ultrasound point shear wave elastography is a noninvasive technique that is implemented in an ultrasound system and is able to assess the mechanical properties of tissues. The stiffness of tissues increases under pathological conditions, such as liver fibrosis. Compared with transient elastography, this technique has the advantage of B-mode image guidance, allowing the best acoustic window for correctly performing an examination to be selected in real time.
The assessment of liver fibrosis is of outmost importance in the management of patients with chronic viral hepatitis. Ultrasound point shear wave elastography is an affordable technique that could help reduce costs compared with more invasive procedures, such as liver biopsy.
The results of this study show that point shear wave elastography values are directly and linearly correlated to the stages of fibrosis identified by histology. Furthermore, the performance of this technique compares with that of transient elastography, which is the most widely accepted method for the noninvasive assessment of liver fibrosis. In our study, both transient elastography and point shear wave elastography showed excellent negative predictive value for cirrhosis and very good positive predictive value for significant fibrosis.
Point shear wave elastography, which is a noninvasive technique readily available in an ultrasound system, could be a useful adjunct tool when performing ultrasound examinations of the liver because it may allow physicians to select patients who need to be further evaluated for chronic liver disease.
Point shear wave elastography is an ultrasound-based technique that generates shear waves inside the liver using the radiation force from a focused ultrasound beam. The ultrasound machine monitors the shear wave propagation and measures the velocity of the shear wave. Assuming that the tissue exhibits very simple behavior, the shear wave velocity is related to stiffness through Young’s modulus. The unit of measure of stiffness is the kilopascal (kPa).
The authors investigated a point shear wave elastography method for assessing liver stiffness. This paper seems to be important and promising.
P- Reviewer: Nakajima A S- Editor: Ma YJ L- Editor: A E- Editor: Zhang DN
| 1. | European Association of the Study of the Liver. 2011 European Association of the Study of the Liver hepatitis C virus clinical practice guidelines. Liver Int. 2012;32 Suppl 1:2-8. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 64] [Cited by in F6Publishing: 84] [Article Influence: 7.0] [Reference Citation Analysis (0)] |
| 2. | Sandrin L, Fourquet B, Hasquenoph JM, Yon S, Fournier C, Mal F, Christidis C, Ziol M, Poulet B, Kazemi F. Transient elastography: a new noninvasive method for assessment of hepatic fibrosis. Ultrasound Med Biol. 2003;29:1705-1713. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1967] [Cited by in F6Publishing: 1813] [Article Influence: 86.3] [Reference Citation Analysis (0)] |
| 3. | Ziol M, Handra-Luca A, Kettaneh A, Christidis C, Mal F, Kazemi F, de Lédinghen V, Marcellin P, Dhumeaux D, Trinchet JC. Noninvasive assessment of liver fibrosis by measurement of stiffness in patients with chronic hepatitis C. Hepatology. 2005;41:48-54. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1090] [Cited by in F6Publishing: 1062] [Article Influence: 55.9] [Reference Citation Analysis (0)] |
| 4. | Castéra L, Vergniol J, Foucher J, Le Bail B, Chanteloup E, Haaser M, Darriet M, Couzigou P, De Lédinghen V. Prospective comparison of transient elastography, Fibrotest, APRI, and liver biopsy for the assessment of fibrosis in chronic hepatitis C. Gastroenterology. 2005;128:343-350. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1796] [Cited by in F6Publishing: 1760] [Article Influence: 92.6] [Reference Citation Analysis (0)] |
| 5. | Friedrich-Rust M, Ong MF, Martens S, Sarrazin C, Bojunga J, Zeuzem S, Herrmann E. Performance of transient elastography for the staging of liver fibrosis: a meta-analysis. Gastroenterology. 2008;134:960-974. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1046] [Cited by in F6Publishing: 1011] [Article Influence: 63.2] [Reference Citation Analysis (1)] |
| 6. | Tsochatzis EA, Gurusamy KS, Ntaoula S, Cholongitas E, Davidson BR, Burroughs AK. Elastography for the diagnosis of severity of fibrosis in chronic liver disease: a meta-analysis of diagnostic accuracy. J Hepatol. 2011;54:650-659. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 490] [Cited by in F6Publishing: 484] [Article Influence: 37.2] [Reference Citation Analysis (0)] |
| 7. | Castera L, Forns X, Alberti A. Non-invasive evaluation of liver fibrosis using transient elastography. J Hepatol. 2008;48:835-847. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 972] [Cited by in F6Publishing: 974] [Article Influence: 60.9] [Reference Citation Analysis (0)] |
| 8. | Talwalkar JA, Kurtz DM, Schoenleber SJ, West CP, Montori VM. Ultrasound-based transient elastography for the detection of hepatic fibrosis: systematic review and meta-analysis. Clin Gastroenterol Hepatol. 2007;5:1214-1220. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1] [Cited by in F6Publishing: 2] [Article Influence: 0.1] [Reference Citation Analysis (0)] |
| 9. | Fraquelli M, Rigamonti C, Casazza G, Conte D, Donato MF, Ronchi G, Colombo M. Reproducibility of transient elastography in the evaluation of liver fibrosis in patients with chronic liver disease. Gut. 2007;56:968-973. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 629] [Cited by in F6Publishing: 618] [Article Influence: 36.4] [Reference Citation Analysis (0)] |
| 10. | Castera L. Transient elastography and other noninvasive tests to assess hepatic fibrosis in patients with viral hepatitis. J Viral Hepat. 2009;16:300-314. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 119] [Cited by in F6Publishing: 108] [Article Influence: 7.2] [Reference Citation Analysis (0)] |
| 11. | Arena U, Vizzutti F, Abraldes JG, Corti G, Stasi C, Moscarella S, Milani S, Lorefice E, Petrarca A, Romanelli RG. Reliability of transient elastography for the diagnosis of advanced fibrosis in chronic hepatitis C. Gut. 2008;57:1288-1293. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 206] [Cited by in F6Publishing: 217] [Article Influence: 13.6] [Reference Citation Analysis (0)] |
| 12. | European Association for the Study of the Liver. EASL Clinical Practice Guidelines: management of hepatitis C virus infection. J Hepatol. 2011;55:245-264. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 889] [Cited by in F6Publishing: 968] [Article Influence: 74.5] [Reference Citation Analysis (0)] |
| 13. | Méthodes non invasives de mesure de la fibrose hépatique diagnostic de la cirrhose non compliquée. accessed January 2, 2014. Available from: http://www.has-sante.fr/portail/upload/docs/application/pdf/2009-05/document_avis_fibrose_cirrhose_dec_2008.pdf. [Cited in This Article: ] |
| 14. | Ebinuma H, Saito H, Komuta M, Ojiro K, Wakabayashi K, Usui S, Chu PS, Umeda R, Ishibashi Y, Takayama T. Evaluation of liver fibrosis by transient elastography using acoustic radiation force impulse: comparison with Fibroscan(®). J Gastroenterol. 2011;46:1238-1248. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 85] [Cited by in F6Publishing: 82] [Article Influence: 6.3] [Reference Citation Analysis (0)] |
| 15. | Karlas T, Pfrepper C, Wiegand J, Wittekind C, Neuschulz M, Mössner J, Berg T, Tröltzsch M, Keim V. Acoustic radiation force impulse imaging (ARFI) for non-invasive detection of liver fibrosis: examination standards and evaluation of interlobe differences in healthy subjects and chronic liver disease. Scand J Gastroenterol. 2011;46:1458-1467. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 75] [Cited by in F6Publishing: 81] [Article Influence: 6.2] [Reference Citation Analysis (0)] |
| 16. | Rizzo L, Calvaruso V, Cacopardo B, Alessi N, Attanasio M, Petta S, Fatuzzo F, Montineri A, Mazzola A, L’abbate L. Comparison of transient elastography and acoustic radiation force impulse for non-invasive staging of liver fibrosis in patients with chronic hepatitis C. Am J Gastroenterol. 2011;106:2112-2120. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 142] [Cited by in F6Publishing: 144] [Article Influence: 11.1] [Reference Citation Analysis (0)] |
| 17. | Friedrich-Rust M, Nierhoff J, Lupsor M, Sporea I, Fierbinteanu-Braticevici C, Strobel D, Takahashi H, Yoneda M, Suda T, Zeuzem S. Performance of Acoustic Radiation Force Impulse imaging for the staging of liver fibrosis: a pooled meta-analysis. J Viral Hepat. 2012;19:e212-e219. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 343] [Cited by in F6Publishing: 329] [Article Influence: 27.4] [Reference Citation Analysis (0)] |
| 18. | Ferraioli G, Tinelli C, Dal Bello B, Zicchetti M, Filice G, Filice C. Accuracy of real-time shear wave elastography for assessing liver fibrosis in chronic hepatitis C: a pilot study. Hepatology. 2012;56:2125-2133. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 476] [Cited by in F6Publishing: 468] [Article Influence: 39.0] [Reference Citation Analysis (0)] |
| 19. | Lupsor M, Badea R, Stefanescu H, Sparchez Z, Branda H, Serban A, Maniu A. Performance of a new elastographic method (ARFI technology) compared to unidimensional transient elastography in the noninvasive assessment of chronic hepatitis C. Preliminary results. J Gastrointestin Liver Dis. 2009;18:303-310. [PubMed] [Cited in This Article: ] |
| 20. | Crespo G, Fernández-Varo G, Mariño Z, Casals G, Miquel R, Martínez SM, Gilabert R, Forns X, Jiménez W, Navasa M. ARFI, FibroScan, ELF, and their combinations in the assessment of liver fibrosis: a prospective study. J Hepatol. 2012;57:281-287. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 118] [Cited by in F6Publishing: 121] [Article Influence: 10.1] [Reference Citation Analysis (0)] |
| 21. | Friedrich-Rust M, Wunder K, Kriener S, Sotoudeh F, Richter S, Bojunga J, Herrmann E, Poynard T, Dietrich CF, Vermehren J. Liver fibrosis in viral hepatitis: noninvasive assessment with acoustic radiation force impulse imaging versus transient elastography. Radiology. 2009;252:595-604. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 452] [Cited by in F6Publishing: 451] [Article Influence: 30.1] [Reference Citation Analysis (0)] |
| 22. | Takahashi H, Ono N, Eguchi Y, Eguchi T, Kitajima Y, Kawaguchi Y, Nakashita S, Ozaki I, Mizuta T, Toda S. Evaluation of acoustic radiation force impulse elastography for fibrosis staging of chronic liver disease: a pilot study. Liver Int. 2010;30:538-545. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 167] [Cited by in F6Publishing: 168] [Article Influence: 12.0] [Reference Citation Analysis (0)] |
| 23. | Bedossa P, Poynard T. An algorithm for the grading of activity in chronic hepatitis C. The METAVIR Cooperative Study Group. Hepatology. 1996;24:289-293. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 2860] [Cited by in F6Publishing: 2932] [Article Influence: 104.7] [Reference Citation Analysis (0)] |
| 24. | Kleiner DE, Brunt EM, Van Natta M, Behling C, Contos MJ, Cummings OW, Ferrell LD, Liu YC, Torbenson MS, Unalp-Arida A. Design and validation of a histological scoring system for nonalcoholic fatty liver disease. Hepatology. 2005;41:1313-1321. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 6807] [Cited by in F6Publishing: 7396] [Article Influence: 389.3] [Reference Citation Analysis (5)] |
| 25. | Boursier J, Zarski JP, de Ledinghen V, Rousselet MC, Sturm N, Lebail B, Fouchard-Hubert I, Gallois Y, Oberti F, Bertrais S. Determination of reliability criteria for liver stiffness evaluation by transient elastography. Hepatology. 2013;57:1182-1191. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 392] [Cited by in F6Publishing: 418] [Article Influence: 38.0] [Reference Citation Analysis (0)] |
| 26. | Ferraioli G, Tinelli C, Dal Bello B, Zicchetti M, Lissandrin R, Filice G, Filice C, Above E, Barbarini G, Brunetti E. Performance of liver stiffness measurements by transient elastography in chronic hepatitis. World J Gastroenterol. 2013;19:49-56. [PubMed] [DOI] [Cited in This Article: ] [Cited by in CrossRef: 29] [Cited by in F6Publishing: 32] [Article Influence: 2.9] [Reference Citation Analysis (0)] |
| 27. | DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837-845. [PubMed] [Cited in This Article: ] |
| 28. | Obuchowski NA. Estimating and comparing diagnostic tests’ accuracy when the gold standard is not binary. Acad Radiol. 2005;12:1198-1204. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 79] [Cited by in F6Publishing: 70] [Article Influence: 3.7] [Reference Citation Analysis (0)] |
| 29. | Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989;45:255-268. [PubMed] [Cited in This Article: ] |
| 30. | Fleiss JL. The measurement of interrater agreement. Statistical method for rates and proportions. New York, NY: John Wiley & Sons 1981; 211-236. [DOI] [Cited in This Article: ] [Cited by in Crossref: 8] [Cited by in F6Publishing: 9] [Article Influence: 0.2] [Reference Citation Analysis (0)] |
| 31. | Bota S, Sporea I, Sirli R, Popescu A, Danila M, Costachescu D. Intra- and interoperator reproducibility of acoustic radiation force impulse (ARFI) elastography--preliminary results. Ultrasound Med Biol. 2012;38:1103-1108. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 80] [Cited by in F6Publishing: 67] [Article Influence: 5.6] [Reference Citation Analysis (0)] |
| 32. | Boursier J, Konaté A, Gorea G, Reaud S, Quemener E, Oberti F, Hubert-Fouchard I, Dib N, Calès P. Reproducibility of liver stiffness measurement by ultrasonographic elastometry. Clin Gastroenterol Hepatol. 2008;6:1263-1269. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 97] [Cited by in F6Publishing: 101] [Article Influence: 6.3] [Reference Citation Analysis (0)] |
| 33. | Ferraioli G, Tinelli C, Zicchetti M, Above E, Poma G, Di Gregorio M, Filice C. Reproducibility of real-time shear wave elastography in the evaluation of liver elasticity. Eur J Radiol. 2012;81:3102-3106. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 175] [Cited by in F6Publishing: 174] [Article Influence: 14.5] [Reference Citation Analysis (0)] |
| 34. | Rifai K, Cornberg J, Mederacke I, Bahr MJ, Wedemeyer H, Malinski P, Bantel H, Boozari B, Potthoff A, Manns MP. Clinical feasibility of liver elastography by acoustic radiation force impulse imaging (ARFI). Dig Liver Dis. 2011;43:491-497. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 98] [Cited by in F6Publishing: 104] [Article Influence: 8.0] [Reference Citation Analysis (0)] |
| 35. | Forns X, Ampurdanès S, Llovet JM, Aponte J, Quintó L, Martínez-Bauer E, Bruguera M, Sánchez-Tapias JM, Rodés J. Identification of chronic hepatitis C patients without hepatic fibrosis by a simple predictive model. Hepatology. 2002;36:986-992. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 672] [Cited by in F6Publishing: 751] [Article Influence: 34.1] [Reference Citation Analysis (0)] |
| 36. | Boursier J, Isselin G, Fouchard-Hubert I, Oberti F, Dib N, Lebigot J, Bertrais S, Gallois Y, Calès P, Aubé C. Acoustic radiation force impulse: a new ultrasonographic technology for the widespread noninvasive diagnosis of liver fibrosis. Eur J Gastroenterol Hepatol. 2010;22:1074-1084. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 100] [Cited by in F6Publishing: 103] [Article Influence: 7.4] [Reference Citation Analysis (0)] |
| 37. | Berzigotti A, Abraldes JG, Tandon P, Erice E, Gilabert R, García-Pagan JC, Bosch J. Ultrasonographic evaluation of liver surface and transient elastography in clinically doubtful cirrhosis. J Hepatol. 2010;52:846-853. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 75] [Cited by in F6Publishing: 65] [Article Influence: 4.6] [Reference Citation Analysis (0)] |
| 38. | Bravo AA, Sheth SG, Chopra S. Liver biopsy. N Engl J Med. 2001;344:495-500. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 1678] [Cited by in F6Publishing: 1647] [Article Influence: 71.6] [Reference Citation Analysis (0)] |
| 39. | Poynard T, de Ledinghen V, Zarski JP, Stanciu C, Munteanu M, Vergniol J, France J, Trifan A, Le Naour G, Vaillant JC. Relative performances of FibroTest, Fibroscan, and biopsy for the assessment of the stage of liver fibrosis in patients with chronic hepatitis C: a step toward the truth in the absence of a gold standard. J Hepatol. 2012;56:541-548. [PubMed] [DOI] [Cited in This Article: ] [Cited by in Crossref: 64] [Cited by in F6Publishing: 60] [Article Influence: 5.0] [Reference Citation Analysis (0)] |