Evaluation of computer aided detection during colonoscopy among Veterans: Randomized clinical trial

Abstract

BACKGROUND

There has been significant interest in use of computer aided detection (CADe) devices in colonoscopy to improve polyp detection and reduce miss rate.

AIM

To investigate the use of CADe amongst veterans.

METHODS

Between September 2020 and December 2021, we performed a randomized controlled trial to evaluate the impact of CADe. Patients at Veterans Affairs Palo Alto Health Care System presenting for screening or low-risk surveillance were randomized to colonoscopy performed with or without CADe. Primary outcomes of interest included adenoma detection rate (ADR), adenomas per colonoscopy (APC), and adenomas per extraction. In addition, we measured serrated polyps per colonoscopy, non-adenomatous, non-serrated polyps per colonoscopy, serrated polyp detection rate, and procedural time.

RESULTS

A total of 244 patients were enrolled (124 with CADe), with similar patient characteristics (age, sex, body mass index, indication) between the two groups. Use of CADe was found to have decreased number of adenomas (1.79 vs 2.53, P = 0.030) per colonoscopy compared to without CADe. There was no significant difference in number of serrated polyps or non-adenomatous non-serrated polyps per colonoscopy between the two groups. Overall, use of CADe was found to have lower ADR (68.5% vs 80.0%, P = 0.041) compared to without use of CADe. Serrated polyp detection rate was lower with CADe (3.2% vs 7.5%) compared to without CADe, but this was not statistically significant (P = 0.137). There was no significant difference in withdrawal and procedure times between the two groups or in detection of adenomas per extraction (71.4% vs 73.1%, P = 0.613). No adverse events were identified.

CONCLUSION

While several randomized controlled trials have demonstrated improved ADR and APC with use of CADe, in this RCT performed at a center with high ADR, use of CADe was found to have decreased APC and ADR. Further studies are needed to understand the true impact of CADe on performance quality among endoscopists as well as determine criteria for endoscopists to consider when choosing to adopt CADe in their practices.

Key Words: Colonoscopy, Colorectal cancer prevention, Artificial intelligence, Computer aided detection, Adenoma detection rate

Core Tip: While several randomized controlled trials have demonstrated improved adenoma detection rate (ADR) and adenomas per colonoscopy (APC) with use of computer aided detection (CADe), in this RCT performed by endoscopists with high ADR, use of CADe was found to be associated with decreased APC and ADR. The results of this study suggest that CADe may not be the right tool for every endoscopist. Further studies are needed to understand the impact of CADe on performance quality among endoscopists as well as determine criteria for endoscopists to consider when choosing to adopt CADe in their practices.

INTRODUCTION

Screening and surveillance colonoscopy has been found to help reduce incidence and mortality from colorectal cancer (CRC)[1-4]. Given concern for interval CRC as well as polyp miss rate, different quality metrics – such as minimum withdrawal time and adenoma detection rate – have been created to try to improve colonoscopy quality[5-7]. In addition, different tools such as Endocuff and EndoRings have been developed to try to enhance polyp detection[8-11].

With the expansion of artificial intelligence research, there has been significant interest in applying computer aided detection (CADe) devices to improve polyp detection[12]. The benefit of CADe has been demonstrated across several studies, including by Repici et al[13]. In September 2020, we performed a randomized clinical trial with Food and Drug Administration (FDA) guidance to evaluate the utility of a CADe. The study initially involved five sites, but partway through the study, due to the high baseline adenoma detection rate (ADR) of endoscopists (> 40%) at the Veterans Affairs Palo Alto Health Care System, the FDA mandated that this site not be included as part of the company’s pivotal FDA study. As a result, the original study was split into two, into a community-based study involving four sites (AI-SEE)[14], and a separate study involving a Veterans hospital. Here we present the results from the Veterans Affairs Palo Alto Health Care System site.

MATERIALS AND METHODS

Study design

The present study is a single-center, prospective, randomized clinical trial, evaluating the utility of CADe in colonoscopy among veterans. The study took place at Veterans Affairs Palo Alto Health Care System, based in Palo Alto, California. The study was performed in accordance with the Declaration of Helsinki. The study was approved by the institutional review board at Stanford University and was registered at ClinicalTrials.gov (NCT04555135).

Inclusion and exclusion criteria

Adults (age ≥ 45) presenting for colonoscopy were enrolled. Indications for colonoscopy included screening or low-risk surveillance. Low-risk surveillance was defined as patients whose findings on previous colonoscopy recommended repeat colonoscopy ≥ 3 years per United States Multi-Society Task Force guidelines[15]. In addition, other inclusion criteria were: Presence of informed consent, adequate bowel preparation [defined by score ≥ 2 in all colonic segments per Boston Bowel Preparation Scoring System (BBPS)]. Patients presenting for high-risk screening or surveillance, such as history of inflammatory bowel disease, known or suspected polyposis or hereditary colon cancer syndrome, prior colon resection (not including appendectomy) were excluded. In addition, patients who presented for diagnostic colonoscopy (such as positive fecal immunochemical test or diarrhea) as well patients with incomplete colonoscopy were excluded (Supplementary Figure 1).

Five patients that were enrolled in the study and randomized were later identified to fit exclusion criteria and removed from the study analysis. Four of the patients received colonoscopy for high-risk surveillance and one for iron deficiency anemia. All instances had occurred in the first two months of enrollment. Three of the patients were enrolled in the CADe arm, and two in the conventional colonoscopy arm. These patients were excluded in the analysis. There were no other patients that were excluded (including for inadequate bowel preparation or inability to reach cecum) following inclusion and randomization in the study.

Device evaluated

Our trial evaluated the use of EndoVigilant (Millersville, MD). EndoVigilant serves in real-time to highlight colon polyps by displaying a blue box around the polyp (Figure 1). EndoVigilant had been measured to have frame-level sensitivity of 0.9 and frame-level specificity of 0.97, utilizing data not used in training or validation of the model[14]. The device accesses the colonoscopy video feed and provides a display in a separate monitor (dual monitor setup). The endoscopist had the option of either looking at both the primary monitor as well as the CADe monitor simultaneously or just the CADe monitor alone. Further, for patients who were randomized to CADe, the endoscopist had the option of turning on CADe either throughout the procedure or only upon withdrawal. For the purposes of the trial, no modifications were made to the device.

Figure 1 Use of computer aided detection to identify polyp, outlined with a blue box.

The study was sponsored by EndoVigilant, Inc (Millersville, MD). Patients did not receive compensation for the study.

Randomization

After a patient was consented for the study, at the beginning of each procedure, our research coordinator (YC) opened an opaque envelope which assigned the colonoscopy to be performed with or without CADe. Patients were block randomized by endoscopist in block sizes of 4, 6, and 8, using a computer-generated sequence[16]. By providing information only at the start of the case, this helped optimize masking of allocation to the investigator as best as possible. Patients and pathologists were also masked to this information.

Colonoscopy performance

All colonoscopies were performed by or under supervision of full-time staff endoscopists (SF, SYQ, JYP, RW) of the VA Palo Alto Health Care System. Each staff member had been in practice for at least 5 years after fellowship. Baseline ADR for all four endoscopists were > 40%. In cases in which a trainee fellow was involved, the attending was present throughout the procedure. Polyps identified were removed at the discretion of the endoscopist.

Data collection

Our research coordinator (YC) collected patient data throughout the study and during each colonoscopy. Data collected included patient demographics (age, sex, height, weight, body mass index, race/ethnicity), procedure indication (screening or low-risk surveillance), sedation (none, conscious, or monitored anesthesia care), as well as colon preparation. All polyps removed were recorded (location, size, and pathology). Patients were monitored for adverse events (up to 30 d).

Outcome measures

Primary outcomes of interest were adenomas per colonoscopy, ADR (all, screening, surveillance colonoscopies) as well as adenomas per extraction. Adenomas per colonoscopy (APC) was defined as the total number of adenomas removed divided by number of colonoscopies. Adenomas per extraction was defined as total number of adenomas removed divided by total number of polyps removed.

Secondary outcomes of interest included serrated polyps per colonoscopy, non-adenomatous, non-serrated polyps per colonoscopy, and serrated polyp detection rate (all, screening, surveillance colonoscopies). Procedural characteristics measured included total length of procedure as well as withdrawal time. Adenoma characteristics measured included polyp distribution (proximal and distal colon), size distribution (< 6 mm, 6-9 mm, ≥ 10 mm), as well as Paris classification distribution (Ip/Is, IIa/b/c, III). Proximal colon was defined as cecum to transverse colon and distal colon was defined as descending colon to rectum.

Statistical analysis

In all analyses, P < 0.05 was considered significant. Student’s t-test was utilized to compare the average of normally distributed continuous variables. χ² test was utilized to compare the proportions of categorical outcomes. All tests were two-tailed. Sample size calculation is described elsewhere[14], but in summary using prior pilot study data[17], we had calculated that 866 patients per group would be needed, with the Veterans Affairs Palo Alto Health Care System contributing approximately 20% of patients (as one of five sites).

Early study termination

The study was terminated early after the interim analysis of APC at the four community-based sites of 769 subjects yielded a new sample size estimate requiring 6557 per group[14]. As such, the company ceased support of the study and we therefore elected to terminate the study in December 2021.

RESULTS

Patient demographics

A total of 244 patients were enrolled – 124 with CADe and 120 without CADe. There was similar mean age, distribution of male and female sex, indication of procedure, as well as race/ethnicity between with and without CADe groups. There was similar mean BBPS between the two groups (7.9 vs 7.8, P = 0.498) (Table 1).

Table 1 Patient characteristics, N (%).

	With CADe (N = 124)	Without CADe (N = 120)	P value
Mean age (± SD)	68.7 (7.5)	67.8 (8.0)	0.356
Male sex	118 (95.2)	117 (97.5)	0.333
Mean BMI (± SD)	30.2 (4.7)	30.7 (5.2)a	0.368
Indication
Screening	13 (10.5)	14 (11.7)	0.768
Surveillance	111 (89.5)	106 (88.3)
Race/ethnicity			0.015
Caucasian	69 (55.6)	85 (70.8)
Asian	13 (10.5)	3 (2.5)
African American	18 (14.5)	12 (10.0)
Hispanic	20 (16.1)	12 (10.0)
Other	4 (3.2)	8 (6.7)
Mean BBPS (± SD)	7.9 (1.1)	7.8 (1.1)	0.498

^aData missing for 1 patient.

CADe: Computer aided detection; SD: Standard deviation; BMI: Body mass index; BBPS: Boston Bowel Preparation Score.

Polyp detection

There was no significant difference in number of polyps per colonoscopy detected between colonoscopies with CADe compared to without CADe (2.51 vs 3.47, P = 2.976) (Table 2). However, compared to with CADe, colonoscopies without CADe showed a higher number of adenomas per colonoscopy (2.53 vs 1.79, P = 0.030), but there was no significant difference with respect to number of serrated polyps (0.13 vs 0.04, P = 0.091) or number of non-adenomatous, non-serrated polyps (0.81 vs 0.68, P = 0.426). In evaluation of adenomas, while there was no significant difference between without CADe and CADe in the distal colon, there were more adenomas detected without CADe in proximal colon (2.35 vs 1.53, P = 0.009). There were a higher number of < 6 mm (2.82 vs 2.06, P = 0.029), as well as Paris Classification Is (3.14 vs 2.21, P = 0.020) adenomas, detected without CADe compared to with CADe, but no significant difference for polyps 6-9 mm, ≥ 10 mm, or Paris classification IIa. Importantly, there was no significant difference in average withdrawal time or overall procedure time.

Table 2 Procedural characteristics.

	With CADe (N = 124)	Without CADe (N = 120)	P value
Number of Polyps Per Colonoscopy, average (± SD)	2.51 (2.35)	3.47 (3.71)	2.976
Number of Adenomas Per Colonoscopy, average (± SD)	1.79 (1.85)	2.53 (3.25)	0.030
Number of Serrated Polyps per Colonoscopy, average (± SD)	0.04 (0.24)	0.13 (0.49)	0.091
Number of Non-adenomatous, non-serrated polyps per colonoscopy, average (± SD)	0.68 (1.06)	0.81 (1.46)	0.426
Adenomatous Polyp Location Distribution Per Colonoscopy, average (± SD)
Proximal Colon, average (± SD)	1.53 (1.72)	2.35 (2.92)	0.009
Distal Colon, average (± SD)	0.82 (1.16)	0.86 (1.33)	0.824
Adenomatous Polyp Size Distribution Per Colonoscopy, average (± SD)
< 6 mm	2.06 (2.03)	2.82 (3.21)	0.029
6-9 mm	0.35 (0.69)	0.53 (0.84)	0.071
≥ 10 mm	0.09 (0.31)	0.13 (0.42)	0.447
Adenomatous Polyp Distribution by Paris Classification Per Colonoscopy, average (± SD)
Is	2.21 (2.25)	3.14 (3.74)	0.020
IIa	0.12 (0.52)	0.18 (0.51)	0.414
Average withdrawal time, mins (± SD)	22.0 (14.8)a	22.5 (11.6)b	0.755
Average procedure time, mins (± SD)	30.7 (16.8)c	32.2 (14.9)a	0.493

^aData missing for 2 patients.

^bData missing for 3 patients.

^cData missing for 1 patients.

CADe: Computer aided detection; SD: Standard deviation.

Detection rate

Compared to with CADe, colonoscopies without CADe were found to have a higher adenoma detection rate across all colonoscopies (80.0% vs 68.5%, P = 0.041) (Table 3). This finding was driven by surveillance colonoscopies (84.9 vs 73.0, P = 0.032), as there was no significant difference when evaluating screening colonoscopies (42.9 vs 30.8, P = 0.516). There was also no significant difference in serrated polyp detection rate or in adenomas per extraction.

Table 3 Detection rate of colonoscopies, n/N (%).

	With CADe (N = 124)	Without CADe (N = 120)	P value
Adenoma Detection Rate in All Colonoscopies	85/124 (68.5)	96/120 (80.0)	0.041
Adenoma Detection Rate in Screening Colonoscopies	4/13 (30.8)	6/14 (42.9)	0.516
Adenoma Detection Rate in Surveillance Colonoscopies	81/111 (73.0)	90/106 (84.9)	0.032
Serrated Polyp Detection Rate in All Colonoscopies	4/124 (3.2)	9/120 (7.5)	0.137
Serrated Polyp Detection Rate in Screening Colonoscopies	0/13 (0.0)	1/14 (7.1)	0.326
Serrated Polyp Detection Rate in Surveillance Colonoscopies	4/111 (3.6)	8/106 (7.5)	0.204
Adenomas Per Extraction in All Colonoscopies	222/311 (71.4)	304/416 (73.1)	0.613
Adenomas Per Extraction in Screening Colonoscopies	6/8 (75.0)	18/25 (72.0)	0.868
Adenomas Per Extraction in Surveillance Colonoscopies	216/303 (71.3)	286/391 (73.1)	0.587

CADe: Computer aided detection.

Physician specific adenoma detection rate

In evaluation of physician specific adenoma detection rate (Table 4), one physician (#1) was found to have significantly increased ADR without CADe (74.3% vs 52.8%, P < 0.0001). While there was a trend towards increased ADR without CADe for physician #3 (88.5% vs 66.7%) this was not found to be statistically significant (P = 0.058). In only one physician was there an increased ADR with use of CADe (#4) [80.0% vs 73.7%], but this was not statistically significant (P = 0.640).

Table 4 Physician specific adenoma detection rate, with or without use of computer aided detection.

Physician (#)	ADR with CADe (%)	ADR without CADe (%)	P value
1	19/36 (52.8)	26/35 (74.3)	< 0.0001
2	32/41 (78.0)	33/40 (82.5)	0.615
3	18/27 (66.7)	23/26 (88.5)	0.058
4	16/20 (80.0)	14/19 (73.7)	0.640

ADR: Adenoma detection rate; CADe: Computer aided detection.

DISCUSSION

There has been great interest in utilizing artificial intelligence in polyp detection[12]. Several meta-analyses, including by Hassan et al[18] and Huang et al[19] have demonstrated significant increase in adenoma and polyp detection rate with use of CADe compared to without use of CADe. However, several real-world studies[17,20,21] have not demonstrated a benefit with use of CADe. As a result, we had performed AI-SEE[14], a multi-center randomized clinical trial involving community-based, non-academic centers, and found no significant difference in adenoma detection with use of CADe. AI-SEE originally had included the Veterans Affairs Palo Alto Health Care System but during the trial, with recommendation from the FDA, the site was not included in subsequent analyses. Therefore, we chose to share the results of the Veterans Affairs Palo Alto Health Care System site here.

In this randomized controlled trial, we found that CADe led to a lower number of adenomas per colonoscopy compared to without CADe. There was no significant difference in detection of serrated polyps or non-adenomatous non-serrated polyps per colonoscopy between the two groups. Compared to without CADe, use of CADe was also found to have lower ADR. While there was lower serrated polyp detection rate (3.2% vs 7.5%) with use of CADe, this was not statistically significant (P = 0.137). While several other studies have demonstrated no significant difference between the two groups, our study surprisingly identified a lower adenoma detection rate with use of CADe compared to without use of CADe across all colonoscopies.

The reasons behind our findings of lack of benefit and lower adenoma detection with use of CADe are not clear. Overall, we do not think this was an issue with the performance of the CADe used. As described in AI-SEE[14], the CADe software used had undergone rigorous external testing and had demonstrated its ability to detect polyps, with comparable to other available systems[13]. As such, we do not believe our findings would be significant different with another CADe system. Further, in our analysis of the four different endoscopists, only one endoscopist was found to have statistically significant lower ADR with CADe compared to without CADe. On sub-analysis excluding this provider (#1), ADR remained lower with use of CADe (75.0% vs 82.4%), though this was not statistically significant (0.238).

The relatively small size of the study (120 patients per arm) may decrease the ability to detect any benefit CADe may have over without CADe. However, given the difficulty of conducting a larger trial with the sample size necessary, this was not pursued. Due to the small size of the study, despite patient randomization, relative to without CADe, the CADe group had lower percentage of Caucasians as well as higher percentage of African Americans. While the variation of patient race/ethnicity was unlikely to have affected polyp detection, black patients have been found to have higher prevalence of polyps > 9 mm compared to whites[22]. However, in this study, the CADe group had higher percentage of black patients without demonstrating benefit of CADe.

An additional limitation was the inability to blind the endoscopist. While patients were randomized, we were unable to blind an endoscopist from the presence of CADe, which was clear to the endoscopist due to presence of the polyp framing box. Due to the nature of the study, our endoscopists could not be blinded (seeing the polyp framing box would clue any endoscopist regarding the presence of CADe), and as such there may be a Hawthorne effect, whether negative or positive. However, given the similar procedure and withdrawal time between the CADe compared to without CADe groups, we believe our endoscopist did not significantly change their colonoscopy approach depending on their allocation.

However, one important consideration is that our endoscopists have a high baseline ADR (range 73.7%-88.5%), which is in part due to the patient population (mean age 68, 95% male), but also underscores the proficiency of our endoscopists. In contrast, in the meta-analysis by Hassan et al[18], baseline pooled ADR was 25.2%, which is significantly lower in comparison. With the high baseline ADR of our endoscopists, it would also be difficult for CADe to demonstrate an improvement in ADR.

The findings of our study help shape the discussion regarding when CADe can be useful for endoscopists. While CADe has been demonstrated to be useful to endoscopists in several RCTs, the benefit may be less clear for endoscopists that have a high baseline demonstrated ADR and adenomas per colonoscopy. With the advent of CADe, ADR alone may not be sufficient criteria for assessing quality of colonoscopy and polyp detection, but it may be worth considering including adenomas per colonoscopy as an additional metric for identifying endoscopists who may benefit from using CADe to help with their colonoscopy performance. Further, this study questions whether CADe would be useful for all endoscopists – our study findings suggest that for high performing endoscopists, CADe does not improve and may potentially lead to decreased ADR and APC. However, CADe may remain useful for endoscopists with lower ADR and APC. Further studies will be needed to determine the utility and criteria of application off CADe among endoscopists.

CONCLUSION

While several randomized controlled trials have demonstrated a benefit with CADe, several real-world studies as well as our recent AI-SEE RCT have not demonstrated significant improvement of CADe compared to standard colonoscopy in the detection of adenomas. This study, evaluating the impact of CADe in a Veterans hospital in California, demonstrated a lower number of adenomas per colonoscopy as well as adenoma detection rate, which is likely attributed to differences in patient population, as well as high baseline ADR of our endoscopists, thereby decreasing any potential benefit CADe may have. Given these findings, we feel that CADe may have limited benefit for endoscopists with high ADR and adenomas per colonoscopy. Further studies are needed to evaluate criteria for endoscopists to consider when choosing to adopt CADe in their practices.

ARTICLE HIGHLIGHTS

Research perspectives

The results of this study suggest that computer aided detection (CADe) may not be the right tool for every endoscopist or center. Further studies are needed to understand the impact of CADe on performance quality among endoscopists as well as determine criteria for endoscopists to consider adoption of CADe in their practices.

Research conclusions

In this randomised controlled trial performed at a center with high adenoma detection rate (ADR), use of CADe was found to have decreased adenomas per colonoscopy (APC) and ADR.

Research results

A total of 244 patients were enrolled (124 with CADe). Use of CADe was found to have decreased number of adenomas (1.79 vs 2.53, P = 0.030) per colonoscopy compared to without CADe. Further, use of CADe was found to have lower ADR (68.5% vs 80.0%, P = 0.041) compared to without use of CADe.

Research methods

Adults aged 45 or older presenting for screening or low-risk surveillance were randomized to colonoscopy performed with or without CADe. Primary outcomes included ADR, APC, and adenomas per extraction.

Research objectives

The study study is a single-center, prospective, randomized clinical trial, evaluating the utility of CADe in colonoscopy among veterans.

Research motivation

In September 2020, we performed a randomized clinical trial with Food and Drug Administration (FDA) guidance to evaluate the utility of a CADe. This study initially involved five sites, but partway through the study, due to the high baseline ADR of endoscopists (> 40%) at the Veterans Affairs Palo Alto Health Care System, the FDA mandated that this site not be included as part of the company’s pivotal FDA study. As a result, the original study was split into two, into a community-based study involving four sites, and a separate study involving a Veterans hospital. We present the results from the Veterans Affairs Palo Alto Health Care System site.

Research background

In an effort to reduce risk for polyp miss rate, multiple quality metrics (such as minimal withdrawal time and ADR have been developed to improve colonoscopy quality. The benefit of CADe has been demonstrated across several studies, including by Repici et al.