Kader R, Hadjinicolaou AV, Georgiades F, Stoyanov D, Lovat LB. Optical diagnosis of colorectal polyps using convolutional neural networks. World J Gastroenterol 2021; 27(35): 5908-5918 [DOI: 10.3748/wjg.v27.i35.5908]
Corresponding Author of This Article
Rawen Kader, BMed, MBBS, MRCP, Research Fellow, Wellcome/ EPSRC Centre for Interventional and Surgical Sciences, University College London, Charles Bell House, 43-45 Foley Street, Fitzrovia, London W1W 7TY, United Kingdom. email@example.com
Checklist of Responsibilities for the Scientific Editor of This Article
This article is an open-access article which was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Author contributions: Kader R, Hadjinicolaou AV and Georgiades F performed the literature review and wrote the manuscript; Stoyanov D and Lovat LB revised the manuscript; All authors have read and approved the final manuscript.
Conflict-of-interest statement: Rawen Kader is supported by the Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) at UCL; [203145Z/16/Z]. Danail Stoyanov owns shares in Odin Vision and Digital Surgery Ltd. Laurence B Lovat owns shares in Odin Vision. The remaining authors declare no conflict of interest.
Open-Access: This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/Licenses/by-nc/4.0/
Corresponding author: Rawen Kader, BMed, MBBS, MRCP, Research Fellow, Wellcome/ EPSRC Centre for Interventional and Surgical Sciences, University College London, Charles Bell House, 43-45 Foley Street, Fitzrovia, London W1W 7TY, United Kingdom. firstname.lastname@example.org
Received: February 27, 2021 Peer-review started: February 27, 2021 First decision: April 18, 2021 Revised: April 29, 2021 Accepted: August 24, 2021 Article in press: August 24, 2021 Published online: September 21, 2021
Colonoscopy remains the gold standard investigation for colorectal cancer screening as it offers the opportunity to both detect and resect pre-malignant and neoplastic polyps. Although technologies for image-enhanced endoscopy are widely available, optical diagnosis has not been incorporated into routine clinical practice, mainly due to significant inter-operator variability. In recent years, there has been a growing number of studies demonstrating the potential of convolutional neural networks (CNN) to enhance optical diagnosis of polyps. Data suggest that the use of CNNs might mitigate the inter-operator variability amongst endoscopists, potentially enabling a “resect and discard“ or ”leave in“ strategy to be adopted in real-time. This would have significant financial benefits for healthcare systems, avoid unnecessary polypectomies of non-neoplastic polyps and improve the efficiency of colonoscopy. Here, we review advances in CNN for the optical diagnosis of colorectal polyps, current limitations and future directions.
Core Tip: A convolutional neural network (CNN) is a specific type of artificial intelligence deep learning. These networks may play an important role in the coming years in assisting endoscopists to optically diagnose colorectal polyps. CNNs can mitigate the inter-operator variability amongst endoscopists, potentially enabling a “resect and discard” or “leave in” strategy to be adopted. This would improve the efficiency of colonoscopy, reduce healthcare costs and reduce adverse events for patients by avoiding unnecessary resections of non-neoplastic polyps. In this article, we expand on the most relevant studies in this field and discuss limitations and future directions that will determine fulfilment of the potential of CNN in the optical diagnosis of colorectal polyps.
Citation: Kader R, Hadjinicolaou AV, Georgiades F, Stoyanov D, Lovat LB. Optical diagnosis of colorectal polyps using convolutional neural networks. World J Gastroenterol 2021; 27(35): 5908-5918
Colorectal cancer (CRC) is the third most commonly diagnosed cancer worldwide and thus, a significant burden on global healthcare systems. Most CRCs develop in a relatively predictable, stepwise sequence from mutation-accumulating neoplastic polyps, such as adenomas and sessile serrated lesions (SSL). Current evidence-based societal guidelines unequivocally accept colonoscopy to be the gold standard tool for screening of CRC. Colonoscopy offers the opportunity to both detect and resect neoplastic polyps and its implementation, especially as part of bowel cancer screening programs, has been linked to a significant reduction in the incidence of the CRC and CRC-related mortality.
Over 90% of polyps detected at colonoscopy are either small (6-9 mm) or diminutive (≤ 5 mm), entities that are thought to harbour a very low risk for developing into CRC. Furthermore, almost half of these polyps are non-neoplastic in nature; and frequently hyperplastic. Accurate differentiation of neoplastic from non-neoplastic polyps can prevent the unnecessary resection of the latter, avoiding an intervention which is not cost-effective and which carries risks of significant morbidity.
Recent years have seen significant research activity in the use of artificial intelligence (AI), particularly convolutional neural networks (CNN), to optically diagnose colorectal polyps. The field is gaining increasing momentum. The aim of this review article is to summarise and critically appraise the available medical literature related to advances in CNN for optical diagnosis of colorectal polyps and highlight the field’s current limitations and future directions.
The term “optical diagnosis” refers to the use of advanced imaging techniques for real-time, in-vivo polyp characterisation and evaluation to guide therapeutic decisions. Accurate optical diagnosis of diminutive polyps would enable identification of hyperplastic polyps in the rectosigmoid region, where they are commonly found, and allow the endoscopist to confidently take a “diagnose and leave” approach instead of resecting the lesion. Equally, for diminutive adenomas, accurate optical diagnosis would prompt the endoscopist to remove the lesion on the spot and discard the specimen without the need for histological assessment (“resect and discard”strategy).
The American Society of Gastrointestinal Endoscopy established the Preservation and Incorporation of Valuable endoscopic Innovations (PIVI) to provide thresholds that are required of endoscopic technology in order to implement a “resect and discard”(PIVI 1) and “diagnose and leave” (PIVI 2) strategy. PIVI 1 requires ≥ 90% concordance in post-polypectomy surveillance intervals when comparing the combination of optical diagnosis for diminutive adenomas with histopathology assessment of all other polyps against decisions based solely on histopathology evaluation of all identified polyps. PIVI 2 requires a technology to achieve a negative predictive value (NPV) of ≥ 90% for diminutive adenomatous polyps in the rectosigmoid region.
There has been extensive research in image enhanced endoscopy (IEE), such as narrow band imaging (NBI), to assist endoscopists in optical diagnosis to characterise diminutive polyps[11-13]. Using IEE, expert endoscopists in academic centres have consistently demonstrated an optical diagnosis accuracy that exceeds PIVI thresholds[14-16], however, studies have often found community and non-expert endoscopists to fall short of these minimal thresholds. An example is the multi-centre DISCARD-2 study which evaluated the optical diagnosis accuracy of 28 community endoscopists using NBI. Disappointingly, the endoscopists’ optical diagnosis derived colonoscopy surveillance intervals only matched 68% of the histopathology derived intervals. Although widely available, technologies for optical diagnosis has not been incorporated into routine clinical practice with one of the main barriers being the inter-operator variability amongst endoscopists.
WHAT IS A CONVOLUTIONAL NEURAL NETWORK?
AI is the ability of computers to perform tasks that traditionally require human intelligence (Figure 1). Machine learning (ML) is a subset of AI, whereby computers continuously learn from data without explicit human programming. This can be used to predicate a polyp’s histology. ML models can be trained using unsupervised or supervised techniques. Unsupervised learning is when the input and output data are not paired. Supervised ML is more labour intensive as it requires paired input and output data for training. An example of a supervised ML model for optical diagnosis is to annotate a bounding box around a polyp (input data), commonly referred to as a region of interest, and label it with the histology of the polyp (output data). The model automatically learns to extract features that allow it to differentiate polyp subtypes and output a diagnosis based on the histology classification system it was trained with but the annotation process is time consuming for the clinician.
Figure 1 The relationship between convolutional neural networks, deep learning, machine learning and artificial intelligence.
Deep learning is a subset of ML, whereby algorithms use multiple layers within a neural network, mimicking the human brain, to extract high level features from input data. CNNs are the most commonly used network in the application of deep learning to optically diagnose polyps. They provide an objective output, bypassing the human inter and intra-operator variability, and can develop classification algorithms without exhaustive effort as they do not require human-crafted feature extraction or extensive pre-processing of data.
Building a CNN model typically involves three separate datasets; a training set, a validation set and a test set. The training set is used to develop the model so that it predicts a label (e.g., adenomatous or hyperplastic polyp for polyp characterisation) based on features extracted from the endoscopic image by the algorithm itself. The validation set is used to avoid over-fitting into the training dataset through fine tuning of the hyperparameters of the model. Finally, the testing set is used as an independent dataset to evaluate the generalisability of the CNN. With smaller datasets, cross-validation can be used to assess the model’s robustness. In cross-validation, the data is split into equal parts (e.g., 4 parts), with one part held out as a validation dataset. This process is repeated multiple times, with the results of each split eventually pooled together to decide how robust the model is. CNNs evaluated using cross-validation should still be assessed against an independent test set to examine their generalisability.
CONVOLUTIONAL NEURAL NETWORKS AND OPTICAL DIAGNOSIS
It is only in the last few years that the use of CNNs in optical diagnosis of colorectal polyps has been extensively investigated, with various studies emerging (Table 1). Many of these studies have in fact demonstrated the capability of CNNs to surpass the PIVI 2 threshold in order to support a “leave in” strategy for rectosigmoid hyperplastic polyps (Table 2). This was first demonstrated by Chen et al, who used a single centre, retrospective, still image dataset of 2157 polyps to train a CNN and reported a sensitivity for identifying adenomas of 96.3% , specificity 78.1%, and NPV of 91.5% when evaluating a test set of 284 colonic and rectal diminutive adenomatous and hyperplastic polyps. Using colonic diminutive polyps is a common strategy to assess against PIVI 2 due to difficulties in obtaining large datasets of diminutive rectosigmoid polyps. An important limitation of this study is that it used magnified narrow-band imaging (NBI) data. This recently developed modality is not yet readily available in most endoscopy departments, although it will become more widely used with time.
Table 1 Summary of the studies on convolutional neural network algorithms for the optical diagnosis of colorectal polyps.
WLI: White light imaging; BLI: Blue light imaging; NBI: Narrow band imaging; NBI-NF: Narrow band imaging–near focus; PIVI: Preservation and Incorporation of Valuable endoscopic Innovations; PPV: Positive predictor value; NPV: Negative predictor value; LC: Low-confidence.
Byrne et al further advanced the field by training a CNN with NBI-near focus (NBI-NF) which is more commonly used in Europe and North America. It was trained with 220 polyp positive videos and when tested against 125 diminutive polyps which were collected prospectively, the model diagnosed 106 polyps with high confidence, achieving a sensitivity for identifying NBI International Colorectal Endoscopic (NICE) type 1 polyps of 98%, specificity 83% and NPV of 97%. A novelty worth highlighting in this study was the use of images derived from videos, an approach that reduces selection bias compared to retrospective still images as endoscopists usually capture high quality polyp views that are free from motion blur and surface artifact. An additional advantage of this CNN is that it simplified the clinical workflow as it automatically diagnoses polyps without requiring a still image of the polyp to be captured. Limitations of the study are that SSLs, normal tissue and lymphoid aggregates were excluded from the final analysis and the videos used to train and test the CNN were captured from colonoscopies performed by a single expert endoscopist and hence, potentially less generalisable to novice users.
The most commonly used imaging modalities amongst community endoscopists are white light imaging (WLI) and NBI without magnification. Using a large retrospective still image training set of 5278 polyps and tested against 634 polyps, Zachariah et al’s CNN fell short of PIVI 2 in WLI (NPV of 88.9% and accuracy 92.8%) but achieved the threshold in NBI without magnification (NPV of 90.8% and accuracy 93.1%). This study advanced the field as it demonstrated the capabilities of CNNs to optically diagnose polyps in standard NBI modality and also to differentiate adenomas from serrated polyps through the inclusion of SSLs in its dataset.
Whilst the majority of CNNs have been trained and tested using Olympus data, studies are emerging using data from other manufacturers. van der Zander et al recently developed a CNN using Fujifilm data in high definition white light (HDWL) and blue light imaging (BLI). The CNN was more efficacious when it used a unique multimodal imaging approach where it combined both HDWL and BLI images of the same polyp in its decision process compared to a single imaging modality. When evaluated against 60 prospectively collected diminutive polyps, it did not reach the PIVI 2 threshold with a NPV of 87.5% but did achieve an optical diagnosis accuracy of 95% (sensitivity for identifying pre-malignant polyps 95.6% and specificity 93.3%) and demonstrated superiority to both expert and novice endoscopists in human benchmark testing.
In comparison to PIVI 2, there are fewer studies evaluating the performance of CNNs against PIVI 1. The CNN presented in Zachariah et al reached PIVI 1 thresholds in both WLI and NBI with normal magnification, achieving concordance with histology-based colonoscopy surveillance intervals in 90.9% and 98.3% of patients, for each respective modality. Rodrigues-Diaz et al used a single centre retrospective still image dataset to train a CNN with 607 polyps and tested against 90 diminutive polyps where it achieved a high confidence diagnosis in 78% of cases, with a 94% agreement with histology-based colonoscopy surveillance intervals. Tested against 68 rectosigmoid polyps, the model diagnosed 88% of polyps with high confidence, achieving PIVI 2 thresholds with a NPV of 97%.
There is also potential to expand the use of optical diagnosis CNNs outside of the ”resect and discard” and “leave in strategy”. A dilemma that can complicate issuing post-polypectomy surveillance intervals is discrepancies between endoscopic and histological diagnosis and classification of polyps with tissue fragmentation in the specimen retrieval process playing an important role. Shahidi et al’s proof of concept study used a CNN to resolve discrepancies in polyps ≤ 3 mm in size. Tested against 900 polyps that were ≤ 3 mm and optically diagnosed as adenomatous by an expert endoscopist, the CNN diagnosed the adenomas with high confidence in 644 polyps, with 256 polyps deemed to be of sub-optimal imaging quality. However, of these high confidence diagnoses, the pathologists diagnosed 15.4% as normal mucosa, 13.2% as hyperplastic polyp and 0.3% as SSL. In this context, a CNN could help to mitigate against the risk of under-surveillance.
Whilst CNN’s diagnostic accuracy excels in many studies, without real-time capabilities, they would have no clinical utility. Prior to the era of deep learning, computer aided diagnosis algorithms lacked real-time capability, but most CNNs do not share this problem and often process data at a rate that exceeds the 25 frames per second that is generated in a video recording of a colonoscopy procedure. Given the excellent performance in ex-vivo studies and the real-time capabilities displayed by CNNs, the future appears promising for their integration in colonoscopy.
TRANSPARENCY OF CONVOLUTIONAL NEURAL NETWORKS
The complexity of CNN models’ decision process is often referred to as a “black box” and represents an important barrier to its acceptance by both clinicians and patients. Opening the ‘’black box’’ to display the raw features which informed the CNN’s decision is important for transparency especially from a safety standpoint. Transparency can help identify biases within the neural network and aid root-cause analyses in cases of patient harm, for example, if a neoplastic polyp that subsequently develops into a CRC is originally misdiagnosed as non-neoplastic by the CNN model.
For polyp characterisation, important steps have been taken to open the black box. Jin et al developed a CNN that generated a coloured heat map, overlaid to the polyp, to help the endoscopist comprehend the specific aspects of the image that contributed to the CNN’s prediction (Figure 2). This could help the endoscopist to decide which information is relevant and which decisions are truly based on appropriate image analysis. If, for example, the heatmap is overlaid to normal mucosa, then the endoscopist would quickly be able to appreciate this and disregard the CNN’s diagnosis.
More recently, in order to further enhance CNN transparency, Rodriguez-Diaz et al developed a colour coded segmentation model (Figure 3). In this model, the CNN divides the polyp into distinct segments to allow the endoscopist to identify the specific regions within the image that is informing the CNN’s decision. The CNN predicts the histology of each subregion of the segmented polyp, with high confidence neoplastic diagnoses coloured in red, high confidence non-neoplastic in green, and low confidence/indeterminate diagnoses in yellow, with the final predication resulting from an aggregate of all the analysed regions. The end result is a detailed spatial colour coded histology map of the polyp surface, which the endoscopist can visualise and incorporate into their decision process, enhancing the interpretability of this CNN model in comparison to others. However, an important limitation to this advanced CNN is that it currently lacks the ability to operate at a video rate.
Further research in the interpretability of CNN models is required to improve its acceptance and accelerate its translation to clinical practise.
LIMITATIONS AND FUTURE DIRECTIONS
Despite the promise shown by CNNs this far, it is crucial to recognise that there are various limitations that need to be overcome before they can become part of the endoscopic clinical workflow. The most significant limitations are the reliance on retrospective datasets, which are inherently subject to selection bias, and the lack of prospective studies and randomised controlled trials. Most studies train and test CNNs using high quality images of polyps, free from “noise” such as motion blur and polyp surface artifact (e.g., mucus, stool or blood). The extent to which CNNs pre-clinical results are reproducible in the real-world setting, where ‘noise’ is frequently encountered, remains to be seen.
To the best of our knowledge, there have been no prospective randomised controlled clinical trials evaluating optical diagnosis CNN in-vivo. This is partly due to clinical trials being time consuming and expensive, and an alternative pragmatic approach could be the use of a benchmark test in the form a publicly available external dataset to compare different CNN models. No such datasets currently exist for polyp characterisation and therefore the generalisability of CNN models remains poorly understood. Generalisability refers to the CNN performance with different endoscope models and clinical settings from the site that the data was generated to train the CNN. To date, only one study has evaluated generalisability, and this was limited to a small testing set of 69 polyp images from two population cohorts (Australian and Japanese) using two separate endoscope manufactures (Olympus and Fujifilm). Despite the small test-set, this study highlighted the concerns of generalisability as the operator area under the curve fell from 94.3% for the internal set, to 84.5% and 90.3% for the external testing sets (NBI and BLI respectively).
Another important limitation is that studies often exclude polyps that are not adenomas or hyperplastic polyps, restricting the possible classification outputs of CNNs. This, in turn, limits their clinical utility as polyps such as SSL and inflammatory polyps would be misclassified due to limitations in the initial training phase of the CNNs when the categorisation system is established.
Research in this field is likely to continue to expand and future directions to consider include: (1) Guidelines to identify the role of CNNs in the clinical workflow, specifically, whether it is a second reader, a concurrent reader or a provider of an independent diagnosis; (2) Prospective multi-centre randomised clinical trials; (3) Publicly available external datasets for benchmark testing and evaluation of the generalisability of CNN models in different clinical settings and population cohorts; and (4) Acquiring datasets inclusive of all polyp sub-types to advance CNN classification systems.
In summary, this is an exciting time for the endoscopy community. CNNs diagnostic performance has excelled in ex-vivo studies and in human benchmarking testing. CNNs are likely to be a key adjunct in optically diagnosing polyps and have renewed optimism that implementation of a “resect and discard” and “leave in” strategy is feasible due to the potential to alleviate the inter-operator variability amongst endoscopists. This would bring significant financial benefits to healthcare systems, avoid unnecessary polypectomies of non-neoplastic polyps and improve the efficiency of colonoscopy. However, prospective multi-centre randomised controlled trials and publicly available datasets for benchmark testing are required to further evaluate the efficacy and generalisability of CNNs. Furthermore, with these models now emerging in endoscopy units, it’s imperative that guidelines are developed to establish their role in the clinical workflow.
US Preventive Services Task Force. Bibbins-Domingo K, Grossman DC, Curry SJ, Davidson KW, Epling JW Jr, García FAR, Gillman MW, Harper DM, Kemper AR, Krist AH, Kurth AE, Landefeld CS, Mangione CM, Owens DK, Phillips WR, Phipps MG, Pignone MP, Siu AL. Screening for Colorectal Cancer: US Preventive Services Task Force Recommendation Statement.JAMA. 2016;315:2564-2575.
[PubMed] [DOI][Cited in This Article: ][Cited by in Crossref: 975][Cited by in F6Publishing: 421][Article Influence: 195.0][Reference Citation Analysis (0)]
Rutter MD, East J, Rees CJ, Cripps N, Docherty J, Dolwani S, Kaye PV, Monahan KJ, Novelli MR, Plumb A, Saunders BP, Thomas-Gibson S, Tolan DJM, Whyte S, Bonnington S, Scope A, Wong R, Hibbert B, Marsh J, Moores B, Cross A, Sharp L. British Society of Gastroenterology/Association of Coloproctology of Great Britain and Ireland/Public Health England post-polypectomy and post-colorectal cancer resection surveillance guidelines.Gut. 2020;69:201-223.
[PubMed] [DOI][Cited in This Article: ][Cited by in Crossref: 62][Cited by in F6Publishing: 39][Article Influence: 31.0][Reference Citation Analysis (0)]
ASGE Technology Committee. Abu Dayyeh BK, Thosani N, Konda V, Wallace MB, Rex DK, Chauhan SS, Hwang JH, Komanduri S, Manfredi M, Maple JT, Murad FM, Siddiqui UD, Banerjee S. ASGE Technology Committee systematic review and meta-analysis assessing the ASGE PIVI thresholds for adopting real-time endoscopic assessment of the histology of diminutive colorectal polyps.Gastrointest Endosc. 2015;81:502.e1-502.e16.
[PubMed] [DOI][Cited in This Article: ][Cited by in Crossref: 162][Cited by in F6Publishing: 100][Article Influence: 27.0][Reference Citation Analysis (0)]
Rees CJ, Rajasekhar PT, Wilson A, Close H, Rutter MD, Saunders BP, East JE, Maier R, Moorghen M, Muhammad U, Hancock H, Jayaprakash A, MacDonald C, Ramadas A, Dhar A, Mason JM. Narrow band imaging optical diagnosis of small colorectal polyps in routine clinical practice: the Detect Inspect Characterise Resect and Discard 2 (DISCARD 2) study.Gut. 2017;66:887-895.
[PubMed] [DOI][Cited in This Article: ][Cited by in Crossref: 87][Cited by in F6Publishing: 54][Article Influence: 17.4][Reference Citation Analysis (0)]
van der Zander QEW, Schreuder RM, Fonollà R, Scheeve T, van der Sommen F, Winkens B, Aepli P, Hayee B, Pischel AB, Stefanovic M, Subramaniam S, Bhandari P, de With PHN, Masclee AAM, Schoon EJ. Optical diagnosis of colorectal polyp images using a newly developed computer-aided diagnosis system (CADx) compared with intuitive optical diagnosis.Endoscopy. 2020;.
[PubMed] [DOI][Cited in This Article: ][Cited by in Crossref: 3][Cited by in F6Publishing: 3][Article Influence: 3.0][Reference Citation Analysis (0)]
Ahmad OF, Mori Y, Misawa M, Kudo SE, Anderson JT, Bernal J, Berzin TM, Bisschops R, Byrne MF, Chen PJ, East JE, Eelbode T, Elson DS, Gurudu SR, Histace A, Karnes WE, Repici A, Singh R, Valdastri P, Wallace MB, Wang P, Stoyanov D, Lovat LB. Establishing key research questions for the implementation of artificial intelligence in colonoscopy: a modified Delphi method.Endoscopy. 2021;53:893-901.
[PubMed] [DOI][Cited in This Article: ][Cited by in Crossref: 4][Cited by in F6Publishing: 2][Article Influence: 4.0][Reference Citation Analysis (0)]