Artificial intelligence for the study of colorectal cancer tissue slides

Abstract

Artificial intelligence (AI) is gaining incredible momentum as a companion diagnostic in a number of fields in oncology. In the present mini-review, we summarize the main uses and findings of AI applied to the analysis of digital histopathological images of slides from colorectal cancer (CRC) patients. Machine learning tools have been developed to automatically and objectively recognize specific CRC subtypes, such as those with microsatellite instability and high lymphocyte infiltration that would optimally respond to specific therapies. Also, AI-based classification in distinct prognostic groups with no studies of the basic biological features of the tumor have been attempted in a methodological approach that we called “biology-agnostic”.

Key Words: Artificial intelligence, Colorectal cancer, Digital pathology, Deep learning, Machine learning, Tumor-infiltrating lymphocytes

Core Tip: Artificial intelligence (AI) is gaining incredible momentum as a companion diagnostic in a number of fields in oncology. In the present mini-review, we summarize the main uses and findings of AI applied to the analysis of digital histopathological images of slides from colorectal cancer patients.

INTRODUCTION

Artificial intelligence (AI) refers to any form of machine activity that attempts to mimic human intelligence. The main abilities of the human intelligence objective of AI research are performing complex tasks and achieving goals through processes that are typical of the human cognitive functions, such as “learning” [which in this case is termed “machine learning” (ML)] and “problem solving”[1-4].

“Deep” learning (DL) is a basic method of ML. It is based on specific computer algorithms that recall neural networking and that try to model data-rich input for the identification of general and meaningful features/patterns or for the prediction of specific outcomes[5]. The term “deep” refers to the organization in layers of the artificial neural network, with deep layers of analysis being crucial to infer more complex and higher-level characteristics up to the final layer of the label/output that is pursued (Figure 1 and Figure 2).

Figure 1 Exemplified representation of hand-crafted feature-based machine learning.

Figure 2 Exemplified explanation of deep learning for tumor tissue slide analysis.

ML has been successfully applied in the so-called digital pathology[6-8], where histological images are deeply analyzed in every single pixel for color and light intensity by computer algorithms, to deduce a final diagnosis or prognostic/predictive feature.

The use of AI in cancer histopathology has been possible since the advent of histologic images digitalization[6]. Widely available image scanners are currently used in clinical practice, also for the acquisition of initial hematoxylin and eosin (HE) stained slides. Mukhopadhyay et al[9] performed a comparison between digital pathology and classical microscopy in a large multi-center cohort of patients and confirmed the exceptional utility of the digital support. Early studies were based on the analysis and segmetation of specific slide regions, such as the tumor center, margins, stroma or others, mainly in order to recognize specific objects and structures[10]. More recently, with the ability of the computational analysis of higher digital dimension, automated analysis of whole HE-stained tumor tissue slides has been possible and inference of patient outcome and prognosis has been attempted[11,12]. In the present mini-review, we summarize the latest findings for the application of AI tools for the digital pathology of colorectal cancer (CRC). In general, two research fields have been identified: AI instruments for identification of specific biologic features and AI instruments for prediction of patient outcome independently of the cancer biology.

ML FOR DETECTION OF SPECIFIC BIOLOGIC FEATURES

One typical approach of AI in pathology is to use pre-defined key image features as building block for AI-algorithm development. These are usually termed “hand-crafted” features and are engineered based on biological insights. Typical examples of “hand-crafted” features are shape and orientation of the cell nuclei[13], that can be used to recognize cancer cells or tumor infiltrating lymphocytes or other cell types.

Assessment of shape and organization of cancer cells

Shape and orientation of cancer cells are among the most commonly assessed variables to predict patient outcome[14]. A predictive, ML-based algorithm for lymph node metastasis (LNM) on whole slide images in early CRC with pathological submucosal invasion (pT1) has proven useful when using data from cancer cell morphology[16]. After delineating cancer cell regions by using immunohistochemistry for cytokeratins, Takamatsu et al[15] analyzed tissue slides from over 300 CRC patients diagnosed as pT1, looking for predictors of LNM. They used the popular ImageJ software, released by the United States’ National Institutes of Health, and JPEG images. Digital parameters extracted from the slides essentially referred to shape, circularity, orientation and organization of cancer cells (e.g., Feret’s diameter), and these were used to feed a supervised ML algorithm based on random forest classifier. AI prediction was proved to be superior to human conventional assessment, with a discriminatory power of area under the curve (commonly known as AUC) 94% vs 83%, respectively.

Sailem et al[16] proposed a data-rich integrated platform that utilized a ML approach to conjugate high-throughput gene perturbation analysis with morphological features of CRC and, in particular, single cell morphology and cell population organization. After identification of TGFβ and WNT signaling genes and olfactory receptors genes as key altered genes in CRC, they validated their association with single cell and cell population morphology and grade of differentiation in CRC. Those specific gene alterations were associated to abnormal organization of HCT116 CRC cell cultures and also to specific CRC molecular subtypes (the well-known consensus molecular subtypes – CSM[17]) in over 550 patients from The Cancer Genome Atlas (referred to as TCGA) database.

Assessment of tumor infiltrating lymphocytes

In recent years, the development of immunotherapy has revolutionized the care of cancer[18], and great attention has been placed upon assessment of the immune response surrounding the tumor tissue[19]. Most reports are focused on specific immune cells and chiefly on tumor-induced cytotoxic T cells. Indeed, a high T-cell infiltration appears to be associated with a decreased risk of tumor dissemination and improved survival in most solid tumors. Väyrynen et al[20] have recently proposed an AI-based assessment of CRC slides, simultaneously evaluating different cell populations involved in immune response against the tumor. Scanned HE-stained images of CRC cases from two United States’ prospective cohorts [the Nurses’ Health Study (referred to as NHS), and the Health Professionals Follow-up Study (referred to as HPFS)] were analyzed with an open source software (QuPath v0.1.2), enabling the recognition of cancer epithelial cells and four immune cell types with distinct morphological features: lymphocytes, plasma cells, eosinophils, and neutrophils. Moreover, QuPath v0.1.2 was able to distinguish between two tissue regions: the intraepithelial region and stromal region[21]. Densities of immune cells in the intraepithelial and stromal regions, together with a function (G-cross function) measuring the proximity of immune cells to cancer cells (expressed as AUC within a distance of 20 μm) were calculated. Findings from NHS and HPFS cohorts were validated in a set of 570 CRC from TGCA with available digitalized pathological images in the data portal. Automated quantification of immune cells was compared to manual count by pathologists.

A high correlation between measurement of immune cell densities performed by automated AI classifiers and manual counts performed by experienced pathologist was found (Spearman’s rho 0.71-0.96). Moreover AI-determined stromal density of lymphocytes and eosinophils were independently correlated with survival in a multivariable Cox regression analysis, with higher densities associated with longer cancer-specific survival (P for trend across quartiles of density < 0.001 for both cell types). The multivariate analysis included very important molecular factors which were available for the post-hoc analysis of NHS and HPFS populations and partly associated with immune response, such as microsatellite instability (MSI), CpG island methylator phenotype, KRAS, BRAF, and PIK3CA mutations, and LINE-1 methylation level. High G cross function for Tumor: Lymphocyte and Tumor: Eosinophil proximity was also associated with longer cancer-specific survival (P_trend 0.002 and < 0.001, respectively). High stromal eosinophil density was associated with better cancer-specific survival also in the TGCA cohort (P_trend < 0.001).

AI can also improve CRC cancer staging and, consequently, patient prognosis. Reichling and co-workers[22], using a LASSO algorithm, DGMate (DiGital tuMor pArameTErs), combined the analysis of tumor stroma and tumor cell intrinsic variables, in association with immune cell infiltrate of tumor samples from stage III CRC patients of the PETACC8 study cohort, and were able to detect digital parameters within the tumor cells related to patients’ outcomes[23]. Similarly, clinically relevant information were gained in the study of Yoo et al[24], in which ML-based analysis was applied to study the quantitative parameters of immune infiltrate within the tumor immune microenvironment. By quantifying intraepithelial tumor-infiltrating lymphocytes, stromal tumor-infiltrating lymphocytes, and the tumor-stroma ratio from CD3 and CD8 immunohistochemical stained whole-slide images, the authors classified five CRC subgroups with distinctive biological features and different prognostic behaviors. Indeed, the CD3+ and CD8+ T-cell infiltration is considered an important independent prognostic factor predicting CRC patient survival[25]. An automated image analysis–based workflow quantifying the tumor-infiltrating immune cells and tumor budding, at the invasive front, combined with a ML approach, demonstrated that the spatial association of lymphocytes and tumor buddings could provide a high prognostic significance in stage II CRC patients[26].

AI for identification of peculiar molecular subgroups

MSI-high (MSI-H) tumors are a peculiar molecular subgroup of CRC characterized by deficient expression of mismatch repair proteins (dMMR phenotype)[27]. MSI-H is associated with the inherited Lynch syndrome and found to be a marker of increased tumor mutational burden, neoantigen generation and exceptional sensitivity to immune checkpoint inhibitors[28]. It accounts for 4%-5% of metastatic CRC and its early recognition is now crucial to avoid useless treatment with conventional chemotherapy and to tailor adequate and effective immunotherapy. Kather et al[29] have recently built an AI-classifier of conventionally HE-stained slides of gastric and CRC for the detection of MSI-H. The model used Resnet18, a residual learning convolutional neural network (CNN), which was trained and validated, both internally and externally, in a very large cohort of gastrointestinal cancer. In particular, Resnet18 was initially trained to automatically identify tumor areas within the normal tissue background in common histological slides. In a second step, it was trained to define the degree of MSI using TCGA bank, and specifically 315 and 360 formalin-fixed paraffin embedded gastric and CRC specimens, respectively, and 378 snap-frozen CRC samples. TGCA of stomach cancer was 80% from non-Asian patients, and authors tested the possible influence of ethnicity on the model performance by validating the AI classifier in an external cohort of 185 Japanese patients.

The discriminatory power of the AI tool for the MSI-H status identification was impressively high (AUC around 80% in almost all experiments) across all the different sets: gastric vs CRC tissues, Asian vs non-Asian patients, and formalin-fixed paraffin embedded vs snap-frozen samples. Moreover, it was found that minimal required number of histological “tiles” on which to train and interrogate the algorithm (the ML process starts by fragmenting the entire histologic digital image in multiple elementary color-normalized slide tiles) was as small as those encompassed in a common core needle biopsy (performance plateaued at approximately 100 “slide tiles” of 256 μm edge length). The logical next step in this field will be to test the predictive value of AI-based MSI-H diagnosis for immunotherapy efficacy.

In another study by the same research group, a DL model for MSI definition was trained and validated in a cohort of nearly 9000 patients from Germany, Netherlands, United Kingdom and United States by using standard HE-stained slides and immunohistochemistry for mismatch repair proteins or conventional genetic test of microsatellite regions as ground truth for MSI status. In the validation step, the DL tool demonstrated an impressively high ability to correctly classify MSI-H patients with a discriminatory power > 95% (AUROC 0.95-096)[30].

AI for the quantification of stromal tissue

In CRC, the so-called desmoplastic reaction, i.e. a pronounced stromal tissue growth within the tumor mass, has been associated with reduced prognosis[31]. Desmoplastic reaction is driven by differentiated cells responsible for the deposition of extracellular fibrotic material, the cancer-associated fibroblasts (CAFs). Kather et al[32] have used DL for automatic quantification of stromal tissue proportion in CRC tissues and set up a “deep stroma score”. They first trained the DL instrument with over 100000 digital image patches of HE-stained tumors from 86 CRC patients where stromal and non-stromal areas were hand-delineated. The DL tool so obtained was then tested in an independent set of over 7000 images from 25 patients, where it demonstrated an accuracy of > 94%.

The tool, that was able to recognize tissue-specific features, was finally used to produce the “deep stroma score” which was calculated for a cohort of 500 colorectal patients from the TCGA repository and correlated with survival. The stroma score significantly and independently associated with overall survival with a hazard ratio of 1.99, P = 0.003. In the same patient group, manual stroma quantification by experienced pathologists or stromal assessment by means of gene expression profiles attributable to CAFs had an inferior performance in terms of survival prediction across the different tumor stages (from I to IV). The DL stroma score yielded similar prognostic efficacy in an independent validation cohort of over 400 patients from Germany: hazard ratio for overall survival 2.29, P = 0.0004.

BIOLOGY-AGNOSTIC ML

DL is a novel type of ML that uses a more “blind” approach to analyze the input data with no pre-definition of hand-crafted features to train the model. An extreme way to apply DL is the direct correlation of the digital input data to the final outcome, for example risk of disease relapse vs definitive care after surgical removal of the tumor, with no specific “biological insights” fed into the machine. Skrede et al[33] have applied an advanced DL method to analyze whole-slide image, using CNN. CNN makes use of mathematical convolutions, i.e. resulting mathematical functions obtained by combining more primary functions reversed and shifted in time. In Skrede et al[33]’s experiment, no clinical or biological information (such as grade of histological differentiation, neurovascular invasion, depth of tumor infiltration within the colon wall, or others) was fed into the model, and the final objective was to identify “biology-agnostic” digital image profiles (purely based on patterns of pixel color or intensity of the digital images) predictive of CRC specific survival in surgically-resected patients (stages II and III). Four large cohorts of approximately 1000 patients each were used (training, tuning, test and validation cohorts). The CNN-based automatic prognostic classifier, that authors called DoMore-v1-CRC, was initially trained for outcome prediction in a cohort of patients constituted of two subgroups with “pre-labeled” distinct outcomes: good (survival > 2 years) vs poor (relapse within 6 mo) outcome. DoMore-v1-CRC was then tuned, tested and validated in cohorts with non-prelabeled outcome to define its prognostic ability. The DoMore-v1-CRC had an output of three possible classes, good vs uncertain vs poor prognosis. The DoMore-v1-CRC–based classification had an independent prognostic value in a multivariable survival analysis that included and adjusted for well-known and broadly-used clinical and biological factors, such as pN stage, pT stage, lymphatic invasion, and venous vascular invasion (hazard ratio for DoMore-v1-CRC-defined poor vs good prognosis 3.04, P < 0.0001). Accuracy in predicting cancer-specific survival was around 70% in all cohorts. The prognostic power remained high for both stage II and stage III tumors, and for groups of slides prepared in different laboratories, thus confirming its robustness against inter-laboratory variability in tissue preparation and staining. Moreover, the prognostic yield remained high when two different image scanners were compared (NanoZoomer XR vs Aperio AT2 scanner). Interestingly, even though derived from a biology-agnostic approach, DoMore-v1-CRC significantly correlated with clinical and biological markers, such as age, pN stage, pT stage, histological grade, tumor sidedness, BRAF mutation, and microsatellite instability.

CONCLUSION

Even if still at the early stages, AI in digital pathology is increasingly gaining momentum, for a number of reasons: (1) worldwide transferability; (2) high-dimension of analyzable data; and (3) automated procedural approaches. Far from being conceived as substitutive of human intelligence, it can be seen as a useful companion diagnostic in oncology and as pre-screening/pre-selection tool. Both biology-driven and biology-agnostic algorithms have been pursued in the implementation of AI in the CRC pathology field (Table 1).

Table 1 Summary of main applications and findings of artificial intelligence in colorectal cancer digital pathology.

Target	Description	Ref.
Cancer cell shape and organization to predict N+	Digital assessment of cancer cells using Feret’s diameters allows to predict lymph node metastasis in pT1 colon cancer	[11]
Assessment of anti-cancer immune response	Simultaneous assessment of all immune cell subpopulations and demonstration that eosinophils, other than T cells, may play a role in CRC immune response	[12,14,15,17]
Identification of Microsatellite instable tumors	Rapid and large size screening of histopathologic features in conventional HE-stained slides increasing the probability of being a MSI-H tumor, without the need of specific immunohistochemical or molecular testing	[18]
Quantification of stroma within the tumor	Algorithms for tissue –specific recognition, even when sparse within the tumor mass have been development. These algorithms have allowed the validation of “deep stroma score” which is significantly associated with survival in CRC	[20]
Biology-agnostic prediction of survival	Development of tissue “digital” profiles without specific underlying biologic background or significance that are predictive of distinct survivals, bad vs good outcome	[21]

CRC: Colorectal cancer; MSI-H: Microsatellite instability high.

The main challenges for AI-supported digital pathology are currently to demonstrate that can diffusely be applied and significantly help in the clinical decision-making especially as predictor of efficacy of specific therapeutic options. Such a demonstration can only come with an enhancement of the AI application itself to different cancer patient settings and different clinical scenarios and problem-solving tasks. Future directions rely on the prospective validation of AI-based tools in randomized phase III trials, which delineates the so-called “IA level” of evidence.