Prediction of clinically actionable genetic alterations from colorectal cancer histopathology images using deep learning

Figure 1 Fully automated prediction of mutation with three consecutive classifiers. A: Proper tissue patches can be selected by the tissue/non-tissue classifier. The four insets in the middle panel demonstrated the tissue patches representing pen marking, blurry scanned area, background rich region, and tissue folding, clockwise from top left, all removed by the tissue/non-tissue classifiers. Then, the normal/tumor classifier delineates the tumor patches among the proper tissue patches; B: The wild-type/mutation classifiers are applied only for patches with tumor probability higher than 0.9. The patch-level probabilities of mutation are averaged to yield the slide-level probability.

Figure 2 Classifiers to predict APC gene mutation for the Cancer Genome Atlas colorectal cancer tissue slides. A: Representative whole slide images (WSIs) of the frozen slides with APC gene mutation correctly classified as mutation, with wild-type gene correctly classified as wild-type, with gene mutation falsely classified as wild-type, and with wild-type gene falsely classified as mutation, from left to right; B: Receiver operating characteristic curves for the fold with lowest area under the curve (AUC), for the fold with highest AUC, and for the concatenated results of all ten folds, from left to right, obtained with the classifiers trained with the frozen tissues; C and D: Same as A and B, but the results were for the formalin-fixed paraffin-embedded WSIs. APC-M: APC mutated; APC-W: APC wild-type; AUC: Area under the curve; FFPE: Formalin-fixed paraffin-embedded.

Figure 3 Classifiers to predict KRAS gene mutation for the Cancer Genome Atlas colorectal cancer tissue slides. A: Representative whole slide images (WSIs) of the frozen slides with KRAS gene mutation correctly classified as mutation, with wild-type gene correctly classified as wild-type, with gene mutation falsely classified as wild-type, and with wild-type gene falsely classified as mutation, from left to right; B: Receiver operating characteristic curves for the fold with lowest area under the curve (AUC), for the fold with highest AUC, and for the concatenated results of all ten folds, from left to right, obtained with the classifiers trained with the frozen tissues; C and D: Same as A and B, but the results were for the formalin-fixed paraffin-embedded WSIs. KRAS-M: KRAS mutated; KRAS-W: KRAS wild-type; AUC: Area under the curve; FFPE: Formalin-fixed paraffin-embedded.

Figure 4 Classifiers to predict PIK3CA gene mutation for the Cancer Genome Atlas colorectal cancer tissue slides. A: Representative whole slide images (WSIs) of the frozen slides with PIK3CA gene mutation correctly classified as mutation, with wild-type gene correctly classified as wild-type, with gene mutation falsely classified as wild-type, and with wild-type gene falsely classified as mutation, from left to right; B: Receiver operating characteristic curves for the fold with lowest area under the curve (AUC), for the fold with highest AUC, and for the concatenated results of all ten folds, from left to right, obtained with the classifiers trained with the frozen tissues; C and D: Same as A and B, but the results were for the formalin-fixed paraffin-embedded WSIs. PIK3CA-M: PIK3CA mutated; PIK3CA-W: PIK3CA wild-type; AUC: Area under the curve; FFPE: Formalin-fixed paraffin-embedded.

Figure 5 Classifiers to predict SMAD4 gene mutation for the Cancer Genome Atlas colorectal cancer tissue slides. A: Representative whole slide images (WSIs) of the frozen slides with SMAD4 gene mutation correctly classified as mutation, with wild-type gene correctly classified as wild-type, with gene mutation falsely classified as wild-type, and with wild-type gene falsely classified as mutation, from left to right; B: Receiver operating characteristic curves for the fold with lowest area under the curve (AUC), for the fold with highest AUC, and for the concatenated results of all ten folds, from left to right, obtained with the classifiers trained with the frozen tissues; C and D: Same as A and B, but the results were for the formalin-fixed paraffin-embedded WSIs. SMAD4-M: SMAD4 mutated; SMAD4-W: SMAD4 wild-type; AUC: Area under the curve; FFPE: Formalin-fixed paraffin-embedded.

Figure 6 Classifiers to predict TP53 gene mutation for the Cancer Genome Atlas colorectal cancer tissue slides. A: Representative whole slide images (WSIs) of the frozen slides with TP53 gene mutation correctly classified as mutation, with wild-type gene correctly classified as wild-type, with gene mutation falsely classified as wild-type, and with wild-type gene falsely classified as mutation, from left to right; B: Receiver operating characteristic curves for the fold with lowest area under the curve (AUC), for the fold with highest AUC, and for the concatenated results of all ten folds, from left to right, obtained with the classifiers trained with the frozen tissues; C and D: Same as A and B, but the results were for the formalin-fixed paraffin-embedded WSIs. TP53-M: TP53 mutated; TP53-W: TP53 wild-type; AUC: Area under the curve; FFPE: Formalin-fixed paraffin-embedded.

Figure 7 Mutation prediction of APC and KRAS genes for the Seoul St. Mary Hospital colorectal cancer tissue slides by the classifiers trained with both The Cancer Genome Atlas and Seoul St. Mary Hospital data. A: Representative whole slide images of the slides with APC gene mutation correctly classified as mutation, with wild-type gene correctly classified as wild-type, with gene mutation falsely classified as wild-type, and with wild-type gene falsely classified as mutation, from left to right; B: Receiver operating characteristic curves for the fold with lowest area under the curve (AUC), for the fold with highest AUC, and for the concatenated results of all ten folds, from left to right; C and D: Same as A and B, but the results were for the KRAS gene. SMH: Seoul St. Mary Hospital; APC-M: APC mutated; APC-W: APC wild-type; KRAS-M: KRAS mutated; KRAS-W: KRAS wild-type; AUC: Area under the curve; FFPE: Formalin-fixed paraffin-embedded.

Figure 8 Mutation prediction of PIK3CA, SMAD4, and TP53 genes for the Seoul St. Mary Hospital colorectal cancer tissue slides by the classifiers trained with both The Cancer Genome Atlas and Seoul St. Mary Hospital data. A: Representative whole slide images of the slides with PIK3CA gene mutation correctly classified as mutation, with wild-type gene correctly classified as wild-type, with gene mutation falsely classified as wild-type, and with wild-type gene falsely classified as mutation, from left to right; B: Receiver operating characteristic curves for the fold with lowest area under the curve (AUC), for the fold with highest AUC, and for the concatenated results of all ten folds, from left to right; C and D: Same as A and B, but the results were for the SMAD4 gene; E and F: Same as A and B, but the results were for the TP53 gene. PIK3CA-M: PIK3CA mutated; PIK3CA-W: PIK3CA wild-type; SMAD4-M: SMAD4 mutated; SMAD4-W: SMAD4 wild-type; TP53-M: TP53 mutated; TP53-W: TP53 wild-type; AUC: Area under the curve; FFPE: Formalin-fixed paraffin-embedded.