Use of shape-from-shading to characterize mucosal topography in celiac disease videocapsule images

Abstract

AIM

To use a computerized shape-from-shading technique to characterize the topography of the small intestinal mucosa.

METHODS

Videoclips comprised of 100-200 images each were obtained from the distal duodenum in 10 celiac and 10 control patients. Images with high texture were selected from each videoclip and projected from two to three dimensions by using grayscale pixel brightness as the Z-axis spatial variable. The resulting images for celiac patients were then ordered using the Marsh score to estimate the degree of villous atrophy, and compared with control data.

RESULTS

Topographic changes in celiac patient three-dimensional constructs were often more variable as compared to controls. The mean absolute derivative in elevation was 2.34 ± 0.35 brightness units for celiacs vs 1.95 ± 0.28 for controls (P = 0.014). The standard deviation of the derivative in elevation was 4.87 ± 0.35 brightness units for celiacs vs 4.47 ± 0.36 for controls (P = 0.023). Celiac patients with Marsh IIIC villous atrophy tended to have the largest topographic changes. Plotted in two dimensions, celiac data could be separated from controls with 80% sensitivity and specificity.

CONCLUSION

Use of shape-from-shading to construct three-dimensional projections approximating the actual spatial geometry of the small intestinal substrate is useful to observe features not readily apparent in two-dimensional videocapsule images. This method represents a potentially helpful adjunct to detect areas of pathology during videocapsule analysis.

Key Words: Celiac disease, Duodenum, Shape-from-shading, Small intestine, Videocapsule

Core tip: Videocapsule images can assist in determining the presence and status of celiac disease; however, pathology is not always apparent by visual inspection. A computerized shape-from-shading technique was used to characterize the topography of the small intestinal mucosa. It was hypothesized that the automated measure would be helpful to distinguish celiac from control images and to gauge the degree of villous atrophy in the celiac patient data.

INTRODUCTION

Celiac disease is prevalent throughout the world and affects approximately 1% of the population[1]. Patients with celiac disease are reactive to the protein gluten, which is present in wheat, rye, and barley grains[2]. Currently, the only treatment is a lifelong gluten-free diet[3]. It is a disease with often occult symptoms that differ from one individual to the next[4]. Diagnosis of celiac disease is difficult owing to the fact that symptoms are highly varied from one individual to another, both in their type and severity. In some patients with celiac disease, no symptoms may be evident[5]. Diagnosis is made by a positive antibody test, which is followed by biopsy of the abnormal appearing mucosa and confirmation of atrophy by light microscopy[6]. Utilizing light microscopy, the small intestinal mucosa is evaluated for presence and degree of villous atrophy and an increase in intraepithelial lymphocytes. Mucosal alterations in celiac disease are assigned a Marsh score, which varies from 1 (normal villous architecture but increased intraepithelial lymphocytes) to IIIA-C (severe degrees of villous atrophy accompanied by crypt hyperplasia and increased intraepithelial lymphocytes)[7].

Recent advances in imaging technology have enabled visualization of the small intestinal mucosa via a videocapsule to assess areas of villous atrophy[8]. This is convenient to use and is minimally invasive. Images are obtained at a rate of 2 per second or more. A light source within the capsule illuminates the mucosal surface. The videocapsule is swallowed, sends information by radio control, and then passes harmlessly through the gastrointestinal system[9]. Analysis of two-dimensional videocapsule endoscopy images by quantitative means can assist in determining areas of pathology in these patients.

A difficulty with the use of videocapsule technology, is that areas of pathology are not always clearly identifiable on review of the images[10]. The two-dimensional images provide a very limited perspective of the actual three-dimensional structure of the substrate when viewed retrospectively by the data analyst. The endoscopist who is performing the procedure, views these same images via an endoscopic system, and is similarly at a disadvantage for understanding the three-dimensional mucosal architecture. Hence, it can be difficult for both endoscopist and data analyst to determine the precise regions and boundaries of any abnormality that is present in the images. This renders the detection of regions of patchy villous atrophy, which are important to biopsy for confirmation that there is pathology, and therefore for diagnosis of the disease and to monitor treatment, difficult at best.

For improved analysis, it would be useful to provide additional information regarding the mucosal substrate throughout the small intestine[11,12]. If it were possible to estimate the three-dimensional architecture of this substrate, and render it visually, it could be useful for improved detection of the presence and severity of villous atrophy, to detect any changes in architecture that occur after onset of a gluten-free diet, as well as to understand the mechanisms by which the structure of the small intestinal mucosa is altered during untreated vs treated celiac disease. In prior quantitative studies, a method was introduced to estimate three-dimensional structure from two-dimensional endoscopic images[13,14]. This technique uses the principle of shape-from-shading. In the shape-from-shading process, as a first approximation, image brightness is linearly related to image depth. Thus a third spatial axis, the Z-axis, is obtained so that a map of the three-dimensional structure of the substrate can be constructed. In this study, the visual manifestations of three-dimensional image projection are shown for celiac patients with various levels of villous atrophy, vs controls. Special attention is paid to the types of structures that are evident in the projections, and their variation from one patient to the next, which can be helpful to detect the presence and severity of villous atrophy during the diagnosis of celiac disease, and to evaluate treatment efficacy. The purpose of the study is to show that visualization of the three-dimensional architecture can be useful to detect pathology in the small intestinal mucosa when the presence of patchy villous atrophy is suspected.

MATERIALS AND METHODS

A retrospective data series from 10 celiac patients and 10 controls were used for analysis. All patients were evaluated at the Columbia University Medical Center, New York, NY using both standard and videocapsule endoscopy. Suspected celiac patients were diagnosed by the presence of villous atrophy in standard endoscopy images and improvement on follow-up endoscopy after onset of the gluten free diet. The indication for endoscopy in control patients included obscure bleeding, suspected Crohn’s disease, and diarrhea. The study exclusion criteria were patients less than 18 years of age, pregnancy, history of intestinal obstruction, presence of a pacemaker, and chronic use of non-steroidal anti-inflammatory drugs. Only studies in which the videocapsule reached the cecum were included for analysis. All included patients, except one celiac patient with hemophilia, first underwent a standard endoscopic procedure with biopsy to determine the presence and severity of any villous atrophy in the proximal duodenum. The patients then underwent videocapsule endoscopy.

Videocapsule endoscopy images were acquired using the PillCam SB2 videocapsule (Given Imaging, Yoqneam, Israel). The device included a recorder unit and its container, battery pack, antenna, harness for the recorder unit, and a battery charger. The capsule dimensions were 26 mm × 11 mm, and the frame rate for acquisiton was two digital images per second (2/s). After a 12 h overnight fast, all subjects swallowed the PillCam SB2 videocapsule with 200 cc water and 80 mg of simethicone. Subjects were permitted to drink water at 2 h following ingestion of the capsule, and to eat a small meal after 4 h. The data recorder was affixed to a belt worn by the patient, and received radio image signals transmitted by the videocapsule via an array sensor as it passed through the gastrointestinal tract. The videocapsule endoscopy images were recorded over an eight hours period. At the end of eight hours, the images were offloaded to a PC-type computer workstation. The videos were subsequently interpreted using Rapid5 software (Given Imaging, Yoqneam, Israel) by gastroenterologists, each with experience in reading many videocapsule endoscopies.

Videoclips of length 100-200 images (50-100 s at 2/s frame rate) were obtained from the distal duodenum in each patient and were deidentified prior to analysis. The use of patient data and the analysis protocols were approved by the Institutional Review Board of Columbia University Medical Center. The quantitative biopsy results obtained during standard endoscopy were used as a reference as to the presence and severity of villous atrophy.

An algorithm was developed to convert the two-dimensional endoscopic images from color to grayscale, and then to project to three dimensions using the shape-from-shading technique. The algorithm used in this study can be described as follows[13,14]: (1) at each pixel (x, y) location, extract the grayscale brightness level; (2) write brightness level b, which ranges from 0-255 (black to white), to file along with (x, y) location; (3) the format of the stored information is trivariate (x, y, b); (4) all (x, y, b) information for all image pixels (x, y), with the dimensions of the image being 576 × 576, are written to file; (5) display the file in map3d, a program which enables viewing of three-dimensional data objects from any perspective[15]; (6) separately, store the trivariate information along with the brightness value, i.e., as (x, y, b, b) where the fourth variable is used as a false color for enhanced display; and (7) organize the original two-dimensional endoscopic image, the three-dimensional projection, and the false-color three-dimensional image according to Marsh score pathology for the celiac patients, vs the control patients.

Once the data were displayed, special attention was given to the presence of certain three-dimensional structures that had been quantitatively modeled by syntactic means in prior work[13]. Specifically, the characteristics of mucosal protrusions present in the mucosa were assessed, highlighting differences in celiac patients with villous atrophy (Marsh IIIA, IIIB, or IIIC score) and celiac patients with little or no evident villous atrophy (Marsh II score) vs control patients lacking villous atrophy. The Marsh score was determined by the pathologist, who evaluated biopsy specimens for the presence of villous atrophy under light microscopy.

From the three-dimensional constructs, topographic variation was calculated using a computerized method. The first derivative of the elevation level of each image pixel, done row-by-row in an automated raster scan fashion, was determined. The mean absolute value of this derivative was used as one measure of topographic variation. The standard deviation of the absolute derivative was used as a second measure of topographic alteration. The mean and standard deviation of these parameters were calculated for celiac vs control image data, and the statistical significance of the difference was determined using the two-tailed t-test (SigmaPlot ver. 13, 2016, Systat Software Inc., San Jose, California). The parameter values were plotted, with celiac data labeled according to the Marsh score. The best linear discriminant function to separate celiac vs control data was determined, and the sensitivity and specificity for detecting pathology in celiacs, vs the lack of pathology in control patient images, were calculated.

RESULTS

The patient data used in the study are depicted in Tables 1 and 2. Information regarding the age, gender and Marsh score of small intestinal biopsies of celiac patients is shown in Table 1. Six of 10 patients (60%) were female. The average age of all celiac patients was 49.5 ± 15.3. The Marsh score of patient 1, who had hemophilia, could not be determined precisely as a biopsy could not be obtained, but significant pathology was apparent from visual inspection of the endoscopic images. This patient was estimated to have Marsh IIIC pathology. Analysis of biopsy results revealed there were two additional patients with Marsh score pathology of IIIC, one with IIIB, two with IIIA, and two with a Marsh score of II. The control patient data is shown in Table 2. There were four male and six female control patients, with an average age of 50.6 ± 23.8, similar to the average age of the celiac patients. Three of the control patients received the videocapsule because of abdominal pain presumed to be due to peptic duodenitis, two for suspected Crohn’s disease, and one each for inflammation of the esophagus due to reflux, severe esophagitis, and obscure bleeding. None of the control patients had any evidence of villous atrophy on biopsy.

Table 1 Patient data - celiac.

Number	Age	Gender	Marsh score
1	19	M	NA
2	44	M	IIIC
3	44	F	IIIC
4	40	F	IIIB
5	63	F	IIIA
6	38	F	IIIA
7	53	F	II
8	65	F	II
9	64	M	IIIC
10	65	M	IIIA

Patient 1 had hemophilia and had no biopsy, but was suspected to have a Marsh IIIC level of villous atrophy. NA: Not applicable, i.e., no biopsy was performed; M: Male; F: Female.

Table 2 Patient data-control.

Number	Age	Gender	Marsh score
1	31	F	Suspected Crohn’s disease
2	36	M	Severe esophagitis
3	36	F	Peptic duodenitis
4	86	F	Obscure bleeding
5	26	F	Peptic duodenitis
6	87	M	Peptic duodenitis
7	55	M	Esophageal inflammation due to reflux
8	28	M	Suspected Crohn’s disease
9	47	F	Suspected Crohn’s disease
10	74	F	Iron deficiency anemia

NA: Not applicable, i.e., no biopsy was performed; M: Male; F: Female.

Examples of image processing results are shown in Figures 1 and 2 for celiac patient images (Patients 1 and 2), each from a region of the distal duodenum with high apparent texture and pathology. The data from two patients is shown in each image. In Figure 1 are shown images from a patient with Marsh IIIC pathology score, and from the patient with hemophilia, who has similar apparent severe pathology and likely Marsh IIIC score. The two-dimensional endoscopic image is to the left for each patient data. There are similar appearances in rough texture, and the images are thought to have been acquired from regions with villous atrophy. Some scalloping of the mucosal folds is apparent in the celiac patient with Marsh IIIC score at lower left in the image, for example at the fold noted by the asterisk. The three-dimensional projection using shape-from-shading is provided at center. Large protrusions are evident throughout each three-dimensional construction. For perspective, the same location noted by an asterisk in the left panel is shown in the center panel for patient 2.

Figure 1 Celiac patient images. A-C: Patient 1; D-F: Patient 2; A, D: The grayscale endoscopic images; B, E: The three-dimensional projections; C, F: Projections in false color. Colors from blue to yellow, orange, and red represent areas with progressively greater amplitude along the Z-axis (vertical axis). Patient 1 did not have a biopsy obtained due to hemophilia but was thought to have a Marsh score IIIC. Patient 2 also had a Marsh score IIIC. Note the prominent protrusions evident in the three-dimensional projections of both patients, and the similar appearance of texture in the original two-dimensional endoscopic images.

Figure 2 Celiac patient images. A-C: Patient 3 (Marsh IIIC score); D-F: Patient 4 (Marsh IIIB pathology); A, D: The grayscale endoscopic images; B, E: The three-dimensional projections; C, F: Projections in false color. Colors from blue to yellow, orange, and red represent areas with progressively greater amplitude along the Z-axis (vertical axis). Again note the prominent protrusions evident in the three-dimensional projections of both patients, even though the appearance of texture is somewhat dissimilar in the original two-dimensional endoscopic images.

Depicted in the right-hand panel are smoothed three-dimensional projections with false color used to show depth. Again for perspective, the location of the scalloped area with asterisk is shown. The highest area in the false color three-dimensional image at right corresponds to the brightest area in the two-dimensional endoscopic image at left (triangular shaped ridge at upper center). Mucosal folds in the two-dimensional endoscopic images of both patients are readily identifiable as three-dimensional structures in the projection panels at center and right. The large fold at top in the two-dimensional endoscopic image of patient 1 is evident as a large mass of three-dimensional tissue structure in the projection image in the center and right-hand panels. Also evident in the three-dimensional images are some artifacts at the edges, which are due to the white lettering in the original endoscopic image prior to framing. These are left in the images to show orientation.

In Figure 2, data from celiac patients 3 and 4 are shown, with Marsh scores IIIC and IIIB, respectively. The distal duodenum of patient 3 has folds with marked scalloping, which are the curved structures at the edge of each mucosal fold (top panels). These folds are evident as very large and prominent three-dimensional structures in the projection images in the center and right panels. The bright horizontally-oriented ridge at center in the patient 3 endoscopic image at left is converted to a very prominent three-dimensional ridge in the center and right-hand panels. In the patient 4 data (Marsh IIIB score) there is marked folding, with many large protrusions on each fold (center panel), similar to the visual appearance of patient 3 data (center panel).

Examples of control patient data are shown in Figures 3 and 4. For the control patients, protrusion features appear in the three-dimensional projection images in the center panels. However, the protrusions appear to be diminished and less connected to meandering ripples, as compared with the celiac patients with Marsh III scores whose data are shown in Figures 1 and 2. The three-dimensional protrusion structures are sometimes markedly diminished or even completely absent over some areas of the projections for the control patient data (Figures 3 and 4).

Figure 3 Control patient images. A-C: Patient 1 (no villous atrophy); D-F: Patient 2 (no villous atrophy); A, D: The grayscale endoscopic images; B, E: The three-dimensional projections; C, F: Projections in false color. Note the lack of prominent protrusions evident in the three-dimensional projections of both patients, even though there is a dissimilar appearance of texture in the original two-dimensional endoscopic images.

Figure 4 Control patient images. A-C: Patient 3 (no villous atrophy); D-F: Patient 4 (no villous atrophy); A, D: The grayscale endoscopic images; B, E: The three-dimensional projections; C, F: Projections in false color. Note the lack of prominent protrusions evident in the three-dimensional projections of both patients, even though there is a dissimilar appearance of texture in the original two-dimensional endoscopic images.

Summary data for all patients are shown in Figure 5. On the abscissa is noted the standard deviation from the mean of the first derivative, while the ordinate axis gives the mean absolute first derivative. Celiac points (black) are labeled according to Marsh score. The patient data can be mostly separated based on the linear discriminant function (straight black line). Thus the sensitivity and specificity for classification are both 80%. The control patient data are mostly clustered together and the celiac patient data are mostly clustered together. Celiac patient data with Marsh IIIC scores as labeled, are all clustered toward the top right in the graph, i.e., they possess larger topographic variation based on both the absolute derivative and the standard deviation from the mean. The mean absolute derivative in elevation was 2.34 ± 0.35 brightness units for celiac images vs 1.95 ± 0.28 for controls (P = 0.014). The standard deviation of the derivative in elevation was 4.87 ± 0.35 brightness units for celiac images vs 4.47 ± 0.36 for controls (P = 0.023).

Figure 5 Scatterplot of celiac (black circles) vs control (open circles) patient topographic data. The ordinate scale gives the standard deviation of the derivative in elevation. The abscissa shows the absolute value of the derivative in elevation. A linear discriminant function mostly separates celiac vs control patient parameter values (black line). Based on the scatterplot, the topography of celiac three-dimensional constructs is typically greater than for controls.

DISCUSSION

Comparison with prior work

The study findings and observations are in accord with prior scanning electron microscopy studies conducted by Michael Marsh[16], and our prior work showing that the protrusions, when modeled syntactically, tend to be taller in height for celiac patients as compared with controls[13,14]. The mean height of protrusions was previously found to be 3.10 ± 2.34 grayscale levels in celiac patients with villous atrophy vs 2.70 ± 0.43 grayscale levels for controls (P < 0.001). The prominence of the mucosal protrusions in celiac patients, particularly IIIC patients, is evident in the three-dimensional constructs of Figures 1 and 2, and the resulting large variation in topography as measured via automated computerized means is evident in Figure 5. Large mucosal protrusions, when villous atrophy is present in celiacs, which likely corresponds to clumping of villi[13,14], translates to greater topographic variation (Figure 5). By comparison, control patient biopsies with normal villi tend to possess narrowed, smaller topographic structures (Figures 3 and 4) corresponding to individual villi, which reduces the overall degree of topographic change as compared with celiac patients with villous atrophy. In the false-color celiac images of Figures 1 and 2C, topographic structure often appears to include more high-elevation areas (combined red-orange false-color regions) as compared to controls (Figures 3 and 4), in agreement with the quantitative topographic results presented in Figure 5 and in the summary statistics. The mechanism by which the prominent three-dimensional architectural alterations occur in the celiac patient small intestinal mucosa is an interesting topic for further research.

Three-dimensional printing

The three-dimensional projections shown in the center panels of Figures 1-4 only provide a single snapshot of the tissue structure. A 3D printer would be useful in this regard to provide additional perspective. By printing the structure in three dimensions, the observer could view it from any perspective. Three-dimensional tissue reconstruction of intestinal villi has been demonstrated previously[17]. This could potentially be helpful for improved understanding concerning the relationship of small intestinal architecture with other disease features in celiacs. It would also be useful to improve syntactic modeling of structure. Protrusions can be modeled as square objects[13,14]. Although not a strictly correct interpretation of structure, the syntax utilized is sufficiently accurate to detect most or all mucosal protrusions automatically, and to estimate the height and width of each. Using a 3D printer for guidance, it would be possible to improve modeling of the projection features, as well as the syntactic modeling of any other pathologic structures embedded in the mucosa. Structural characteristics could be incorporated into the algorithm for improved, automated detection and measurement of the quantitative tissue structural characteristics in suspected and actual celiac patients.

Other rendering methods

Use of endoscopy for patient diagnosis and intervention is based on image sequences that are acquired via a video-camera. Yet, the sequence of images lacks depth information. Recognition and evaluation of pathology therefore becomes more difficult. In this study, a straightforward method was described for rendering three-dimensional surfaces, which can be useful in real time during endoscopic procedures, as well as for retrospective analysis. More complex procedures for three-dimensional rendering of the mucosa include one that solves the structure-from-motion problem using parallax, i.e., camera motion is estimated, and it is used to reconstruct the three-dimensional scene[18]. Another method is to use an optical fiber to act as a probe to transmit three-dimensional information regarding the internal landscape[19]. Volumetric images are obtained by using a spectrally encoded endoscopy system. The resulting images provide depth information, although the process would be slow to complete for all areas where villous atrophy is suspected, due to the need to direct the probe across small areas at a time. An advantage of syntactic and textural-based methods for videocapsule analysis is the speed of computation, which would be useful for real-time analysis[20,21]. These methods, like the calculation done in this study to generate Figure 5, are entirely automated, thus eliminating observer bias.

Limitations

The method uses shape-from-shading as a linear function. Thus depth was as a first approximation, linearly dependent on brightness. However, this is not precisely correct, as the actual function will be nonlinear. Construction and update of the three-dimensional projections in real-time by computerized means could be done during the endoscopic procedure, but was not included in this study. Small intestinal pathology is patchy in celiac patients, thus the results obtained from analysis of selected videocapsule images may differ from histopathologic findings, as the sites chosen for biopsy do not correspond to the images and they only represent limited sampling of the mucosa. The series of patients used for celiac vs control cohorts, n = 8 each, should be increased for confirmation of the results. Use of a larger sample size in subsequent studies may assist in clarifying the specific clinical settings in which this methodology will be useful. The Marsh scoring of celiac patient data shown in Figure 5 is not entirely aligned with the magnitude of the x and y variables, perhaps suggesting that there is a lag between cellular-level phenomena and gross architectural structure.

COMMENTS

Background

Celiac disease is prevalent throughout the world and affects approximately 1% of the population. Patients with celiac disease are reactive to the protein gluten, which is present in wheat, rye, and barley grains. Currently, the only treatment is a lifelong gluten-free diet. It is a disease with often occult symptoms that differ from one individual to the next. Diagnosis of celiac disease is difficult owing to the fact that symptoms are highly varied from one individual to another, both in their type and severity. In some patients with celiac disease, no symptoms may be evident. Diagnosis is made by a positive antibody test, which is followed by biopsy of the abnormal appearing mucosa and confirmation of atrophy by light microscopy. Utilizing light microscopy, the small intestinal mucosa is evaluated for presence and degree of villous atrophy and an increase in intraepithelial lymphocytes. Mucosal alterations in celiac disease are assigned a Marsh score, which varies from 1 (normal villous architecture but increased intraepithelial lymphocytes) to IIIA-C (severe degrees of villous atrophy accompanied by crypt hyperplasia and increased intraepithelial lymphocytes).

Research frontiers

Recent advances in imaging technology have enabled visualization of the small intestinal mucosa via a videocapsule to assess areas of villous atrophy, which is convenient to use and is minimally invasive. Analysis of two-dimensional videocapsule endoscopy images by quantitative means can assist in determining areas of pathology in these patients. A difficulty with the use of videocapsule technology, is that areas of pathology are not always clearly identifiable on review of the images. The two-dimensional images provide a very limited perspective of the actual three-dimensional structure of the substrate. It’s difficult to determine the precise regions and boundaries of any abnormality that is present in the images. In prior quantitative studies, a method was introduced to estimate three-dimensional structure from two-dimensional endoscopic images, which uses the principle of shape-from-shading. As a first approximation, image brightness is linearly related to image depth. Thus a third spatial axis, the Z-axis, is obtained so that a map of the three-dimensional structure of the substrate can be constructed. The purpose of this study is to show that visualization of the three-dimensional architecture can be useful to detect pathology in the small intestinal mucosa when the presence of patchy villous atrophy is suspected.

Innovations and breakthroughs

In this study, the visual manifestations of three-dimensional image projection are shown for celiac patients with various levels of villous atrophy, vs controls. Special attention is paid to the types of structures that are evident in the projections, and their variation from one patient to the next, which can be helpful to detect the presence and severity of villous atrophy during the diagnosis of celiac disease, and to evaluate treatment efficacy.

Applications

This study showed that visualization of the three-dimensional architecture can be useful to detect pathology in the small intestinal mucosa when the presence of patchy villous atrophy is suspected.

Peer-review

This study shows an interesting new approach to be validated in a prospective way and bigger sample size in order to clarify the potential use in specific clinical situations of celiac patients.