Performance of the Montreal classification for inflammatory bowel diseases

Abstract

AIM: To validate the Montreal classification system for Crohn’s disease (CD) and ulcerative colitis (UC) within the Netherlands.

METHODS: A selection of 20 de-identified medical records with an appropriate representation of the inflammatory bowel disease (IBD) sub phenotypes were scored by 30 observers with different professions (gastroenterologist specialist in IBD, gastroenterologist in training and IBD-nurses) and experience level with IBD patient care. Patients were classified according to the Montreal classification. In addition, participants were asked to score extra-intestinal manifestations (EIM) and disease severity in CD based on their clinical judgment. The inter-observer agreement was calculated by percentages of correct answers (answers identical to the “expert evaluation”) and Fleiss-kappa (κ). Kappa cut-offs: < 0.4-poor; 0.41-0.6-moderate; 0.61-0.8-good; > 0.8 excellent.

RESULTS: The inter-observer agreement was excellent for diagnosis (κ = 0.96), perianal disease (κ = 0.92) and disease location in CD (κ = 0.82) and good for age of onset (κ = 0.67), upper gastrointestinal disease (κ = 0.62), disease behaviour in CD (κ = 0.79) and disease extent in UC (κ = 0.65). Disease severity in UC was scored poor (κ = 0.23). The additional items resulted in a good inter-observer agreement for EIM (κ = 0.68) and a moderate agreement for disease severity in CD (κ = 0.44). Percentages of correct answers over all Montreal items give a good reflection of the inter-observer agreement (> 80%), except for disease severity (48%-74%). IBD-nurses were significantly worse in scoring upper gastrointestinal disease in CD compared to gastroenterologists (P = 0.008) and gastroenterologists in training (P = 0.040). Observers with less than 10 years of experience were significantly better at scoring UC severity than observers with 10-20 years (P = 0.003) and more than 20 years (P = 0.003) of experience with IBD patient care. Observers with 10-20 years of experience with IBD patient care were significantly better at scoring upper gastrointestinal disease in CD than observers with less than 10 years (P = 0.007) and more than 20 years (P = 0.007) of experience with IBD patient care.

CONCLUSION: We found a good to excellent inter-observer agreement for all Montreal items except for disease severity in UC (poor).

Key Words: Crohn’s disease, Ulcerative colitis, Montreal classification, Phenotypes- inter-observer agreement

Core tip: According to our study, the Montreal is a reliable classification system for phenotypes in inflammatory bowel disease, except for disease severity in ulcerative colitis. The inter-observer agreement for scoring Crohn’s disease severity was moderate. This highlights the need for accurate medical reporting and the use of additional parameters to define and classify disease severity. Such alternations are necessary to ensure high-quality data in multicentre prospective data collections.

INTRODUCTION

Inflammatory bowel diseases (IBD) are common, chronic relapsing gastrointestinal inflammatory disorders consisting of mainly two diseases: Crohn’s disease (CD) and ulcerative colitis (UC). IBD affects approximately 1 in 1000 individuals in Western Europe[1,2].

In CD inflammation is transmural and can occur throughout the entire gastrointestinal tract, in UC the inflammation is limited to the mucosal layer of the colon[3,4]. In addition to intestinal inflammation, up to 25% of the patients have extra-intestinal symptoms like uveitis, arthritis and erythema nodosum. Management of IBD with drug therapy consists of mesalazine, corticosteroids, and immunosuppressants like azathioprine and anti-tumor necrosis factor (TNF) antibody therapies. Most of these treatments have significant side effects, are expensive and often ineffective. Half of the patients (25%-30% in UC, 70%-75% in CD) require surgical intestinal resections because of refractory disease, fibrostenotic disease, abscesses, fistulae or the development of colorectal cancer[5-9].

The pathogenesis of IBD is still not fully understood. The current hypothesis is that it arises from an inappropriate activation of the mucosal immune system in response to commensal bacteria in a genetically susceptible host[10,11]. Several biological pathways that play a role in this inappropriate inflammation have been identified through genetic studies. Recently, the International IBD Genetics Consortium has identified 163 independent genetic susceptibility loci[12-15]. However, the translation of biological knowledge on the pathogenesis of IBD towards the clinic is complicated by the great variety in the clinical presentation of IBD. For both clinical and genetic research it is of great importance that phenotypes of patients are described in a consistent manner.

In 2000 the Vienna classification was introduced, which was the first attempt to classify different clinical phenotypes of CD[16]. The Vienna classification was followed by the Montreal classification in 2008[17]. The Montreal classification describes the extent and behaviour of CD in more detail and includes a classification system for UC (Table 1)[17]. Although the Montreal classification is widely used in both research and clinical practice, there is very limited data available on its reliability. Only two studies assessed the inter-observer reliability and validity of the Montreal classification, an Australian-New Zealand study and a study of the National Institutes of Diabetes and Digestive and Kidney Diseases IBD Genetics Consortium. Both studies had a small number of observers. The Australian-New Zealand study assessed only reliability of the Montreal classification in CD. In both studies the inter-observer agreement was good for disease location, but only moderate/fair for upper gastrointestinal involvement[18,19].

Table 1 Montreal classification of Crohn’s disease, ulcerative colitis, non-classified chronic colitis and indeterminate colitis.

Diagnosis (20 case-vignettes)
Crohn’s Disease (CD)
Ulcerative Colitis(UC)
Non-classified chronic colitis (IBD-U)
Indeterminate colitis (IBD-I)
Age of onset (A) (20 case-vignettes)
A1: 16 yr or younger
A2: 17-40 yr
A3: over 40 yr
CD (10 case-vignettes)	UC, IBD-U, IBD-I (10 case-vignettes)
Localization (L)	Disease extent (E)
L1: Terminal ileum	E1: Proctitis
L2: Colon	E2: Left-sided UC; proximal extent of inflammation is distal to the rectosigmoid
L3: Ileocolon
L4: Upper gastrointestinal	E3: Extensive UC; involvement extends proximal to the splenic flexure.
P: Perianal disease
Behavior (B)	Disease severity (S)
B1: Nonstricturing, nonpenetrating B2: Stricturing B3: Penetrating	S0: Remission, no symptoms S1: Mild, ≤ 4 ×/d stools, no systemic signs of toxicity, normal ESR S2: Moderate, > 4 ×/d stools, minimal systemic signs of toxicity S3: Severe, ≥ 6 ×/d stools, pulse > 90 beats/min, temperature > 37.5, Hemoglobin < 6.5 mmol/L, ESR > 30 mm

EIM: Extra-intestinal manifestations; ESR: Erythrocyte sedimentation rate.

The aim of this study is to validate the Montreal phenotype classification for both CD and UC in the Netherlands. Secondly, we will assess the influence of one’s profession (gastroenterologist, gastroenterologist in training, IBD-nurse) and level of experience (< 10 years, 10-20 years, > 20 years) on the reliability of the Montreal classification scoring.

MATERIALS AND METHODS

Cases and observers

Twenty patient records were selected from the specialized IBD unit of the Department of Gastroenterology and Hepatology of the University Medical Center Groningen, the Netherlands (10 case vignettes) and the IBD unit of the Gastroenterology and Hepatology department of the non-university medical center Isala Clincs, Zwolle, the Netherlands (10 case vignettes). The case vignettes consisted of clinical-, endoscopy-, pathology- and operation reports. All case vignettes were anonymized and the selection gave an appropriate representation of the IBD sub phenotypes (Figure 1).

Figure 1 Distribution of the different categories of the Montreal classification for all 20 case vignettes that were scored by 30 observers. CD: Crohn’s disease; IBD: Inflammatory bowel disease; UC: Ulcerative colitis; EIM: Extra-intestinal manifestations.

The expert panel consisted of two gastroenterologists experienced in IBD care (Dijkstra G, Weersma RK), and one gastroenterologist/PhD in training (Visschedijk MC). The expert panel first assessed the 20 case vignettes separately, discussed their findings and developed an “expert evaluation” for all Montreal items in the 20 case vignettes. This “expert evaluation” was considered as the correct answer. Two additional items were added. Firstly, CD severity was added, because the Montreal classification only allows scoring severity of UC. The Montreal classification does not include any parameters to score severity of CD, therefore observers were asked to give an impression of CD severity (mild, moderate, severe) based on their own clinical experience and judgment. Secondly, observers were asked to score whether any extra-intestinal manifestations (EIM) were present.

The 20 case vignettes were sent to 49 observers with different experience levels and professions: gastroenterologists specialized in IBD, gastroenterologists in training and IBD-nurses with a day-to-day experience with IBD patients, all from university and non-university medical centers. The observers received the selected 20 case vignettes, instructions by e-mail and a hyperlink to fill out the online survey (https://http://www.enquetesmaken.com/), in which the Montreal classification and the two additional items, EIM and CD severity, had to be scored.

The online survey contained the following main items: diagnosis, age of onset and EIM. For the CD case vignettes the observers had to fill in disease location, disease behavior and disease severity. For the UC case vignettes the observers had to score disease extent and disease severity (Table 1). The diagnosis of CD and UC is standardized and uniformly accepted. However, in 10%-20% of the patients it is difficult to differentiate between CD and UC. These patients are classified as having non-classified chronic colitis (IBD-U). If the pathologist can’t differentiate between CD and UC after a colectomy, the patient is classified as having indeterminate colitis (IBD-I)[20-22]. Case vignettes with the diagnosis IBD-U or IBD-I are scored as UC, according to the Montreal classification.

Statistical analysis

Statistical analysis was performed using R statistical software. Firstly the inter-observer agreement was calculated using percentages of correct answers. An answer was scored correct if the answer of the observer was identical to the “expert evaluation”, percentages of correct answers were calculated for all items.

Secondly Fleiss-kappa (k) was calculated, which is the standard method to calculate the inter-observer agreement for multiple observers[23]. An observer can only be included in the statistical analysis on the condition that one Montreal item is scored by the observer in all case vignettes. In case of one missing value in one case vignette the observer was excluded from the statistical analysis for this item. The Fleiss-kappa cut-offs were set as follows: < 0.4 poor agreement; 0.41-0.60 moderate agreement; 0.61-0.8 good agreement; > 0.8 excellent agreement.

Subgroup analyses for the inter-observer agreement between profession (gastroenterologist, gastroenterologist in training, IBD-nurse) and level of experience (< 10 years; 10-20 years; > 20 years) were performed by percentages of correct answers. An additional Fisher exact test was used to identify significant differences between the subgroups.

RESULTS

Observers

The online survey was available for six weeks, in which the 49 observers received several reminders. Eventually 30 of the 49 observers completed the survey, a response rate of 61%. Details of the observers are depicted in Table 2. Fifty-four percent of the responders were gastroenterologist and 67% of the observers had less than 10 years experience with IBD patient care.

Table 2 Characteristics of 30 observers n (%).

	< 10 yr of experience with IBD patients	10-20 yr of experience with IBD patients	> 20 yr of experience with IBD patients	Non-university center	University medical center	Total
Gastroenterologist	7 (23%)	5 (17%)	4 (13%)	5 (17%)	11 (37%)	16 (54%)
Gastroenterologist in training	10 (33%)			3 (10%)	7 (23%)	10 (33%)
IBD-nurse	3 (10%)	1 (3%)		1 (3%)	3 (10%)	4 (13%)
Total	20 (67%)	6 (20%)	4 (13%)			100%

IBD: Inflammatory bowel disease.

Correct ratings

Average percentage of correctly answered questions for all Montreal and additional items (CD severity and EIM) by different professions was 85%. Age of onset, disease location, perianal disease and disease behaviour in CD had more than 90% correct score over all professions. Disease severity in UC was the worst scored item overall with less than 55% correctly scored by all three professions (Table 3).

Table 3 Percentages of correct answers according to the “expert evaluation” for overall and divided for different professions.

	Overall correct answers	Gastroenterologist	Gastroenterologist in training	IBD-nurse
Age of onset	94.0%	94.2%	96.8%	86.0%
Diagnosis	96.9%	96.0%	96.0%	98.8%
CD disease Localization	94.0%	90.6%	93.8%	97.5%
CD upper gastrointestinal	91.3%	95.6%	94.0%	84.2%
CD perianal disease	98.0%	99.4%	98.0%	96.7%
CD Disease behavior	92.4%	92.4%	94.8%	90.0%
CD severity (mild, moderate, severe)	73.9%	68.7%	72.9%	80.0%
UC disease extent	84.0%	89.3%	85.2%	77.5%
UC disease severity colitis	50.7%	53.9%	50.7%	47.5%
EIM	82.1%	82.1%	85.8%	78.5%

EIM: Extra-intestinal manifestations; CD: Crohn’s disease; UC: Ulcerative colitis.

When observers were grouped according to their profession, the additional item severity of the disease (CD), was scored worst by the gastroenterologists (69%) and best by the IBD nurses (80%). IBD-nurses had an excellent correct score on diagnosis (99%) as well as the gastroenterologists (96%) and gastroenterologists in training (96%). According to the fisher exact test, no significant differences were found between the three professions except for scoring of upper gastrointestinal disease in CD, in which IBD-nurses scored significantly worse than gastroenterologists (P = 0.008) and gastroenterologists in training (P = 0.040).

After calculation of the percentages of correct answers for the observers based on their level of experience, all items of the Montreal classification and the EIM scored above 80%, except for disease severity in UC (48%). The additional item, CD severity, was scored correctly in 70% of cases. Observers with less than 10 years of experience performed best at scoring disease severity (Table 4) and were significantly better at scoring UC severity than observers with 10-20 years (P = 0.003) and more than 20 years (P = 0.003) of experience with IBD patient care. Observers with 10-20 years of experience with IBD patient care were significantly better at scoring upper gastrointestinal disease in CD than observers with less than 10 years (P = 0.007) and more than 20 years (P = 0.007) of experience with IBD patient care.

Table 4 Percentages of correct answers compared to the “expert evaluation” overall and divided for different years of experience with inflammatory bowel disease patients.

	Overall correct answers	< 10 yr of experience	10-20 yr of experience	> 20 yr of experience
Age of onset	92.3%	95.2%	93.0%	88.6%
Diagnosis	96.2%	97.0%	93.3%	98.3%
CD disease localization	90.6%	94.2%	90.0%	87.5%
CD upper gastrointestinal	94.8%	91.8%	100.0%	92.5%
CD perianal disease	98.9%	98.5%	98.3%	100.0%
CD disease behavior	93.0%	92.3%	96.7%	90.0%
CD severity (mild, moderate, severe)	69.9%	73.1%	70.0%	66.7%
UC disease extent	86.6%	86.2%	85.3%	88.3%
UC disease severity colitis	48.0%	56.2%	37.7%	50.0%
EIM	83.4%	82.7%	81.6%	86.0%

EIM: Extra-intestinal manifestations; CD: Crohn’s disease; UC: Ulcerative colitis.

For scoring disease severity in UC, the Montreal requires to score the maximum disease severity ever experienced. Therefore, scoring S0 (meaning clinical remission) would be impossible. We therefore removed observers that scored an S0, and disease severity in UC was calculated again for gastroenterologists and gastroenterologists in training. The percentages of correct answers were 69% and 71%. IBD-nurses were not considered in this analysis because all scored an S0 once or more. Removing S0 for disease severity in UC led to a correct score of 77%, 56% and 56% for observers with less than 10 years, 10-20 years and more than 20 years of experience with IBD patient care.

Inter-observer agreement

The inter-observer agreement was excellent for diagnosis (κ = 0.96), CD location (κ = 0.82) and perianal disease (κ = 0.91). Age of onset (κ = 0.67) and upper gastrointestinal disease (κ = 0.62) were scored with a good inter-observer agreement. Disease severity was scored poorly (κ = 0.23) for UC. The additional clinical item, CD severity, was scored with moderate concordance (κ = 0.44). In total there were 19 EIMs in 12 case vignettes. The inter-observer agreement for occurrence of EIM, was good (κ = 0.68) (Table 5).

Table 5 Inter-rater agreement kappa for all items of all categories of Montreal classification.

Item Montreal classification	Overall kappa
Age of onset	0.67 (n = 28)
Diagnosis	0.96 (n = 28)
CD disease localization	0.82 (n = 28)
CD upper gastrointestinal	0.62 (n = 28)
CD perianal disease	0.91 (n = 28)
CD disease behavior	0.79 (n = 26)
CD severity (mild, moderate, severe)	0.44 (n = 24)
UC disease extent	0.65 (n = 21)
UC disease severity colitis	0.23 (n = 20)
EIM	0.68 (n = 28)

Observers were only included if they scored at least one item in all case-vignettes. EIM: Extra-intestinal manifestations; CD: Crohn’s disease; UC: Ulcerative colitis.

By removing all the observers that stated an S0 once or more, only 7 observers remained which led to a kappa of 0.57, resulting in moderate inter-observer agreement for severity in UC. Kappa was also calculated again for disease extent and disease severity in UC, but now for 30 observers with all the missing values being replaced by the correct answer as established by the “expert evaluation”. No significant differences in the inter-observer agreement for 30 and 20/21 observers scoring disease severity and disease extent in UC were found.

DISCUSSION

The aim of this study was to assess the validity of accurate phenotyping using the Montreal IBD classification system with 2 additional items (CD severity and EIM) for both CD and UC within the Netherlands.

According to our study, the Montreal is a reliable classification system for phenotypes in IBD, except for disease severity in UC. The assessment of disease severity for UC as described in the Montreal classification system is difficult in the case of retrospective chart reviews. Since severity in CD is not a classification item in the Montreal, we asked the observers to score CD severity based on their personal interpretation of the case vignettes. This resulted in a low consistency between observers, but this item was scored with higher concordance (with fewer instructions) than disease severity in UC.

Until now only limited data on the reliability and reproducibility of the Montreal classification is available. An Australian-New Zealand and United States study[18,19] found a good inter-observer agreement for CD, however the Australian-New Zealand study did not include the scoring of UC and neither study included an assessment of disease severity for both UC and CD. In our study the inter-observer agreement for diagnosis was excellent (κ = 0.96), which was comparable to the Australian-New Zealand study (κ = 0.82). The inter-observer agreement for age of onset was only “good” in our cohort (κ = 0.67) as compared to excellent in the Australian-New Zealand (κ = 0.84) and the US study (κ = 0.98). The observers in our cohort were better at scoring disease localization in CD, upper gastrointestinal involvement, perianal disease and disease behaviour in CD. Disease extent in UC was similarly scored in our cohort (κ = 0.65) as in the Australian-New Zealand study (κ = 0.67)[18,19].

Classifying disease severity in patients’ records (“real life”) is still a problem because of missing or unclear descriptions. Clinicians should strive to be complete and accurate in their medical reporting. A clearer definition of disease severity is needed because apparently there is no consensus between clinicians about mild, moderate or severe disease in (real life) patients. For disease severity there are several classification systems e.g., CD activity index[24] and the UC activity index[25] that assess disease severity by clinical symptoms, however these symptoms are present at a specific time point and cannot be assessed in a retrospective manner. The CD digestive damage score (Lemann score) is a measurement for cumulative structural bowel damage, assessed by scoring disease severity for damage location, severity, extent, progression and reversibility, diagnosed by image modalities and the history of surgical resections. Ultimately a prediction model gives a reflection of progressive and destructive disease course[26]. The Lemann score might be a good instrument for classifying disease severity.

Since IBD is a chronic disease with unpredictable disease behaviour, it is very important that clinicians can identify those individuals with a severe disease course, risk of side effects to therapy or those who would benefit from lifestyle or environmental changes. It is expected that molecular and/or pharmaco-genetic markers will play an increasing role in predicting disease course or response to medication in the future[27]. A good opportunity to predict individual disease behaviour is by linking their uniform phenotypic characteristics with our knowledge of the molecular basis of the disease. In IBD research an increasing number of biobanks are being set up worldwide allowing for linking molecular data to phenotypic data. To ensure high-quality data, validation of the Montreal classification is mandatory for these kinds of multicenter prospective data collections.

This Dutch validation study has a larger observer group than the previously mentioned studies. It is the first to include UC and CD disease severity, and to differentiate between professions. We found a good inter-observer agreement for diagnosis, localization, disease behaviour, disease extent and the occurrence of EIM. The reliability for assessment of disease severity for UC was poor, and moderate for the additional CD severity item. Optimal reporting of uniform phenotypes of patient cohorts is of utmost importance, especially in genetic and clinical research. Uniform phenotyping will ultimately allow for integration of clinical phenotypes with high-throughput–omics data (integration of genetic, expression or metagenomic data), which will increase our understanding of IBD pathogenesis, and allow for better patient stratification and classification.

ACKNOWLEDGMENTS

We would like to thank all gastroenterologists, residents and IBD nurses who participated in the study. Collaborators within the “Dutch Initiative on Crohn and Colitis”: van Dullemen H (Department of Gastroenterology and Hepatology, University Medical Centre Groningen, Groningen), Russel M (Department of Gastroenterology and Hepatology, Medisch Spectrum Twente, Enschede), van der Waaij LA, Thijs WJ (Department of Gastroenterology and Hepatology, Martini ziekenhuis, Groningen), van de Woude CJ (Department of Gastroenterology and Hepatology, Erasmus MC, Rotterdam), de Jong DJ, Hoentjen F (Department of Gastroenterology and Hepatology, University Medical Centre Nijmegen St Radboud, Nijmegen), ter Steege R (Department of Gastroenterology and Hepatology, Medical Centre Leeuwarden, Leeuwarden), de Boer KHN, van Bodegraven AA, Bouma G, (Department of Gastroenterology and Hepatology, Medical centre VU, Amsterdam), Pierik M (Department of Gastroenterology and Hepatology, University Medical Centre Maastricht, Maastricht), van der Meulen AE (Department of Gastroenterology and Hepatology, Leids University Medical Centre, Leiden), Oldenburg B (Department of Gastroenterology and Hepatology, University Medical Centre Utrecht, Utrecht), The Netherlands.

COMMENTS

Background

Inflammatory bowel disease (IBD) consisting of Crohn’s disease (CD) and ulcerative colitis (UC) is a heterogeneous disease, in which the pathogenesis is not fully understood. Multiple biological pathways have been implicated by for instance genetic studies, but the translation to the clinic is difficult partly because of a great variety in the clinical presentation. It is therefore very important that IBD sub-phenotypes are described in a consistent and reliable manner. The Montreal classification is a system to classify IBD phenotypes, but data on its reliability are scarce.

Research frontiers

The Montreal classification is widely used in research and clinical practice, but there is only limited data available on its reliability. This study validates the Montreal classification system for CD and UC within the Netherlands.

Innovations and breakthroughs

Only two studies assessed the reliability of the Montreal classification. One study did not include UC and both studies had only a small number of observers. The results in the current study, including a larger group of observers in both UC and CD, were similar for diagnosis. Age of onset was ”good” in our study compared to “excellent” in the other two studies. The observers were better at scoring disease localization in CD, upper gastrointestinal involvement, perianal disease and disease behaviour in CD. Disease extent in UC was scored similarly. This Dutch validation study has a larger observer group than the previously studies. It is the first to include UC and CD disease severity, and to differentiate between professions. The reliability for assessment of disease severity for UC was poor, and moderate for CD severity (which is not part of the Montreal system, but independently defined). In addition Extra Intestinal Manifestations were scored with a good inter-observer agreement. The use of additional parameters to define and classify disease severity is needed.

Applications

This study highlights the need for accurate medical reporting and a reliable classification system. Validation of the Montreal classification is necessary to ensure high-quality data in multicentre prospective data collections. These phenotypes will be linked to molecular data, with the prospective of finding molecular and/or pharmaco-genetic markers. Eventually these markers will help clinicians to predict a patient’s disease course or response to medication in the future.

Peer review

The authors validated the Montreal classification system for CD and UC within the Netherlands. An expert panel first assessed 20 case vignettes separately, developed an evaluation for all Montreal items, and this was considered the correct answer. A score for CD severity and extra-intestinal manifestations was added. Thirty observers with different professions and experience level with IBD patient care scored all the items. A good to excellent inter-observer agreement for all Montreal items except for disease severity in UC was found. The study is well conducted and data well presented. Reliability and reproducibility of the Montreal classification in the “real life” are warranted to be assessed with similar studies in other countries.