Using motion capture to assess colonoscopy experience level

Abstract

AIM: To study technical skills of colonoscopists using a Microsoft Kinect™ for motion analysis to develop a tool to guide colonoscopy education.

RESULTS: Ten experienced endoscopists (gastroenterologists, n = 2; colorectal surgeons, n = 8) and 11 novices participated in the study. A Microsoft Kinect™ recorded the movements of the participants during the insertion of the colonoscope. We used a modified script from Microsoft to record skeletal data. Data were saved and later transferred to MatLab for analysis and the calculation of statistics. The test was performed on a physical model, specifically the “Kagaku Colonoscope Training Model” (Kyoto Kagaku Co. Ltd, Kyoto, Japan). After the introduction to the scope and colonoscopy model, the test was performed. Seven metrics were analyzed to find discriminative motion patterns between the novice and experienced endoscopists: hand distance from gurney, number of times the right hand was used to control the small wheel of the colonoscope, angulation of elbows, position of hands in relation to body posture, angulation of body posture in relation to the anus, mean distance between the hands and percentage of time the hands were approximated to each other.

RESULTS: Four of the seven metrics showed discriminatory ability: mean distance between hands [45 cm for experienced endoscopists (SD 2) vs 37 cm for novice endoscopists (SD 6)], percentage of time in which the two hands were within 25 cm of each other [5% for experienced endoscopists (SD 4) vs 12% for novice endoscopists (SD 9)], the level of the right hand below the sighting line (z-axis) (25 cm for experienced endoscopists vs 36 cm for novice endoscopists, P < 0.05) and the level of the left hand below the z-axis (6 cm for experienced endoscopists vs 15 cm for novice endoscopists, P < 0.05). By plotting the distributions of the percentages for each group, we determined the best discriminatory value between the groups. A pass score was set at the intersection of the distributions, and the consequences of the standard were explored for each test. By using the contrasting group method, we showed a discriminatory value of Z = 1.51 to be the pass/fail value of the data showing discriminatory ability. The pass score allowed all ten experienced endoscopists as well as five novice endoscopists to pass the test.

CONCLUSION: Identified metrics can be used to discriminate between experienced and novice endoscopists and to provide non-biased feedback. Whether it is possible to use this tool to train novices in a clinical setting requires further study.

Key Words: Colonoscopy, Assessment, Simulation, Motion-capture, Motion-analysis

Core tip: Motion capture for motion analysis can be used to discriminate between experienced and novice performers of colonoscopy. We analyzed the motion patterns of the technical procedure of inserting the colonoscope from anus to cecum in a simulation set-up. The technical differences between novice and experienced endoscopists observed in this study are important because they can help shape skills that will lead to competence in colonoscopy. In the future, this technique might be useful in the training and education of future colonoscopists in a clinical setting.

INTRODUCTION

Screening programs for colorectal cancer and concern for patient safety have increased the importance of training endoscopists for competency in colonoscopy. The workload of existing endoscopy units is often high, with units performing an increasing number of endoscopies in addition to supervising, training and instructing future endoscopists. The quality of colonoscopies was questioned in an investigation of 68 endoscopy units in the United Kingdom with a cecal intubation rate of 56%. Only 17% of the endoscopists had supervised training during their introduction to colonoscopy, and only 33% attended a colonoscopy course[1]. In Denmark, colonoscopy competence is solely based upon educational level, such as having a specialized degree in gastroenterology and/or surgery. The number of colonoscopies performed has conventionally defined technical competence in colonoscopy, and a threshold number of up to 275-500 has been suggested[2,3]. Previous methods for assessing skills in colonoscopy have been based upon subjective expert ratings, and previous tools have been based upon the procedural endpoints, time to cecum, depth of insertion or complication rate of therapeutic procedures[4-6]. No automatic assessment tools have been developed, although it has been noted that an optimal assessment tool in surgical skills should be based upon objective and structured criteria[7]. However, some progress has been made with regards to a benchmarked curriculum for virtual reality colonoscopy simulators[8].

Colonoscopy is very much dependent upon manual dexterity, correct stance and hand-eye coordination. The correct way to perform a colonoscopy is greatly debated, and some variations have been noted among experts advocating for the single-handed technique[9-13]. Video imaging has been found to be valuable in assessing competence in surgical skills[7], and video-based judgment of the handling of endoscopes is one of the main basic colonoscopy procedures tested with the “Direct Observation of Procedural Skills Score (DOPS)”[14].

It is a well-known but unproven fact among professional gastroenterologists that the stance of the performer shows the level of competence. Defining a “correct” basic handling in colonoscopy is not easy, but certain facts are clear: when adhering to the single-handed technique[9,10,13], the procedure should be conducted in a relaxed fashion with a straight scope, with minimal discomfort for the endoscopist as well as for the patient. Concerning movements of the tip of the scope, torque steering and steering with the small wheel of the colonoscope has very little effect when the tip is angulated[15].

The correct single-handed technique has been tested by video imaging with an objective structured video assessment tool where instrument grip, tip steering, and letting go of the instrumental shaft all were found to correlate with the competence level of the endoscopist[9]. The same basic colonoscopy metrics were found to improve significantly in an intensive training program[10].

Motion analysis has been used to teach correct skiing technique in downhill skiing to prevent injuries[16] and to correct golf swings[17]. Motion analysis can also be used to determine joint movements in different procedures, such as walking in high-heeled shoes to explain the occurrence of gait related diseases[18]. We speculate that motion analysis could also be used to teach correct movements in colonoscopy performance, if correct movements can be identified and verified.

In medicine, motion analysis has been used to identify skilled performers in emergent endotracheal intubation in physical models[19] as well as in an infant airway trainer[20]. Previously, motion analysis demanded the use of sensors on the body, making analysis of movements a costly process. In 2012, Microsoft launched the Microsoft Kinect (MS Kinect) system for Windows, designed for the XBOX gaming platform. The MS Kinect camera has become increasingly popular in many areas aside from entertainment. It provides a quick, cheap and easy way of analyzing position and mapping 3-dimensional (3D) pose data, providing skeletal movement tracking. The accuracy of the system as a peripheral device measuring 3D depth is estimated to be 1-4 cm at a range of 1-4 m[21].

The aim of this study was to use the MS Kinect system to automatically record and analyze the components of the basic techniques of endoscopists (experienced endoscopists and novices), selecting discriminatory metrics to develop a tool which can monitor competence in endoscopists and guide education in a non-biased way.

MATERIALS AND METHODS

Participants

Ten consultants experienced in endoscopy (gastroenterologists, n = 2; colorectal surgeons, n = 8) and eleven novices participated in the study. Novices were recruited from fellows in gastroenterology and gastroenterological surgery during their first or second year of fellowship and had very limited experience in colonoscopy (median 0 procedures, range 0-2). The experienced group had an average of 18.3 years of endoscopic experience (range 7-30) and had performed a median of 2000 (range 350-4000) colonoscopies, including a median of 75 (range 0-450) colonoscopies within the last year (Table 1). All participants were recruited and tested between November 2012 and March 2013.

Table 1 Demographic details on the participation physicians.

	Sex		Age, yr		Colonoscopy experience		Colonoscopies performed in past 12 mo
	Male	Female	Median	Range	Median	Range	Median	Range
Novices (n = 11)	4	7	32	(28-37)	0	(0-2)	0	(0-2)
Experienced endoscopists (n = 10)
	8	2	55	(42-63)	2000	(350-4000)	52.5	(0-450)

Study set-up

We used a virtual reality simulator for the introduction to the functions of the colonoscope (GI Mentor, Simbionix Corporation, United States). For the test, we used the Kagaku Colonoscope Training Model (Kyoto Kagaku Co. Ltd, Kyoto, Japan) and a colonoscope (Olympus™ CF 180AL) with air insufflation, suction-water knobs and a scope guide from Olympus. The physical model consisted of a flexible rubber “colon” tube inside of a life-size mannequin. The colon tube could be adjusted into 6 different positions using Velcro-strips and rubber bands. Tasks 1 (test introduction) and 3 (test) were chosen for this study. Task 1 was a technically easy procedure, whereas task 3 was more challenging, with a loop formation in the sigmoid colon. The test setting was a fixed set up in a dedicated room and was not changed during the study period.

Data collection

Testing was conducted in a medical simulation center. The novices were introduced to the functions of the colonoscope including handling the colonoscope, using the controls (i.e., dials, insufflation, suction, and water), manipulating the endoscope tip, and torque steering in a virtual-reality simulator. The training session was 1 hour. All participants were asked to fill out a brief questionnaire, which included demographics, such as gender, age, years of endoscopic experience and the number of colonoscopies performed during the past 12 mo. A letter of acceptance of participation was handed out, signed and returned prior to the test. Participants were instructed to treat the model as if it were a real patient. The participants were informed that their movements would be recorded but were given no details of which metrics would be measured. They were given a maximum of 15 min to perform the procedure. A Kinect camera recorded the movements of the participant during insertion of the scope. Recording was initiated at intubation of the rectum and stopped when the scope reached the cecum.

Microsoft Kinect

The Kinect camera consists of a series of external sensors for image capturing and is motorized to make the box adjustable. The sensors are able to detect movements without requiring the participants to wear tracking sensors.

The Kinect creates a map of reflections from the person in the scene, which can be used for skeleton analysis. We used a modified script from Microsoft for recording skeletal XYZ-data. Data were saved and later transferred to MatLab^® 2012a for analysis and the calculation of statistics. The range of the Kinect system for depth analysis is 1.2-3.5 m; the test set-up was adjusted to this distance. The box was placed above the endoscopy screen pointing at the chest of the participants, producing an image of the upper part of the body (Figure 1). The setup was not adjusted according to the height of the participants but all participants were within the range of the camera. The coordinates of the Kinect system are shown in Figure 2. The Z-axis was pointed at the chest, and the X-axis was longitudinal to the gurney.

Figure 1 The simulator set-up. The Kinect was placed behind the two screens.

Figure 2 3D coordinates of the Kinect system.

Measured metrics

Validated tools, such as DOPS, suggest metrics related to basic techniques, such as the correct use of the left and right hands and understanding looping and cecal intubation[10]. However, there is no defined correct way of handling the scope during insertion. We chose a number of measures, skeletal angles and joint movements we thought appropriate to the procedures based on the literature[15].

Metrics used for motion analysis were the distance of the right and left hands from the gurney, the number of times the right hand was used to control the small wheel (distance between hands less than 25 cm), the angulation of the right and left elbows, the position of the hands in relation to the torso, the angulation of body posture in relation to the orifice, the mean distance of the hands from each other during the procedure and the percentage of time the hands were approximated. Measurements were conducted at 30 frames per second, and for each person, a mean of values was calculated in relation to coordinates of the MS Kinect.

Statistical analysis

All variables showing statistically significant differences between novices and experienced operators were identified using independent sample t-tests. The means and standard deviations of the experienced group were used to transform variables with discriminative ability into Z-scores. These Z-scores, when discriminatory, were intended to be averaged into a single score for each participant; i.e., a score of 2 indicated that the participant, on average, was two standard deviations off the “gold standard”, defined by the mean of experienced operators. A pass-fail standard was set using the contrasting groups method to further explore the ability of this aggregated score to discriminate between the two groups.

An independent samples t-test, Mann-Whitney test and Levene’s test for equality of variance were performed to compare the performances of the two groups. Spearman’s rho was used for non-parametric correlation analysis.

Statistical analysis was performed using a statistical software package (r-project.org, R v 3.0.2; MatLab^® 2012a). Differences were considered to be statistically significant for P values < 0.05.

RESULTS

The two groups differed in gender (experienced endoscopists: 2 females, 8 males; novices: 7 females, and 4 males) (NS, Fisher’s test) and age. For details see Table 1.

Only four of our seven metrics showed discriminatory ability between novice endoscopists and experienced endoscopists in the t-test (test of mean), and only two showed a difference in group-variance.

Three metrics: “angulation of right elbow,”“angulation of left elbow” and “angulation of shoulders to the anus of physical model” did not show discriminatory ability.

We found discriminatory values for the following metrics: “level of left hand“ and “level of right hand” below the z-axis (experienced: 6 cm for the left hand; novices: 15 cm for the left hand, P < 0.05; experienced: 25 cm for the right hand; novices: 36 cm for the right hand, P < 0.05). The difference subsided when correcting for the height of the person by analyzing the distance of the left and right hands from the left and right shoulders accordingly (31 cm vs 32 cm, NS). The two groups differed in height when shoulder height was analyzed. For details see Table 2.

Table 2 Metrics analysed for discriminatory ability.

Kinect metrics	Experienced endoscopist	Novice	Levene’s test	P value
Percentage of time with hands closer than 25 cm (%)	7	23	0.048	0.02
Distance between hands (cm)	45	37	0.09	0.01
Angle of shoulders (degrees)	17	20	0.95	0.38
Right hand below z-line (cm)	25	36	0.95	0.01
Mean distance shoulder-hand (cm)	31	31
Left hand above z-line (cm)	6	15	0.03	0.005
Mean distance: shoulder-hand (cm)	32	31
Left elbow (degrees)	91	92	0.86	0.81
Right elbow (degrees)	144	140	0.55	0.55
Height participants (cm)	39	30	0.08	0.02
Compared to coordinates

Two metrics showed discriminatory ability: “mean distance between hands” [experienced: 45 cm (SD 2); novices: 37 cm (SD 6)] and “percentage of time with hands less than 25 cm apart” [experienced: 5% (SD 4); novices: 12% (SD 9)].

Absolute Z-scores (average standard deviations from the “gold standard”) were calculated for each of the discriminatory metrics and summed to a total mean Z-score for each participant. For details, see Table 3.

Table 3 Experience correlated to pass score and cecal intubation rate.

Test person	Competence	Total colonoscopies	Colonoscopies in last year	Z mean	Z-score passed	Cecal intubation
1	Experienced Endoscopist	3000	300	1.15	Yes	Yes
2	Experienced endoscopist	400	10	1.32	Yes	Yes
3	Experienced endoscopist	2000	0	1.25	Yes	Yes
4	Experienced endoscopist	1000	17	0.08	Yes	Yes
5	Experienced endoscopist	1700	150	0.30	Yes	Yes
6	Experienced endoscopist	4000	232	0.70	Yes	Yes
7	Experienced endoscopist	3000	14	0.78	Yes	Yes
8	Experienced endoscopist	2000	450	0.76	Yes	Yes
9	Experienced endoscopist	2000	75	0.82	Yes	Yes
10	Experienced endoscopist	350	30	0.73	Yes	Yes
11	Novice	0	0	1.54	No	No
12	Novice	0	0	1.48	Yes	No
13	Novice	0	0	1.28	Yes	Yes
14	Novice	0	0	2.26	No	Yes
15	Novice	1	1	6.19	No	Yes
16	Novice	0	0	0.27	Yes	Yes
17	Novice	0	0	6.50	No	No
18	Novice	0	0	5.25	No	Yes
19	Novice	0	0	1.17	Yes	No
20	Novice	0	0	3.25	No	Yes
21	Novice	2	2	0.65	Yes	No

By plotting the distributions for each group, we could determine the best discriminatory value between the groups. The pass score was set at the intersection of the distributions, and the consequence of the standard was explored for each test. By using this contrasting group method, we showed a discriminatory value of Z = 1.51 to be the pass score. The pass score allowed all of the experienced as well as the five novices to pass the test (Figure 3).

Figure 3 Establishing a pass/fail standard using the contrasting-groups method. The distribution of scores of novices (dotted line) and experienced (solid line). The pass score (Z-1,51) is set at the intersection of the score distributions of the two groups.

Nine of ten of the experienced operators reached the cecum within 15 min (the cut-off time), as did seven out of 11 novices (64%). Comparing Z-scores, pass vs cecal intubation ability’s positive PV for a passing Z-score was found to be 80%, while the negative PV for cecal intubation for a failed Z-score was 33%.

There was no difference in the novices who reached the cecum and the novices who did not reach the cecum in regards to percentage of time when the hands were less than 25 cm apart (14%). When measuring the distance between hands, there were no group differences among the novices (37 cm).

Time to reach the cecum was measured. A positive correlation Rho was found for “percentage of time with hands too close,”“hand distance” and “cecal intubation time“ (Rho = -0.58; P = 0.005 and Rho = 0.60; P = 0.004).

There was no correlation between the aggregated Z-score and time to cecum (Rho = 0.40; NS). However, when analyzing the correlation between the numbers of colonoscopies performed in the past year (Log routine) and the aggregated Z-score, a correlation was found (Rho = -0.54; P = 0.01).

DISCUSSION

Our data showed a difference in motion patterns of the colonoscopy procedure when comparing novices to experienced endoscopists. By using the MS Kinect, we could identify a common stance used by experienced endoscopists. Our data made it possible to note how the novices handled the colonoscope as they tried to control the tip. We found that excessive correction movements halted the progress of the colonoscope.

We found no correlation between the total score and time to cecum, which indicates that progression does not entirely depend on manual handling of the control dials of the colonoscope. We did, however, find a correlation between current routine (past years experience) and the Z-score, suggesting that other aspects of the steering process must be important. The reason for this might be the ability to keep the scope straight. Having a straight scope inside of the patient depends on a scope without loops and bends outside of the patient and a slack loop between hands. We found that the distance between hands was significantly wider in the experienced group, which might make “torque steering” easier[15]. The MS Kinect could not record the motion pattern of torque steering.

Assessment tools based on tri-split video monitoring and evaluation by trained judges have been made and validated by others[9,22]. DOPS assesses different domains of the colonoscopy procedure: basic handling of the colonoscope, such as “grip of instrument with accurate finger/thumb,”“control of wheels,”“tip steering” and “manipulation of the shaft”.

The metric “distance between hands” was a surrogate measure of keeping the scope straight. We considered our data for “percentage of time with hands too close” to be a surrogate measure of using both hands on the control dials (distance less than 25 cm). The DOPS metric “incorrect use of hand grip” was found to be one of the most significant metrics, showing improvement with a week of intensive training[10]. Using both hands on the steering wheel stopped the progression of the scope. We demonstrated an unbiased measure using “percentage of time with hands too close” to assess this parameter.

The ability to reach the control dials with the thumb can be challenging because the grip of the standard colonoscope has been developed for large hands. Endoscopists compensate by bringing the right hand up to help adjust the control dials. Cohen and colleges made an informative survey and found that 23% of fellow gastroenterologists felt that they had some difficulties in reaching and manipulating the horizontal control dial (small wheel). A considerable portion of the female fellows reported that their hands were too small to reach the horizontal control dial (40%), and nearly 80% reported that their hand size affected their ability to learn endoscopy[23]. The “left-hand grip”[12] and the “pinkie maneuver”[11] are methods to maneuver the control dials to compensate for this challenge. However, both methods result in a bended scope, which might negatively affect progression. Traditionally, ergonomic concerns have not played a large role in teaching the colonoscopy procedure. Tenosynovitis of the left thumb associated with overuse during endoscopy has been described (DeQuervain’s syndrome), and this problem has increased with an increased number of procedures performed per endoscopist[24]. A solution to both problems could be the introduction of the scope-dock system developed for the ERCP procedure in colonoscopy. A docking system would allow for free handling of control dials simultaneous to torque steering and advancing the tip of the scope.

The aggregated score of our two significant metrics has demonstrated the ability to differentiate between experienced and novice endoscopists, and the pass score had a predictive value of 80% for reaching the cecum. Current routine in colonoscopy was highly correlated to the metrics with discriminatory ability. The combined Z-score, with a correlation coefficient of 0.54, made the Z-score an objective assessment tool to predict the ability to reach the cecum in a routine colonoscopy.

The advantage of the MS Kinect system is that it provides information on the motion pattern with an inexpensive and simple method[25]. The method has been found to be accurate in skeletal tracking of upper body movements as well as for joint measurements with an accuracy of 1-2 cm for a distance of up to 4 m[21].

Our assessment tool provides information that emphasizes that training should focus on handling the control dials, especially the small horizontal control dial, with the left hand and keeping a straight scope with a distance between the hands. Our data show that it was possible to recognize the motion pattern of experienced endoscopists by external motion capture and to distinguish the experienced from the novices in an objective way. We found a correlation between the current routine and the metrics with discriminatory ability, suggesting that correcting the stance might be relevant, not only in novices.

Whether it is possible to use this information from stance recognition and pose enforcement to train novices in a clinical setting remains to be determined, but this unbiased tool does provide useful information to guide teaching. Our tool might help colonoscopy trainees gain competence in the technical part of the colonoscopy procedure, which is the difficult and strenuous part of the procedure for the endoscopists, as well as for the patient.

COMMENTS

Background

Colonoscopy is a technically challenging procedure that requires the development of advanced psychomotor skills. During the past decade there has been increased focus on structured medical education and the assessment of procedural skills. Competence in colonoscopy has conventionally been defined as the number of colonoscopies performed or as rates of successful intubation of the cecum. Only a few attempts have been made to measure technical competence in colonoscopy. This study brings new objective information for learning technical skills of colonoscopy using motion capture.

Research frontiers

Motion analysis has been used in sports for decades. Using motion capture in colonoscopy provides new, useful, objective information to guide novices and those teaching colonoscopy so that competence in this procedure can be measured.

Innovations and breakthroughs

Previously, motion analysis required the use of sensors on the body, making the analysis of movements a costly process. In 2012, Microsoft launched the Microsoft Kinect (MS Kinect) system for Windows, designed for a gaming platform. The MS Kinect camera has become increasingly popular in many areas aside from entertainment and has been used in other areas of medical education.

Applications

Motion capture seems useful for obtaining objective measures to guide colonoscopy education and can be used in a clinical setting to provide unbiased feedback to guide novices and those teaching colonoscopy so that competence in this procedure measured by other gauges can be achieved.

Terminology

Motion capture: The process of recording the movement of people; Assessment tool: A method to assess performance; DOPS: Direct Observation of Procedural Skills Score is a validated assessment tool for the colonoscopy procedure.

Peer review

The authors address an important and relevant topic in today’s teaching and learning of endoscopy and bring new objective information for learning its technical skills. The paper is very interesting.