Subtle skills: Using objective structured clinical examinations to assess gastroenterology fellow performance in system based practice milestones

ARTICLE HIGHLIGHTS

Research background

Medical education assessment in the United States is currently based on six competencies as defined by the Accreditation Council for Graduate Medical Education. Gastroenterology (GI) training programs must assess trainees according in all six competencies as part of formative feedback. The system based practice (SBP) competencies require trainees to effectively recognize and navigate the larger healthcare system for optimal patient care. These milestones may be considered more subtle skills to master than other concrete milestones such as patient care and medical knowledge.

Research motivation

SBP milestones are difficult to observe and assess in daily clinical encounters between patients and trainees. Therefore, alterative objective activities may be necessary to adequately assess achievement in SBP. Simulation based medical education such as the Objective Structured Clinical Examinations (OSCEs) are now a standard methodology for assessing clinical skill and knowledge in medical education. OSCEs have not been previously used to assess SBP milestones specifically.

Research objectives

The main objective of this research study was to create and then evaluate OSCEs as an effective assessment tool for the evaluation of SBP milestones. We aimed to see if new clinical scenarios commonly encountered by GI trainees would be useful in this assessment. We also sought to evaluate how trainees felt about the experience.

Research methods

We developed four cases to help assess the Accreditation Council for Graduate Medical Education milestones with a focus on SBP. Trainees went through these four simulations with standardized patients and were evaluated by faculty experts using standardized checklists. Their performance from the checklists were aggregated and used to produce a scorecard which was sent to program directors at the conclusion of the OSCE. The trainees were then given direct feedback from the standardized patients and the faculty observer. Finally, the trainees were asked to complete a survey on the experience.

Research results

We ran three OSCE sessions involving 26 GI trainees. Scorecards indicated that, on average, trainees scored lower on SBP milestones than on other milestones categories. We identified and reported discrepancies between how well trainees believed they achieved objectives, and how they were rated by the standardized patients and faculty observers. Overall, trainees reflected positively on the experience in the post participation survey. They universally felt that the immediate feedback was useful and would improve their clinical skills. All participants stated they would recommend this OSCE as an assessment and training tool.

Research conclusions

In this study we demonstrated that OSCEs may be utilized to assess SBP milestones in an objective manner. Since SBP milestones may be difficult to assess in day-to-day activities in the hospital or clinic setting, training programs may want to utilize this type of standardized case-based simulation for assessment. Likewise, trainees reflected positively on the experience and felt they would incorporate feedback into their daily practice.

Research perspectives

Future studies are needed to assess if OSCEs may be useful teaching tools for SBP milestones. This would require repeat assessment with the same OSCE at the GI fellows’ completion of training and comparison of this group to a group who did not participate in the initial OSCE in their first year.