Ileal pouch anal anastomosis is an important part of colorectal surgery training. However, decreasing case numbers create fewer chances to teach the knowledge and skills to trainees.

The purpose of this study is to define the components and format of an adjunctive curriculum for teaching colon and rectal surgery residents ileal pouch anal anastomosis. More specifically, it aims to create expert consensus on the essential and unique elements of a curriculum, determine which alternative strategies will be acceptable, and identify methods of assessment to evaluate the effectiveness of the curriculum.

This was a modified Delphi Consensus study. Semistructured interviews with thematic analysis were used to develop the initial round of the survey.

Interviews, surveys, and online consensus conference.

Faculty who were at North American Accreditation Council for Graduate Medical Education colorectal residencies, and participants from each round were invited to subsequent rounds.

Three rounds of surveys were used to select high-priority items for inclusion in the curriculum, educational strategies, and assessments. This was followed by a consensus conference to clarify prioritization and acceptable options for teaching and assessment. Participants from the third round of surveys, as well as Program Directors from colorectal residencies, were invited to the consensus conference.

Twelve semistructured interviews defined the educational needs of colorectal residents, the appropriateness of alternative educational strategies, and concerns about case minimums as an assessment of readiness for practice for this procedure. The needs were divided into preoperative, intraoperative, and postoperative phases, with knowledge and skills in each phase. The resulting survey was completed by 85 colorectal surgeons in round 1, 43 in round 2, and 38 in round 3. These results were subsequently summarized by a team of 9 participants in a final consensus conference, where the participants made suggestions on the recommended educational strategies for each educational need.

Inclusion of colorectal surgeons involved in training programs only, bias in respondents, and participation rate.

This shared understanding of educational needs for colorectal residents that includes knowledge, judgment, and technical skills is the first step toward developing a curriculum to teach ileal pouch anal anastomosis. A collaborative effort to create the necessary curriculum is underway. See Video Abstract .

ANTECEDENTES:La anastomosis ileoanal con reservorio (IPAA) es una parte importante de la formación en cirugía colorrectal. Sin embargo, la disminución en la cantidad de casos genera menos oportunidades para enseñar los conocimientos y las habilidades a los residentes.OBJETIVO:El propósito de este estudio es definir los componentes y el formato de un plan de estudios complementario para la enseñanza de la IPAA a los residentes de cirugía de colon y recto. Más específicamente, apunta a crear un consenso de expertos sobre los elementos esenciales y únicos de este plan de estudios, determinar qué estrategias alternativas serán aceptables e identificar métodos de evaluación para evaluar la efectividad del plan de estudios.DISEÑO:Este fue un estudio de consenso Delphi modificado. Se utilizaron entrevistas semiestructuradas con análisis temático para desarrollar la ronda inicial de la encuesta.ESCENARIO/PARTICIPANTES:Se invitó a los profesores de las residencias colorrectales de la ACGME de América del Norte a participar en las encuestas electrónicas, y los participantes de cada ronda fueron invitados a las rondas posteriores.MEDIDAS DE RESULTADOS:Se utilizaron tres rondas de encuestas para seleccionar elementos de alta prioridad para su inclusión en el plan de estudios, las estrategias educativas y las evaluaciones. A esto le siguió una conferencia de consenso para aclarar la priorización y las opciones aceptables para la enseñanza y la evaluación. Los participantes de la tercera ronda de encuestas, así como los directores de programas de residencias colorrectales, fueron invitados a la conferencia de consenso.RESULTADOS:Doce entrevistas semiestructuradas definieron las necesidades educativas de los residentes colorrectales, la idoneidad de las estrategias educativas alternativas y las preocupaciones sobre los mínimos de casos como una evaluación de la preparación para la práctica de este procedimiento. Las necesidades se dividieron en fases preoperatorias, intraoperatorias y posoperatorias con conocimientos y habilidades en cada una. La encuesta resultante fue completada por 85 cirujanos colorrectales en la Ronda 1, 43 en la Ronda 2 y 38 en la Ronda 3. Estos resultados fueron resumidos posteriormente por un equipo de 9 participantes en una conferencia de consenso final donde los participantes hicieron sugerencias sobre las estrategias educativas recomendadas para cada necesidad educativa.LIMITACIONES:Inclusión únicamente de cirujanos colorrectales involucrados en programas de entrenamiento; sesgo en los encuestados; tasa de participación.CONCLUSIONES:Esta comprensión compartida de las necesidades educativas de los residentes colorrectales que incluye conocimientos, criterio y habilidades técnicas es el primer paso hacia el desarrollo de un plan de estudios para enseñar IPAA. Se está llevando a cabo un esfuerzo colaborativo para crear el plan de estudios necesario. (Traducción-Dr. Jorge Silva Velazco ).

Perineal proctectomy is a complex procedure that requires advanced skills. Currently, there are no simulators for training in this procedure. As part of our objective of developing a virtual reality simulator, our goal was to develop and validate task-specific metrics for the assessment of performance for this procedure. We conducted a three-phase study to establish task-specific metrics, obtain expert consensus on the appropriateness of the developed metrics, and establish the discriminant validity of the developed metrics.

In phase I, we utilized hierarchical task analysis to formulate the metrics. In phase II, a survey involving expert colorectal surgeons determined the significance of the developed metrics. Phase III was aimed at establishing the discriminant validity for novices (PGY1-3) and experts (PGY4-5 and faculty). They performed a perineal proctectomy on a rectal prolapse model. Video recordings were independently assessed by two raters using global ratings and task-specific metrics for the procedure. Total scores for both metrics were computed and analyzed using the Kruskal-Wallis test. A Mann-Whitney U test with Benjamini-Hochberg correction was used to evaluate between-group differences. Spearman's rank correlation coefficient was computed to assess the correlation between global and task-specific scores.

In phase II, a total of 23 colorectal surgeons were recruited and consensus was obtained on all the task-specific metrics. In phase III, participants (n = 22) included novices (n = 15) and experts (n = 7). There was a strong positive correlation between the global and task-specific scores (rs = 0.86; P < 0.001). Significant between-group differences were detected for both global (χ2 = 15.38; P < 0.001; df = 2) and task-specific (χ2 = 11.38; P = 0.003; df = 2) scores.

Using a biotissue rectal prolapse model, this study documented high IRR and significant discriminant validity evidence in support of video-based assessment using task-specific metrics.

The study aims to develop a data-driven methodology to assess bone drilling in preparation for future clinical trials in residency training. The existing assessment methods are either subjective or do not consider the interdependence among individual skill factors, such as time and accuracy. This study uses quantitative data and radar plots to visualize the balance of the selected skill factors.

In the experiment, straight vertical drilling was assessed across 3 skill levels: expert surgeons (N = 10), intermediate residents (postgraduate year-2-5, N = 5), and novice residents (postgraduate year-1, N = 10). Motion and force were measured for each drilling trial, and data from multiple trials were then converted into 5 performance indicators, including overshoot, drilling time, overshoot consistency, time consistency, and force fluctuation. Each indicator was then scored between 0 and 10, with 10 being the best, and plotted into a radar plot.

Statistical difference (p < 0.05) was confirmed among 3 skill levels in force, time, and overshoot data. The radar plots revealed that the novice group exhibited the most distorted pentagons compared with the well-formed pentagons observed in the case of expert participants. The intermediate group showed slight distortion that was between the expert and novice groups.

This research shows the utility of radar plots in drilling assessment in a comprehensive manner and lays the groundwork for a data-driven training scheme to prepare novice residents for clinical practice.

Surgery is constantly evolving with the assistance of rapidly developing novel technology. Eye-tracking devices provide opportunities to monitor the acquisition of surgical skills, gain insight into performance, and enhance surgical practice. The aim of this review was to consolidate the available evidence for the use of eye-tracking in the surgical disciplines.

A systematic literature review was conducted in accordance with PRISMA guidelines. A search of OVID Medline, EMBASE, Cochrane library, Scopus, and Science Direct was conducted January 2000 until December 2022. Studies involving eye-tracking in surgical training, assessment and technical innovation were included in the review. Non-surgical procedures, animal studies, and studies not involving surgical participants were excluded from the review.

The search returned a total of 12 054 articles, 80 of which were included in the final analysis and review. Seventeen studies involved eye-tracking in surgical training, 48 surgical assessment, and 20 were focussing on technical aspects of this technology. Twenty-six different eye-tracking devices were used in the included studies. Metrics such as the number of fixations, duration of fixations, dwell time, and cognitive workload were able to differentiate between novice and expert performance. Eight studies demonstrated the effectiveness of gaze-training for improving surgical skill.

The current literature shows a broad range of utility for a variety of eye-tracking devices in surgery. There remains a lack of standardization for metric parameters and gaze analysis techniques. Further research is required to validate its use to establish reliability and create uniform practices.

Distractions during surgical procedures are associated with team inefficiency and medical error. Little is published about the healthcare provider's perception of distraction and its adverse impact in the operating room. We aim to explore the perception of the operating room team on multiple distractions during surgical procedures.

A 26-question survey was administered to surgeons, anesthesia team members, nurses, and scrub technicians at our institution. Respondents were asked to identify and rank multiple distractions and indicate how each distraction might affect the flow of surgery.

There was 160 responders for a response rate of 19.18% (160/834), of which 71 (44.1%) male and 82 (50.9%) female, 48 (29.8%) surgeons, 59 (36.6%) anesthesiologists, Certified Registered Nurse Anesthetists (CRNA), and 53 (32.9%) OR nurses and scrub technicians. Responders were classified into a junior group (< 10 years of experience) and a senior group (≥ 10 years). Auditory distraction followed by equipment were the most distracting factors in the operating room. All potential auditory distractions in this survey were associated with higher percentage of certain level of negative impact on the flow of surgery except for music. The top 5 distractors belonged to equipment and environment categories. Phone calls/ pagers/ beepers and case relevant communications were consistently among the top 5 most common distractors. Case relevant communications, music, teaching, and consultation were the top 4 most perceived positive impact on the flow of surgery. Distractors with higher levels of "bothersome" rating appeared to associate with a higher level of perceived negative impact on the flow of surgery. Vision was the least distracting factor and appeared to cause minimal positive impact on the flow of surgery.

To our knowledge, this is the first survey studying perception of surgery, anesthesia, and OR staff on various distractions in the operating room. Fewer unnecessary distractions might improve the flow of surgery, improve OR teamwork, and potentially improve patient outcomes.

Assessing performance automatically in a virtual reality trainer or from recorded videos is advantageous but needs validated objective metrics. The purpose of this study is to obtain expert consensus and validate task-specific metrics developed for assessing performance in double-layered end-to-end anastomosis.

Subjects were recruited into expert (PGY 4-5, colorectal surgery residents, and attendings) and novice (PGY 1-3) groups. Weighted average scores of experts for each metric item, completion time, and the total scores computed using global and task-specific metrics were computed for assessment.

A total of 43 expert surgeons rated our task-specific metric items with weighted averages ranging from 3.33 to 4.5 on a 5-point Likert scale. A total of 20 subjects (10 novices and 10 experts) participated in validation study. The novice group completed the task significantly more slowly than the experienced group (37.67 ± 7.09 vs 25.47 ± 7.82 min, p = 0.001). In addition, both the global rating scale (23.47 ± 4.28 vs 28.3 ± 3.85, p = 0.016) and the task-specific metrics showed a significant difference in performance between the two groups (38.77 ± 2.83 vs 42.58 ± 4.56 p = 0.027) following partial least-squares (PLS) regression. Furthermore, PLS regression showed that only two metric items (Stay suture tension and Tool handling) could reliably differentiate the performance between the groups (20.41 ± 2.42 vs 24.28 ± 4.09 vs, p = 0.037).

Our study shows that our task-specific metrics have significant discriminant validity and can be used to evaluate the technical skills for this procedure.

Task-specific metrics facilitate the assessment of surgeon performance. This 3-phased study was designed to (1) develop task-specific metrics for stapled small bowel anastomosis, (2) obtain expert consensus on the appropriateness of the developed metrics, and (3) establish its discriminant validity.

In Phase I, a hierarchical task analysis was used to develop the metrics. In Phase II, a survey of expert colorectal surgeons established the importance of the developed metrics. In Phase III, to establish discriminant validity, surgical trainees and surgeons, divided into novice and experienced groups, constructed a side-to-side anastomosis on porcine small bowel using a linear cutting stapler. The participants' performances were videotaped and rated by 2 independent observers. Partial least squares regression was used to compute the weights for the task-specific metrics to obtain weighted total score.

In Phase II, a total of 45 colorectal surgeons were surveyed: 28 with more than 15 years, 13 with 5 to 15 years, and 4 with less than 5 years of experience. The consensus was obtained on all the task-specific metrics in the more experienced groups. In Phase III, 20 subjects participated equally in both groups. The experienced group performed better than the novice group regardless of the rating scale used: global rating scale (p = 0.009) and the task-specific metrics (p = 0.012). After partial least squares regression, the weighted task-specific metric score continued to show that the experienced group performed better (p < 0.001).

Task-specific metric items were developed based on expert consensus and showed good discriminant validity compared with a global rating scale between experienced and novice operators. These items can be used for evaluating technical skills in a stapled small bowel anastomosis model.

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.

The aim of this study was to develop a simulated technical skills examination specific to oral and maxillofacial surgery, assessing the validity of the test and its psychometric properties and obtaining feedback on the examination from both resident trainees and expert evaluators. The Oral and Maxillofacial Objective Structured Assessment of Technical Skills (OMOSATS) examination is a simulated procedure-based eight-station technical skills examination. Fourteen oral and maxillofacial surgery (OMFS) residents of various levels participated in 12-minute surgical tasks and were evaluated by blinded expert OMFS surgeons. The tasks included suturing, tracheostomy, arthroscopy, dental implant placement, plate bending and lag screw application, microvascular anastomosis, and skin grafting. These tasks reflect a broad range of technical milestones necessary for the practice of OMFS. Tasks were evaluated using a validated global rating scale and task-specific checklists. The results of the pilot study indicated strong face and construct validity for seven of the eight OMOSATS stations in discerning the level of training based on technical skill demonstrated at each of the eight surgical stations. Additionally, through one-way ANOVA, P-values were observed to confirm the level of discernment. The OMOSATS examination is a novel way to assess the technical skills of OMFS residents. This type of performance-based assessment has been used in other surgical domains for many years. There is strong evidence to support the use of technical skills examinations to help evaluate, teach, and provide surgeon educators with training outcomes for our OMFS trainees.

Skills assessment in Robotics-Assisted Minimally Invasive Surgery (RAMIS) is mainly performed based on temporal, motion-based and outcome-based metrics. While these components are essential for the proper assessment of skills in RAMIS, they do not suffice for full representation of all underlying aspects of skilled performance. Besides such commonplace components of skills, there exist other elements to be taken into account for comprehensive skills assessment. Among such elements are cognitive states (such as levels of stress, attention, concentration) that can directly affect performance. Investigating the impact of electrocortical activity and cognitive states of RAMIS surgeons over their performance has, however, received little attention in the literature. Therefore, in this paper, novel performance metrics based on electroencephalography (EEG) signals are studied for potential augmentation into RAMIS training and its assessment platform. For this purpose, a user study was conducted involving 23 novices and 9 expert RAMIS surgeons. The participants were asked to perform two tasks on the dv-Trainer®, (Mimic Technologies) RAMIS simulator, while their brain EEG signals were being measured using the Muse EEG headband (InteraXon Inc.). The performance metrics were defined as mean values of band powers of EEG signals over various ranges of frequency. Statistical analysis was performed to evaluate metrics over 5 different ranges of frequency for 4 electrode locations and during 2 RAMIS training tasks. The results indicated statistically significant differences in electrocortical activity between novices and experts in temporoparietal and left frontal regions of their brain for mid to high-frequency ranges. Overall, RAMIS experts showed lower levels of electrocortical activity in those regions compared to novices. The results indicate that electrocortical activity measured by EEG signals have the potential to provide useful information for skills assessment in RAMIS.

Surgical education serves as the foundation for high-quality, efficient patient care. Because clinical knowledge is gained and newer surgical techniques developed, this material is incorporated into the curricula of surgical trainees. Methods for studying this integration vary widely, however, providing data that, at times, is challenging to interpret in the context of patient care and outcomes.

A review of the literature was conducted to evaluate current practices in surgical education research, as well as topics areas of focus.

Several techniques in surgical education research currently exist, including surveys, knowledge assessments, tests of skill, and single-arm educational interventions, which may or may not include a prepost design with assessments administered both before and immediately following the intervention. The applicability of these measures to patient outcomes is variable.

Research in surgical education represents a field of great interest, with opportunity for novel investigations among a broad collection of topic areas. Educational research should be approached in a methodologically rigorous fashion with high investigational standards in order to advance the education of surgical trainees and the care of surgical patients.

Operative skills correlate with patient outcomes, yet at the completion of training or after learning a new procedure, these skills are rarely formally evaluated. There is interest in the use of summative video assessment of laparoscopic benign foregut and hiatal surgery (LFS). If this is to be used to determine competency, it must meet the robust criteria established for high-stakes assessments. The purpose of this review is to identify tools that have been used to assess performance of LFS and evaluate the available validity evidence for each instrument.

A systematic search was conducted up to July 2017. Eligible studies reported data on tools used to assess performance in the operating room during LFS. Two independent reviewers considered 1084 citations for eligibility. The characteristics and testing conditions of each assessment tool were recorded. Validity evidence was evaluated using five sources of validity (content, response process, internal structure, relationship to other variables, and consequences).

There were six separate tools identified. Two tools were generic to laparoscopy, and four were specific to LFS [two specific to Nissen fundoplication (NF), one heller myotomy (HM), and one paraesophageal hernia repair (PEH)]. Overall, only one assessment was supported by moderate evidence while the others had limited or unknown evidence. Validity evidence was based mainly on internal structure (all tools reporting reliability and item analysis) and content (two studies referencing previous papers for tool development in the context of clinical assessment, and four listing items without specifying the development procedures). There was little or no evidence supporting test response process (one study reporting rater training), relationship to other variables (two comparing scores in subjects with different clinical experience), and consequences (no studies). Two tools were identified to have evidence for video assessment, specific to NF.

There is limited evidence supporting the validity of assessment tools for laparoscopic foregut surgery. This precludes their use for summative video-based assessment to verify competency. Further research is needed to develop an assessment tool designed for this purpose.

Faculty evaluations, ABSITE scores, and operative case volumes often tell little about true resident performance. We developed an objective structured clinical examination called the Surgical X-Games (5 rooms, 15 minutes each, 12-15 tests total, different for each postgraduate [PGY] level). We hypothesized that performance in X-Games will prove more useful in identifying areas of strength or weakness among general surgery (GS) residents than faculty evaluations, ABSITE scores, or operative cases volumes.

PGY 2 to 5 GS residents (n = 35) were tested in a semiannual X-Games assessment using multiple simulation tasks: laparoscopic skills, bowel anastomosis, CT/CXR analysis, chest tube placement, etc. over 1 academic year. Resident scores were compared to their ABSITE, in-training evaluation reports, and operating room case numbers.

Academic medical center.

PGY-2, 3, 4, and 5 GS residents at Mayo Clinic in Rochester, MN.

Results varied greatly within each class except for staff evaluations: in-training evaluation reports medians for PGY-2s were 5.3 (range: 5.0-6.0), PGY-3s 5.9 (5.5-6.3), PGY-4s 5.6 (5.0-6.0), and PGY-5s were 6.1 (5.6-6.9). Although ABSITE and operating room case volumes fluctated greatly with each PGY class, only X-Games scores (median: PGY-2 = 82, PGY-3 = 61, PGY-4 = 76, and PGY-5 = 60) correlated positively (p < 0.05) with operative case volume and negatively (p < 0.05) with staff evaluations.

X-Games assessment generated wide differentiation of resident performance quickly, inexpensively, and objectively. Although "Minnesota-nice" surgical staff may feel all GS trainees are "above average," objective assessment tells us otherwise.

Primary: Assess the ability of faculty graders to predict the objectively measured strength of distal radius fracture fixation. Secondary: Compare resident skill variation and retention related to other knowable training data.

Residents were allowed 60 minutes to stabilize a standardized distal radius fracture using an assigned fixed-angle volar plate. Faculty observed and subjectively graded the residents without providing real-time feedback. Objective biomechanical evaluation (construct strength and stiffness) was compared to subjective grades. Resident-specific characteristics (sex, PGY, and ACGME case log) were also used to compare the objective data.

A simulated operating room in our laboratory.

Post-graduate year 2, 3, 4, and 5 orthopedic residents.

Primary: Faculty were not successful at predicting objectively measured fixation, and their subjective scoring suggests confirmation bias as PGY increased. Secondary: Resident year-in-training alone did not predict objective measures (p = 0.53), but was predictive of subjective scores (p < 0.001). Skills learned were not always retained, as 29% of residents objectively failed subsequent to passing. Notably, resident-reported case-specific experience alone was inversely correlated with objective fixation strength.

This testing model enabled the collection of objective and subjective resident skill scores. Faculty graders did not routinely predict objective measures, and their subjective assessment appears biased related to PGY. Also, in vivo case volume alone does not predict objective results. Familiar faculty teaching consistency, and resident grading by external faculty unfamiliar with tested residents, might alter these results.

Surgical residents' exposure to aesthetic procedures remains limited in residency training. The development of the Montreal augmentation mammaplasty operation (MAMO) simulator aims to provide an adjunctive training method and assessment tool to complement the evolving competency-based surgical curriculum.

To perform face, content, and construct validations of the MAMO simulator for subpectoral breast augmentation procedures and assess the reliability of the assessment scales used.

Plastic surgery staff and residents were recruited to perform a subpectoral breast augmentation on the simulator. Video recordings of their performance were blindly evaluated using the objective structured assessment of technical skills (OSATS) system consisting of the global rating scale (GRS), mammaplasty objective assessment tool (MOAT), and a surgery-specific Checklist score.

Fourteen plastic surgery residents and seven expert plastic surgeons were enrolled. Experts' performance was significantly higher than residents' according to each of GRS, MOAT, and Checklist scores. Mean values of residents and experts were 23.4 (2.5) vs 36.9 (3.1) (P < 0.0001) for GRS score, 30.4 (2.2) vs 40 (3.2) (P < 0.0001) for MOAT scores, and 9.7 (1.5) vs 12 (1) (P < 0.001) for Checklist scores, respectively. Face and content validations showed excellent results among parameters evaluated, with an overall mean score of 4.8 (0.3) on 5. Cronbach's alpha was 0.96 and 0.83 for GRS and MOAT scores, respectively. Intraclass correlation coefficients for interrater reliability were excellent at 0.93, 0.92, and 0.89 for the GRS, MOAT, and Checklist scores, respectively.

This study proves the construct simulator to be valid and the assessment scales to be reliable.

Endoscopic ear surgery (EES) is an emerging technique requiring single-handed dissection with limited depth perception. Current options for EES simulation and training are limited. Herein, we introduce a versatile, low-cost surgical skills trainer that aims to improve the fine motor control necessary for EES.

Prospective validation study.

Surgical simulation laboratory.

Seven subjects ranging in experience from medical students ("Novices") to experienced ear surgeons ("Experts") participated in the validation study. Experts (n = 3) were defined as performing >10 EES cases per year.

The skills trainer was constructed from a 3" diameter polyvinyl chloride pipe cap modified with two ports for instrument passage. A wooden platform was placed inside at an appropriate working distance for ear surgery. Eight interchangeable skills modules were fabricated on wooden squares (3 cm × 3 cm) using materials such as #19 wire brads, 1.6 mm glass beads, and 26-gauge jewelry wire. The material cost of this reusable model was $15. Subjects completed each skills module in triplicate, followed by a Likert-based survey.

Expert performance was superior to novices in 100% (8/8) of skills modules, i.e., threading beads on a wire (43 versus 127 s, p < 0.001) and placing a simulated prosthesis (13 versus 68 s, p = 0.01). Most participants (86%) agreed the trainer orientation was accurate and all participants (100%) were satisfied with the experience.

This low-cost modular task trainer may help fill a void in otologic training by allowing efficient, deliberate practice of validated exercises designed to improve fine motor control with EES instrumentation.

There is a recognized need to develop high-stakes technical skills assessments for decisions of certification and resident promotion. High-stakes examinations requires a rigorous approach in accruing validity evidence throughout the developmental process. One of the first steps in development is the creation of a blueprint which outlines the potential content of examination. The purpose of this validation study was to develop an examination blueprint for a Canadian General Surgery assessment of technical skill certifying examination.

A Delphi methodology was used to gain consensus amongst Canadian General Surgery program directors as to the content (tasks or procedures) that could be included in a certifying Canadian General Surgery examination. Consensus was defined a priori as a Cronbach's α ≥ 0.70. All procedures or tasks reaching a positive consensus (defined as ≥80% of program directors rated items as ≥4 on the 5-point Likert scale) were then included in the final examination blueprint.

Two Delphi rounds were needed to reach consensus. Of the 17 General Surgery Program directors across the country, 14 (82.4%) and 10 (58.8%) program directors responded to the first and second round, respectively. A total of 59 items and procedures reached positive consensus and were included in the final examination blueprint.

The present study has outlined the development of an examination blueprint for a General Surgery certifying examination using a consensus-based methodology. This validation study will serve as the foundational work from which simulated model will be developed, pilot tested and evaluated.

Ultrasound guidance is used increasingly to perform the following 6 bedside procedures that are core competencies of hospitalists: abdominal paracentesis, arterial catheter placement, arthrocentesis, central venous catheter placement, lumbar puncture, and thoracentesis. Yet most hospitalists have not been certified to perform these procedures, whether using ultrasound guidance or not, by specialty boards or other institutions extramural to their own hospitals. Instead, hospital privileging committees often ask hospitalist group leaders to make ad hoc intramural certification assessments as part of credentialing. Given variation in training and experience, such assessments are not straightforward "sign offs." We thus convened a panel of experts to conduct a systematic review to provide recommendations for credentialing hospitalist physicians in ultrasound guidance of these 6 bedside procedures. Pathways for initial and ongoing credentialing are proposed. A guiding principle of both is that certification assessments for basic competence are best made through direct observation of performance on actual patients.

Simulation stands to serve an important role in modern competency-based programs of assessment in postgraduate medical education. Our objective was to compare the performance of individual emergency medicine (EM) residents in a simulation-based resuscitation objective structured clinical examination (OSCE) using the Queen's Simulation Assessment Tool (QSAT), with portfolio assessment of clinical encounters using a modified in-training evaluation report (ITER) to understand in greater detail the inferences that may be drawn from a simulation-based OSCE assessment.

A prospective observational study was employed to explore the use of a multicenter simulation-based OSCE for evaluation of resuscitation competence. EM residents from five Canadian academic sites participated in the OSCE. Video-recorded performances were scored by blinded raters using the scenario-specific QSATs with domain-specific anchored scores (primary assessment, diagnostic actions, therapeutic actions, communication) and a global assessment score (GAS). Residents' portfolios were evaluated using a modified ITER subdivided by CanMEDS roles (medical expert, communicator, collaborator, leader, health advocate, scholar, and professional) and a GAS. Correlational and regression analyses were performed comparing components of each of the assessment methods.

Portfolio review and ITER scoring was performed for 79 residents participating in the simulation-based OSCE. There was a significant positive correlation between total OSCE and ITER scores (r = 0.341). The strongest correlations were found between ITER medical expert score and each of the OSCE GAS (r = 0.420), communication (r = 0.443), and therapeutic action (r = 0.484) domains. ITER medical expert was a significant predictor of OSCE total (p = 0.002). OSCE therapeutic action was a significant predictor of ITER total (p = 0.02).

Simulation-based resuscitation OSCEs and portfolio assessment captured by ITERs appear to measure differing aspects of competence, with weak to moderate correlation between those measures of conceptually similar constructs. In a program of competency-based assessment of EM residents, a simulation-based OSCE using the QSAT shows promise as a tool for assessing medical expert and communicator roles.

Research has clearly shown the benefits of surgical simulators to train laparoscopic motor skills required for positive patient outcomes. We have developed the Virtual Basic Laparoscopic Skill Trainer (VBLaST) that simulates tasks from the Fundamentals of Laparoscopic Surgery (FLS) curriculum. This study aims to show convergent validity of the VBLaST pattern cutting module via the CUSUM method to quantify learning curves along with motor skill transfer from simulation environments to ex vivo tissue samples.

18 medical students at the University at Buffalo, with no prior laparoscopic surgical skills, were placed into the control, FLS training, or VBLaST training groups. Each training group performed pattern cutting trials for 12 consecutive days on their respective simulation trainers. Following a 2-week break period, the trained students performed three pattern cutting trials on each simulation platform to measure skill retention. All subjects then performed one pattern cutting task on ex vivo cadaveric peritoneal tissue. FLS and VBLaST pattern cutting scores, CUSUM scores, and transfer task completion times were reported.

Results indicate that the FLS and VBLaST trained groups have significantly higher task performance scores than the control group in both the VBLaST and FLS environments (p < 0.05). Learning curve results indicate that three out of seven FLS training subjects and four out of six VBLaST training subjects achieved the "senior" performance level. Furthermore, both the FLS and VBLaST trained groups had significantly lower transfer task completion times on ex vivo peritoneal tissue models (p < 0.05).

We characterized task performance scores for trained VBLaST and FLS subjects via CUSUM analysis of the learning curves and showed evidence that both groups have significant improvements in surgical motor skill. Furthermore, we showed that learned surgical skills in the FLS and VBLaST environments transfer not only to the different simulation environments, but also to ex vivo tissue models.

The ability of an assessment to predict performance would be of major benefit to residency programs, allowing for early identification of residents at risk.

We sought to establish whether passing the Objective Structured Assessment of Technical Skills (OSATS) examination in postgraduate year 1 (PGY-1) predicts future performance.

Between 2002 and 2012, 133 PGY-1 surgery residents at the University of Toronto (Toronto, Ontario, Canada) completed an 8-station, simulated OSATS examination as a component of training. With recently set passing scores, residents were assigned a pass/fail status using 3 standards setting methods (contrasting groups, borderline group, and borderline regression). Future in-training performance was compared between residents who had passed and those who failed the OSATS, using in-training evaluation reports from resident files. A Mann-Whitney U test compared performance among groups at PGY-2 and PGY-4 levels.

Residents who passed the OSATS examination outperformed those who failed, when compared during PGY-2 across all 3 standard setting methodologies (P < .05). During PGY-4, only the contrasting groups method showed a significant difference (P < .05).

We found that PGY-1 surgical resident pass/fail status on a technical skills examination was associated with future performance on in-training evaluation reports in later years. This provides validity evidence for the current PGY-1 pass/fail score, and suggests that this technical skills examination may be used to predict performance and to identify residents who require remediation.

Assessment is an integral component of training and credentialing surgeons for practice. Traditional methods of cognitive and technical appraisal are well established but have clear shortcomings. This review outlines the components of the surgical care assessment model, identifies the deficits of current evaluation techniques, and discusses novel and emerging technologies that attempt to ameliorate this educational void.

Training skillful and competent surgeons is critical to ensure high quality of care and to minimize disparities in access to effective care. Traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. Simultaneously, technological developments are enabling capture and analysis of large amounts of complex surgical data. These developments are motivating a "surgical data science" approach to objective computer-aided technical skill evaluation (OCASE-T) for scalable, accurate assessment; individualized feedback; and automated coaching. We define the problem space for OCASE-T and summarize 45 publications representing recent research in this domain. We find that most studies on OCASE-T are simulation based; very few are in the operating room. The algorithms and validation methodologies used for OCASE-T are highly varied; there is no uniform consensus. Future research should emphasize competency assessment in the operating room, validation against patient outcomes, and effectiveness for surgical training.

Minimally invasive surgery (MIS) requires the mastery of manual skills and a specific training is required. Apart from residencies and fellowships in MIS, other learning opportunities utilize massive training, mainly with use of simulators in short courses. A long-term postgraduate course represents an opportunity to learn through training using distributed practice.

The objective of this study is to assess the use of distributed practice for acquisition of basic minimally invasive skills in surgeons who participated in a long-term MIS postgraduate course.

A prospective, longitudinal and quantitative study was conducted among surgeons who attended a 1-year postgraduate course of MIS in Brazil, from 2012 to 2014. They were tested through five different exercises in box trainers (peg-transfer, passing, cutting, intracorporeal knot, and suture) in the first (t0), fourth (t1) and last, eighth, (t2) meetings of this course. The time and penalties of each exercise were collected for each participant. Participant skills were assessed based on time and accuracy on a previously tested score.

Fifty-seven surgeons (participants) from three consecutive groups participated in this study. There was a significant improvement in scores in all exercises. The average increase in scores between t0 and t2 was 88% for peg-transfer, 174% for passing, 149% for cutting, 130% for intracorporeal knot, and 120% for suture (p < 0.001 for all exercises).

Learning through distributed practice is effective and should be integrated into a MIS postgraduate course curriculum for acquisition of core skills.

Percutaneous renal access (PCA) is a challenging step during percutaneous nephrolithotomy. The aim of this study is to review the literature for different types of simulators described for PCA.

Databases of Medline, Embase, Cochrane Library, OvidSP, and Google Scholar were systematically searched until May 2016. The studies were analyzed regarding the type of simulator (nonbiologic, biologic, live animal, and virtual reality [VR]), type of validity (face, content, construct, and predictive), cost-effectiveness, and whether these simulators have been used for training and/or assessment of PCA. In addition, the study looked at the educational impact of these simulators in terms of the transfer of PCA skills to the operating room.

Several bench, animal, and VR simulators for training in PCA were identified. Only few studies were found on assessment of PCA skills. Biological bench models used porcine or bovine kidneys wrapped within foam, silicone, chicken carcass, or full-thickness skin flap alone. Other biological models used additional subcutaneous fascia, muscle, or ribs. Nonbiological models used prototypes, including 3D printing. Only one study reported the use of anesthetized live pig for training. The PERC Mentor™ was the only VR simulator, which has been validated for training and assessment of PCA skills. However, none of these studies assessed the educational impact of PCA simulators. Furthermore, most of the studies did not address the validity and the cost of the simulator.

While several biological and nonbiological PCA models exist, there is paucity of literature regarding the validity and educational impact of these simulators. The PERC Mentor simulator is the sole validated simulator for training and assessment of PCA skills. However, it is expensive and there is little evidence of its educational impact. Therefore, more research is needed to validate the available simulators and assess their educational impact for urology trainees.

Competency-based education necessitates assessments that determine whether trainees have acquired specific competencies. The evidence on the ability of internal raters (staff surgeons) to provide accurate assessments is mixed; however, this has not yet been directly explored in the operating room. This study's objective is to compare the ratings given by internal raters vs an expert external rater (independent to the training process) in the operating room.

Raters assessed general surgery residents during a laparoscopic cholecystectomy for their technical and nontechnical performance.

Fifteen cases were observed. There was a moderately positive correlation (r_s = .618, P = .014) for technical performance and a strong positive correlation (r_s = .731, P = .002) for nontechnical performance. The internal raters were less stringent for technical (mean rank 3.33 vs 8.64, P = .007) and nontechnical (mean rank 3.83 vs 8.50, P = .01) performances.

This study provides evidence to help operationalize competency-based assessments.

The objectives of this study were to assemble an international perspective on (1) current, and (2) ideal technical performance assessment methods, and (3) barriers to their adoption during: selection, in-training, and certification.

A questionnaire was distributed to international educational directorates.

Eight of 10 jurisdictions responded. Currently, aptitude tests or simulated tasks are used during selection, observational rating scales during training and nothing is used at certification. Ideally, innate ability should be determined during selection, in-training evaluation reports, and global rating scales used during training, whereas global and procedure-specific rating scales used at the time of certification. Barriers include lack of predictive evidence for use in selection, financial limitations during training, and a combination with respect to certification.

Identifying current and ideal evaluation methods will prove beneficial to ensure the best assessments of technical performance are chosen for each training time point.

The aim of the present study was to assess the content and construct validity of the Canine Laparoscopic Simulator (CLS). Forty-two veterinarians were assigned to experienced (n=12), control (n=15), and training (n=15) groups, which were assessed while performing four laparoscopic tasks on the CLS. The initial and final assessments of all tasks were performed blindly by two experienced surgeons using the Global Operative Assessment of Laparoscopic Skills (GOALS) and a task-specific checklist. At the end of the study, the subjects completed an anonymous survey. The experienced group performed all of the tasks faster, with higher GOALS and checklist scores than the training and control groups (p≤.001). In the second assessment, the training group reduced the time needed to complete all of the tasks and obtained significantly higher GOALS and checklist scores than the control group. The participants perceived the CLS and its training program to be positive or very positive. The CLS and its training program demonstrated content and construct validity, supporting the suitability of the simulator for training and teaching and its ability to distinguish the degree of experience in laparoscopic surgery among veterinarians. In addition, face validity showed that the veterinarians fully accepted the CLS's usefulness for learning basic laparoscopic skills.

Distractions during surgical procedures have been linked to medical error and team inefficiency. This systematic review identifies the most common and most significant forms of distraction in order to devise guidelines for mitigating the effects of distractions in the OR.

In January 2015, a PubMed and Google Scholar search yielded 963 articles, of which 17 (2 %) either directly observed the occurrence of distractions in operating rooms or conducted a laboratory experiment to determine the effect of distraction on surgical performance.

Observational studies indicated that movement and case-irrelevant conversation were the most frequently occurring distractions, but equipment and procedural distractions were the most severe. Laboratory studies indicated that (1) auditory and mental distractions can significantly impact surgical performance, but visual distractions do not incur the same level of effects; (2) task difficulty has an interaction effect with distractions; and (3) inexperienced subjects reduce their speed when faced with distractions, while experienced subjects did not.

This systematic review suggests that operating room protocols should ensure that distractions from intermittent auditory and mental distractions are significantly reduced. In addition, surgical residents would benefit from training for intermittent auditory and mental distractions in order to develop automaticity and high skill performance during distractions, particularly during more difficult surgical tasks. It is unclear as to whether training should be done in the presence of distractions or distractions should only be used for post-training testing of levels of automaticity.

An arteriovenous fistula (AVF), performed for hemodialysis access, provides one of the few remaining opportunities for general surgery residents to perform an open vascular anastomosis (VA). Limited data exist regarding the learning curve of residents performing this procedure. The objective of this study was to determine how residents improve in performance of VA by implementing real-time tracking of anastomosis time as well as technical errors.

From April 2012 to January 2014, we conducted a prospective intraoperative assessment of 9 postgraduate year 3 general surgery residents during the performance of AVFs using a checklist of common errors in VA. Time for AVF anastomosis completion and number and types of technical errors during anastomosis were recorded. Primary end points were the change in anastomosis time and change in technical errors over time.

A total of 86 AVFs were performed and assessed intraoperatively. Each resident performed a median of 10 AVFs (interquartile range [IQR]: 7-11). The mean anastomosis time was 18.1 minutes. The mean number of technical errors was 13.8 per case. Overall, for every additional AVF performed, mean anastomosis time decreased by 0.63 minutes (95% CI: 0.45-0.81, p < 0.0001) and the mean number of technical errors decreased by 1.0 (95% CI: 0.7-1.3, p < 0.0001). The greatest improvement in overall errors (mean difference = 7.9, p = 0.03) and time (mean difference = 4.7min, p = 0.03) occurred after the performance of 3 AVFs. However, when analyzed by individual resident, the R(2) value for anastomotic time by number of AVFs performed ranged from 0.01 to 0.69. Similarly, for technical errors, the R(2) value by number of AVFs performed ranged from 0.04 to 0.62.

In novice surgical residents performing AVFs, improvement in VA skill can readily be tracked via anastomosis time and technical errors. Collectively, there is a strong association between number of cases performed and reduction in time and errors. However, individually, the number of cases completed did not correlate well with time and errors. These findings suggest that for VA skills, determining progression from novice to competence cannot rely on case volume but rather needs to be individualized.

The Fundamentals of Laparoscopic Surgery (FLS) program uses five simulation stations (peg transfer, precision cutting, loop ligation, and suturing with extracorporeal and intracorporeal knot tying) to teach and assess laparoscopic surgery skills. We sought to summarize evidence regarding the validity of scores from the FLS assessment.

We systematically searched for studies evaluating the FLS as an assessment tool (last search update February 26, 2013). We classified validity evidence using the currently standard validity framework (content, response process, internal structure, relations with other variables, and consequences).

From a pool of 11,628 studies, we identified 23 studies reporting validity evidence for FLS scores. Studies involved residents (n = 19), practicing physicians (n = 17), and medical students (n = 8), in specialties of general (n = 17), gynecologic (n = 4), urologic (n = 1), and veterinary (n = 1) surgery. Evidence was most common in the form of relations with other variables (n = 22, most often expert-novice differences). Only three studies reported internal structure evidence (inter-rater or inter-station reliability), two studies reported content evidence (i.e., derivation of assessment elements), and three studies reported consequences evidence (definition of pass/fail thresholds). Evidence nearly always supported the validity of FLS total scores. However, the loop ligation task lacks discriminatory ability.

Validity evidence confirms expected relations with other variables and acceptable inter-rater reliability, but other validity evidence is sparse. Given the high-stakes use of this assessment (required for board eligibility), we suggest that more validity evidence is required, especially to support its content (selection of tasks and scoring rubric) and the consequences (favorable and unfavorable impact) of assessment.

To assess the acquisition of basic laparoscopic skills of Medical students trained on a surgical simulator.

First- and second-year Medical students participated on a laparoscopic training program on simulators. None of the students had previous classes of surgical technique, exposure to surgical practice nor training prior to the enrollment in to the study. Students´ time were collected before and after the 150-minute training. Skill acquisition was measured comparing time and scores of students and senior instructors of laparoscopic surgery.

Sixty-eight students participated of the study, with a mean age of 20.4 years, with a predominance of first-year students (62%). All students improved performance in score and time, after training (p<0,001). Score improvement in the exercises ranged from 294.1 to 823%. Univariate and multivariate analyses identified that second-year Medical students have achieved higher performance after training.

Medical students who had never been exposed to surgical techniques can acquire basic laparoscopic skills after training in simulators. Second-year undergraduates had better performance than first-year students.

To conduct a review of the state of virtual reality (VR) simulation technology, to identify areas of surgical education that have the greatest potential to benefit from it, and to identify challenges to implementation.

Simulation is an increasingly important part of surgical training. VR is a developing platform for using simulation to teach technical skills, behavioral skills, and entire procedures to trainees and practicing surgeons worldwide. Questions exist regarding the science behind the technology and most effective usage of VR simulation. A symposium was held to address these issues.

Engineers, educators, and surgeons held a conference in November 2013 both to review the background science behind simulation technology and to create guidelines for its use in teaching and credentialing trainees and surgeons in practice.

Several technologic challenges were identified that must be overcome in order for VR simulation to be useful in surgery. Specific areas of student, resident, and practicing surgeon training and testing that would likely benefit from VR were identified: technical skills, team training and decision-making skills, and patient safety, such as in use of electrosurgical equipment.

VR simulation has the potential to become an essential piece of surgical education curriculum but depends heavily on the establishment of an agreed upon set of goals. Researchers and clinicians must collaborate to allocate funding toward projects that help achieve these goals. The recommendations outlined here should guide further study and implementation of VR simulation.

There is no objective methodology to assess trainee progress in endoscopy. Our prior work has detailed the development of the endoscopic part-task training box.

To assess validity evidence regarding relationship to other variables by evaluating a correlation between level of endoscopic experience and training box score.

Prospective validation study.

Three academic institutions.

A total of 42 participants: 7 novices, 7 first-year GI fellows, 7 second-year GI fellows, 7 third-year GI fellows, 7 attending physicians, and 7 interventional attending physicians.

The training box consists of 5 modules: retroflexion, knob control, torque, polypectomy, and navigation/loop reduction. Performance is scored for precision and speed. Each participant was required to complete the training box once. Additionally, 5 participants at different endoscopic levels completed the training box 3 times at 1-week intervals.

A correlation between level of endoscopic experience and training box score.

All 42 participants completed the 5 modules during a single session. Aggregate training box scores differed significantly between each training level (P values < .05). Individual modules significantly differentiated between experience-level groups (novices, fellows, and attending physicians; P values < .01). Participants who repeated the training box demonstrated score improvement over time, with persistence of separation between training levels.

The training box focuses only on the technical aspects of endoscopy and does not address the cognitive elements of endoscopic training.

The endoscopic part-task training box is able to objectively assess endoscopic ability by differentiating scores based on clinical experience. Further multicenter efforts are now needed to establish learning curves and to correlate use of the simulator with improved clinical aptitude.