Evaluating learning and change in orthopaedics: What is the evidence-base?

Abstract

Learning and change are key elements of clinical governance and are responsible for the progression of our specialty. Although orthopaedics has been slow to embrace quality improvement, recent years have seen global developments in surgical education, quality improvement, and patient outcome research. This review covers recent advances in the evaluation of learning and change and identifies the most important research questions that remain unanswered. Research into proxies of learning is improving but more work is required to identify the best proxy for a given procedure. Learning curves are becoming commonplace but are poorly integrated into postgraduate training curricula and there is little agreement over the most appropriate method to analyse learning curve data. With various organisations promoting centralisation of care, learning curve analysis is more important than ever before. The use of simulation in orthopaedics is developing but is yet to be formally mapped to resident training worldwide. Patient outcome research is rapidly changing, with an increased focus on quality of life measures. These are key to patients and their care. Cost-utility analysis is increasingly seen in orthopaedic manuscripts and this needs to continue to improve evidence-based care. Large-scale international, multi-centre randomised trials are gaining popularity and updated guidance on sample size estimation needs to become widespread. A global lack of surgeon equipoise will need to be addressed. Quality improvement projects frequently employ interrupted time-series analysis to evaluate change. This technique’s limitations must be acknowledged, and more work is required to improve the evaluation of change in a dynamic healthcare environment where multiple interventions frequently occur. Advances in the evaluation of learning and change are needed to drive improved international surgical education and increase the reliability, validity, and importance of the conclusions drawn from orthopaedic research.

Key Words: Learning, Change, Quality improvement, Orthopaedics, Surgical education

Core tip: Learning and change are integral to clinical governance. Despite orthopaedics being slow to embrace quality improvement, recent years have seen global improvements in the field. This review covers various aspects of learning and change including: proxies of learning, learning curve analysis, simulation, outcome measures, retrospective and prospective studies as well as time-series analysis. It summarises the current evidence-base and identifies research questions that remain unanswered.

INTRODUCTION

Learning and change are key elements of clinical governance, a framework through which healthcare organisations are accountable for continuously improving the quality of their services[1]. Historically, despite a growing interest within medicine, orthopaedics has been slow to embrace quality improvement. However, in recent years there has been a global drive towards evidence-based improvement in the quality of service provision[2], surgical education[3], and outcome research[4,5].

The process of evaluating learning and change is what guides improvement strategy. We must accept that “not all change is improvement, but all improvement is change”[6]. Proxies of performance and methods to analyse the change in performance over time are core themes of current healthcare research and play a critical role in the development of our specialty. This is evident in the increasing use of patient-reported outcome measures (PROMs) to guide evidence-based care and in the use of learning curve data as an assessment metric to promote self-regulated learning[7].

The aim of this review is to provide orthopaedic surgeons with an evidence-based introduction to the evaluation of learning and change in this era of healthcare quality improvement reform.

LEARNING

Proxies of learning

In order to draw meaningful conclusions from data, learning variables need to demonstrate high validity. Validity is “the extent to which an assessment measures what it intends to measure”[8]. This is a judgment based on several factors, including whether the variable correlates with other ‘gold standard’ measures.

Proxies of learning are largely divided into surgical process and patient outcome variables. Surgical process variables include operative factors such as operative time, intraoperative blood loss, implant alignment, and fluoroscopy dose. Patient outcome variables include PROMs, mortality, morbidity, length of hospital stay, and transfusion requirement. A key systematic review by Ramsey and colleagues found that operative time was the most commonly used proxy of learning[9]. Although this variable is easily accessible, its validity in the context of learning is less robust. Global rating scales for surgical procedures have been increasingly used to evaluate learning in orthopaedic surgery, and are probably a better surrogate marker of learning[10]. In particular, their combination with motion analysis seems to offer a valid proficiency metric for arthroscopy simulators[11]. More work is required to directly compare the validity of different proxies of learning in different orthopaedic procedures.

Learning curves

A learning curve is a graphical representation of the relationship between learning effort and learning outcome[12]. It serves as a visual representation of the process of learning and allows researchers to employ statistical techniques to draw conclusions from the data. A typical learning curve resembles that of a negative exponential: With experience, a greater learning effort is required to produce the same improvement in performance[13]. However, due to the high variability of surgical data, this is rarely the case in practice. Researchers are then faced with interpreting highly variable data from which to draw meaningful conclusions.

The most commonly employed technique to detect learning is the ‘split-group’ method[14]. The data is chronologically split into two or three consecutive groups of arbitrary size, and groups are compared by t-tests or equivalent. Although simple, this technique is fraught with bias and is increasingly disapproved by researchers. For example, a recent systematic review investigating the learning curve of the Latarjet procedure found that most included studies used the split-group method, and called for more rigorous, continuous learning curve modelling techniques[15].

Although other methods for modelling learning curves do exist (e.g., cumulative sum methods), the widespread use of mathematically valid regression techniques in orthopaedics remains sparse[16]. Researchers have recently developed mathematically rigorous segmented linear regression techniques that test multiple learning models and applied these to investigate the learning curves austerity across healthcare systems of total knee and total hip replacements when using imageless navigation[17,18] (Figures 1 and 2). Further studies are required to ensure that mathematically rigorous learning curve techniques become commonplace when evaluating the learning curves of new orthopaedic procedures. Indeed, accurate and informative learning curve analysis is even more important in an era of centralisation of care, where difficult procedures are increasingly reserved for supra-specialist, high-volume surgeons[19].

Figure 1 Learning curve for navigated total hip replacements. Segmented linear regression technique was employed to model learning[17]. Line-plateau model fits the data best, with a plateau being attained at 12 operations.

Figure 2 Learning curve for navigated total knee replacements. Segmented linear regression technique was employed to model learning[17]. Line-plateau model fits the data best, with plateau being attained at 26 operations.

Simulation

The ongoing emphasis on patient safety in conjunction with reduced working hours and financial austerity across healthcare systems has led to improved methods to train surgeons outside the operating room[20]. Simulation-based training has been successfully incorporated into the general surgery training curriculum in the United States[21], and randomised controlled trials (RCT) have proved its benefits[22]. The use of simulation in arthroscopy[23] and trauma[24] is increasing, though the level of evidence for simulation studies in orthopaedics remains low with a lack of focus on nontechnical skills and cost analyses[25]. There are ongoing consultations to map simulation to the trauma and orthopaedics postgraduate curriculum in the United Kingdom[26]. A stronger drive is required to formally integrate simulation training within orthopaedic residency training at an international level.

CHANGE

Change in outcomes in orthopaedics can be considered following operative intervention, and by examining time-series following system interventions. The measures of performance in both settings are similar and reflect the variables we consider to lie at the core of orthopaedic practice. Although there is a degree of overlap with variables used to measure learning, these are largely related to patient outcomes and health economics.

Outcome measures

Prior to implementing and evaluating change, researchers must identify appropriate measures to determine whether an intervention works[27]. Ideally, these should be part of routinely collected data for quality improvement purposes. An example includes the National Hip Fracture Database in the United Kingdom that routinely collects standardised outcome data[28]. It is based on this that the World Hip Trauma Evaluation (WHiTE) study has founded a reliable and organised framework for comprehensive cohort studies on fragility hip fractures[29].

Patient outcomes in orthopaedics mainly include mortality, postoperative complications, infection, performance testing, and PROMs[30]. Of these there has been a recent surge in PROMs research[31]. This is because PROMs lie at the heart of patient-centred care. There is no surprise that health-related quality of life measures such as the EuroQol are increasingly being employed to guide operative decision making in trauma[29,32]. Simultaneously, there is a trend towards including patients in setting research questions through priority setting partnerships[33], and patient and public involvement is now indispensable to healthcare research[34]. Cost-utility, the financial cost for health gain, is the variable that the National Institute for Health and Care Excellence (NICE) uses when forming guidelines for healthcare provision. It is thus very important that orthopaedic surgeons understand and incorporate cost-utility analysis in their research[35].

Variables used to evaluate an intervention are usually divided into outcome measures, process measures, and balancing measures[5,36]. Outcome measures monitor how a system is performing, process measures assess the implementation of an intervention, and balancing measures assess unintended consequences of the intervention.

Once outcome measures are identified and data is collected, analysis of the data is required to evaluate change.

Evaluating change

Operative intervention: Analysing change following operative intervention forms the basis of retrospective and prospective research studies. The level of evidence for a given study depends on a multitude of factors, most importantly study design[37]. There are three types of outcome variables: Continuous (e.g., operative time), categorical (e.g., presence or absence of a complication), and time-to-event (e.g., time to revision of a joint replacement). Statistical tests comparing outcomes consider the type of variable and can include parametric (t-test) and non-parametric (Mann-Whitney) tests, crosstabs (e.g., Chi-squared test and Fischer’s test), and survival analysis. These tests usually output a significance value (P-value) which is a measure of the likelihood that the result was due to chance.

Increased focus is being placed on the minimal clinically important difference - the smallest change in an outcome that a patient would identify as important, and which would usually indicate a change in patient management. Even a very small change can be shown to be statistically significant with a large enough sample size, but this may not be important. There is significant variation in the reporting of sample size calculations in orthopaedic literature[38] and until recently, reporting guidelines were lacking. Adoption of the DELTA² guidance on choosing a target difference and reporting sample size in RCTs should improve this[39].

RCTs are considered the gold-standard hypothesis-testing study design. This is mainly because they allow for controlling of confounding variables that complicate observational studies. Over the last decade there has been a surge in trauma trials on an international scale, starting with the CRASH-2 trial on the effectiveness of tranexamic acid in trauma[40]. Other large-scale randomised trials have followed suit, investigating fixation of intracapsular neck of femur fractures[41], fixation of distal radius fractures[42] and ongoing research on the optimal timing of hip fracture surgery[43] to mention a few.

Although RCTs are excellent for answering certain research questions, retrospective studies remain indispensable. In the era of information technology, ‘Big Data’ is becoming ubiquitous[44]. Using Big Data to identify research questions, guide efficient targeting of resources and subsequently address these questions with randomised trials may not be the exception in a few years. It is definitely appearing promising so far[29]. One major limitation that will need to be addressed in future if RCTs are to output the highest quality data is surgeon equipoise. Surgeons are rarely in true equipoise and they usually have a clear idea of what management option is the best for a given patient. Although few would question the importance of decision making in surgery, it can present an obstacle when patient randomisation is required[45]. This must be addressed through improved surgeon education and standardised randomisation processes.

Time-series analysis: A toolbox for detecting change: Many quality improvement projects evaluate the effectiveness of an intervention by collecting data over time. Data can be graphically displayed as control charts, also known as Shewart charts. They are a statistical process control tool used to determine whether a system is in control and provide immediate feedback about performance[46].

Orthopaedic surgeons may be more familiar with audit cycles. Audit is a framework of quality improvement where performance is compared to a published standard[47]. Part of this process includes introducing an intervention and assessing its effectiveness by comparing performance before and after the intervention by simple statistical group tests. Although ubiquitous in clinical orthopaedics and indeed in all medical specialties, such approaches are sensitive to secular (background) trends. Interrupted time-series (ITS) analysis is a useful tool for evaluating the effectiveness of interventions where data is collected at several time-points before and after the intervention to determine whether any change could be explained by secular trends[48]. Cochrane recommends this tool to evaluate interventions[49] and several recent orthopaedic studies have employed this technique[50,51].

ITS does not come without limitations, and is known to display bias for detecting change at the time of the studied intervention where other changes at different time-points may be equally, if not more important[52,53]. Segmented linear regression models have been developed for evaluating change in retrospective studies by enabling more than one linear segment to describe the periods before and after an intervention. A recent study employing this technique revealed that improvements in time to surgery and 30-d mortality following hip fracture over a 6-year period were likely the result of a combination of surgical, anaesthetic, and procedural improvements over time, rather than due to the introduction of a dedicated hip fracture unit[53] (Figure 3). Future work is required to determine the optimal way to describe retrospective time-series: How many linear segments should be used, and how to best model binary outcomes.

Figure 3 Time to surgery for neck of femur fractures. The vertical dashed line marks the onset of a dedicated hip fracture unit. The line-plateau is the best-fitting linear model for the entire period: the line has equation y = −0.0414t + 40.1868; plateau at y = 24.7033 reached after 375 d. The initial drop may be related to the introduction of the Best Practice Tariff. The hip fracture unit did not significantly affect time to surgery[53].

CONCLUSION

Learning and change are integral to quality improvement and surgical education, and strongly influence the development of our specialty. The orthopaedic community has seen several improvements in PROMs research, learning curve analysis, randomised trial design, and time-series analysis.

Future work is required to improve and standardise learning variables and formally implement simulation in orthopaedic residency education. Global collaborative research networks are developing but integrating randomised trials with Big Data on an international scale to improve orthopaedics will require a concerted effort.