Minireviews
Copyright ©The Author(s) 2019. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Orthop. Nov 18, 2019; 10(11): 378-386
Published online Nov 18, 2019. doi: 10.5312/wjo.v10.i11.378
Evaluating learning and change in orthopaedics: What is the evidence-base?
Epaminondas Markos Valsamis, Mohamed Sukeik
Epaminondas Markos Valsamis, Nuffield Orthopaedic Centre, Oxford University Hospitals NHS Trust, Oxford, OX3 7LD, United Kingdom
Mohamed Sukeik, Department of Trauma and Orthopaedics, Dr. Sulaiman Al-Habib Hospital – Al Khobar, King Salman Bin Abdulaziz Rd, Al Bandariyah, Al Khobar 34423, Saudi Arabia
ORCID number: Epaminondas Markos Valsamis (0000-0003-4154-9596); Mohamed Sukeik (0000-0001-9204-9757).
Author contributions: Valsamis EM and Sukeik M contributed equally to this paper with regard to conception and design of the study, literature review and analysis, drafting, critical revision, editing and approval of the final version.
Conflict-of-interest statement: No potential conflicts of interest to declare. No financial support.
Open-Access: This article is an open-access article which was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Corresponding author: Mohamed Sukeik, MD (Hons), FRCSEd (Tr&Orth), PGA, MD (Res), Consultant Hip and Knee Surgeon, Department of Trauma and Orthopaedics, Dr. Sulaiman Al-Habib Hospital – Al Khobar, King Salman Bin Abdulaziz Rd, Al Bandariyah, Al Khobar 34423, Saudi Arabia. m.sukeik@nhs.net
Telephone: +966-13-8711111
Received: March 11, 2019
Peer-review started: March 12, 2019
First decision: June 12, 2019
Revised: June 27, 2019
Accepted: September 5, 2019
Article in press: September 5, 2019
Published online: November 18, 2019

Abstract

Learning and change are key elements of clinical governance and are responsible for the progression of our specialty. Although orthopaedics has been slow to embrace quality improvement, recent years have seen global developments in surgical education, quality improvement, and patient outcome research. This review covers recent advances in the evaluation of learning and change and identifies the most important research questions that remain unanswered. Research into proxies of learning is improving but more work is required to identify the best proxy for a given procedure. Learning curves are becoming commonplace but are poorly integrated into postgraduate training curricula and there is little agreement over the most appropriate method to analyse learning curve data. With various organisations promoting centralisation of care, learning curve analysis is more important than ever before. The use of simulation in orthopaedics is developing but is yet to be formally mapped to resident training worldwide. Patient outcome research is rapidly changing, with an increased focus on quality of life measures. These are key to patients and their care. Cost-utility analysis is increasingly seen in orthopaedic manuscripts and this needs to continue to improve evidence-based care. Large-scale international, multi-centre randomised trials are gaining popularity and updated guidance on sample size estimation needs to become widespread. A global lack of surgeon equipoise will need to be addressed. Quality improvement projects frequently employ interrupted time-series analysis to evaluate change. This technique’s limitations must be acknowledged, and more work is required to improve the evaluation of change in a dynamic healthcare environment where multiple interventions frequently occur. Advances in the evaluation of learning and change are needed to drive improved international surgical education and increase the reliability, validity, and importance of the conclusions drawn from orthopaedic research.

Key Words: Learning, Change, Quality improvement, Orthopaedics, Surgical education

Core tip: Learning and change are integral to clinical governance. Despite orthopaedics being slow to embrace quality improvement, recent years have seen global improvements in the field. This review covers various aspects of learning and change including: proxies of learning, learning curve analysis, simulation, outcome measures, retrospective and prospective studies as well as time-series analysis. It summarises the current evidence-base and identifies research questions that remain unanswered.



INTRODUCTION

Learning and change are key elements of clinical governance, a framework through which healthcare organisations are accountable for continuously improving the quality of their services[1]. Historically, despite a growing interest within medicine, orthopaedics has been slow to embrace quality improvement. However, in recent years there has been a global drive towards evidence-based improvement in the quality of service provision[2], surgical education[3], and outcome research[4,5].

The process of evaluating learning and change is what guides improvement strategy. We must accept that “not all change is improvement, but all improvement is change”[6]. Proxies of performance and methods to analyse the change in performance over time are core themes of current healthcare research and play a critical role in the development of our specialty. This is evident in the increasing use of patient-reported outcome measures (PROMs) to guide evidence-based care and in the use of learning curve data as an assessment metric to promote self-regulated learning[7].

The aim of this review is to provide orthopaedic surgeons with an evidence-based introduction to the evaluation of learning and change in this era of healthcare quality improvement reform.

LEARNING
Proxies of learning

In order to draw meaningful conclusions from data, learning variables need to demonstrate high validity. Validity is “the extent to which an assessment measures what it intends to measure”[8]. This is a judgment based on several factors, including whether the variable correlates with other ‘gold standard’ measures.

Proxies of learning are largely divided into surgical process and patient outcome variables. Surgical process variables include operative factors such as operative time, intraoperative blood loss, implant alignment, and fluoroscopy dose. Patient outcome variables include PROMs, mortality, morbidity, length of hospital stay, and transfusion requirement. A key systematic review by Ramsey and colleagues found that operative time was the most commonly used proxy of learning[9]. Although this variable is easily accessible, its validity in the context of learning is less robust. Global rating scales for surgical procedures have been increasingly used to evaluate learning in orthopaedic surgery, and are probably a better surrogate marker of learning[10]. In particular, their combination with motion analysis seems to offer a valid proficiency metric for arthroscopy simulators[11]. More work is required to directly compare the validity of different proxies of learning in different orthopaedic procedures.

Learning curves

A learning curve is a graphical representation of the relationship between learning effort and learning outcome[12]. It serves as a visual representation of the process of learning and allows researchers to employ statistical techniques to draw conclusions from the data. A typical learning curve resembles that of a negative exponential: With experience, a greater learning effort is required to produce the same improvement in performance[13]. However, due to the high variability of surgical data, this is rarely the case in practice. Researchers are then faced with interpreting highly variable data from which to draw meaningful conclusions.

The most commonly employed technique to detect learning is the ‘split-group’ method[14]. The data is chronologically split into two or three consecutive groups of arbitrary size, and groups are compared by t-tests or equivalent. Although simple, this technique is fraught with bias and is increasingly disapproved by researchers. For example, a recent systematic review investigating the learning curve of the Latarjet procedure found that most included studies used the split-group method, and called for more rigorous, continuous learning curve modelling techniques[15].

Although other methods for modelling learning curves do exist (e.g., cumulative sum methods), the widespread use of mathematically valid regression techniques in orthopaedics remains sparse[16]. Researchers have recently developed mathematically rigorous segmented linear regression techniques that test multiple learning models and applied these to investigate the learning curves austerity across healthcare systems of total knee and total hip replacements when using imageless navigation[17,18] (Figures 1 and 2). Further studies are required to ensure that mathematically rigorous learning curve techniques become commonplace when evaluating the learning curves of new orthopaedic procedures. Indeed, accurate and informative learning curve analysis is even more important in an era of centralisation of care, where difficult procedures are increasingly reserved for supra-specialist, high-volume surgeons[19].

Figure 1
Figure 1 Learning curve for navigated total hip replacements. Segmented linear regression technique was employed to model learning[17]. Line-plateau model fits the data best, with a plateau being attained at 12 operations.
Figure 2
Figure 2 Learning curve for navigated total knee replacements. Segmented linear regression technique was employed to model learning[17]. Line-plateau model fits the data best, with plateau being attained at 26 operations.
Simulation

The ongoing emphasis on patient safety in conjunction with reduced working hours and financial austerity across healthcare systems has led to improved methods to train surgeons outside the operating room[20]. Simulation-based training has been successfully incorporated into the general surgery training curriculum in the United States[21], and randomised controlled trials (RCT) have proved its benefits[22]. The use of simulation in arthroscopy[23] and trauma[24] is increasing, though the level of evidence for simulation studies in orthopaedics remains low with a lack of focus on nontechnical skills and cost analyses[25]. There are ongoing consultations to map simulation to the trauma and orthopaedics postgraduate curriculum in the United Kingdom[26]. A stronger drive is required to formally integrate simulation training within orthopaedic residency training at an international level.

CHANGE

Change in outcomes in orthopaedics can be considered following operative intervention, and by examining time-series following system interventions. The measures of performance in both settings are similar and reflect the variables we consider to lie at the core of orthopaedic practice. Although there is a degree of overlap with variables used to measure learning, these are largely related to patient outcomes and health economics.

Outcome measures

Prior to implementing and evaluating change, researchers must identify appropriate measures to determine whether an intervention works[27]. Ideally, these should be part of routinely collected data for quality improvement purposes. An example includes the National Hip Fracture Database in the United Kingdom that routinely collects standardised outcome data[28]. It is based on this that the World Hip Trauma Evaluation (WHiTE) study has founded a reliable and organised framework for comprehensive cohort studies on fragility hip fractures[29].

Patient outcomes in orthopaedics mainly include mortality, postoperative complications, infection, performance testing, and PROMs[30]. Of these there has been a recent surge in PROMs research[31]. This is because PROMs lie at the heart of patient-centred care. There is no surprise that health-related quality of life measures such as the EuroQol are increasingly being employed to guide operative decision making in trauma[29,32]. Simultaneously, there is a trend towards including patients in setting research questions through priority setting partnerships[33], and patient and public involvement is now indispensable to healthcare research[34]. Cost-utility, the financial cost for health gain, is the variable that the National Institute for Health and Care Excellence (NICE) uses when forming guidelines for healthcare provision. It is thus very important that orthopaedic surgeons understand and incorporate cost-utility analysis in their research[35].

Variables used to evaluate an intervention are usually divided into outcome measures, process measures, and balancing measures[5,36]. Outcome measures monitor how a system is performing, process measures assess the implementation of an intervention, and balancing measures assess unintended consequences of the intervention.

Once outcome measures are identified and data is collected, analysis of the data is required to evaluate change.

Evaluating change

Operative intervention: Analysing change following operative intervention forms the basis of retrospective and prospective research studies. The level of evidence for a given study depends on a multitude of factors, most importantly study design[37]. There are three types of outcome variables: Continuous (e.g., operative time), categorical (e.g., presence or absence of a complication), and time-to-event (e.g., time to revision of a joint replacement). Statistical tests comparing outcomes consider the type of variable and can include parametric (t-test) and non-parametric (Mann-Whitney) tests, crosstabs (e.g., Chi-squared test and Fischer’s test), and survival analysis. These tests usually output a significance value (P-value) which is a measure of the likelihood that the result was due to chance.

Increased focus is being placed on the minimal clinically important difference - the smallest change in an outcome that a patient would identify as important, and which would usually indicate a change in patient management. Even a very small change can be shown to be statistically significant with a large enough sample size, but this may not be important. There is significant variation in the reporting of sample size calculations in orthopaedic literature[38] and until recently, reporting guidelines were lacking. Adoption of the DELTA2 guidance on choosing a target difference and reporting sample size in RCTs should improve this[39].

RCTs are considered the gold-standard hypothesis-testing study design. This is mainly because they allow for controlling of confounding variables that complicate observational studies. Over the last decade there has been a surge in trauma trials on an international scale, starting with the CRASH-2 trial on the effectiveness of tranexamic acid in trauma[40]. Other large-scale randomised trials have followed suit, investigating fixation of intracapsular neck of femur fractures[41], fixation of distal radius fractures[42] and ongoing research on the optimal timing of hip fracture surgery[43] to mention a few.

Although RCTs are excellent for answering certain research questions, retrospective studies remain indispensable. In the era of information technology, ‘Big Data’ is becoming ubiquitous[44]. Using Big Data to identify research questions, guide efficient targeting of resources and subsequently address these questions with randomised trials may not be the exception in a few years. It is definitely appearing promising so far[29]. One major limitation that will need to be addressed in future if RCTs are to output the highest quality data is surgeon equipoise. Surgeons are rarely in true equipoise and they usually have a clear idea of what management option is the best for a given patient. Although few would question the importance of decision making in surgery, it can present an obstacle when patient randomisation is required[45]. This must be addressed through improved surgeon education and standardised randomisation processes.

Time-series analysis: A toolbox for detecting change: Many quality improvement projects evaluate the effectiveness of an intervention by collecting data over time. Data can be graphically displayed as control charts, also known as Shewart charts. They are a statistical process control tool used to determine whether a system is in control and provide immediate feedback about performance[46].

Orthopaedic surgeons may be more familiar with audit cycles. Audit is a framework of quality improvement where performance is compared to a published standard[47]. Part of this process includes introducing an intervention and assessing its effectiveness by comparing performance before and after the intervention by simple statistical group tests. Although ubiquitous in clinical orthopaedics and indeed in all medical specialties, such approaches are sensitive to secular (background) trends. Interrupted time-series (ITS) analysis is a useful tool for evaluating the effectiveness of interventions where data is collected at several time-points before and after the intervention to determine whether any change could be explained by secular trends[48]. Cochrane recommends this tool to evaluate interventions[49] and several recent orthopaedic studies have employed this technique[50,51].

ITS does not come without limitations, and is known to display bias for detecting change at the time of the studied intervention where other changes at different time-points may be equally, if not more important[52,53]. Segmented linear regression models have been developed for evaluating change in retrospective studies by enabling more than one linear segment to describe the periods before and after an intervention. A recent study employing this technique revealed that improvements in time to surgery and 30-d mortality following hip fracture over a 6-year period were likely the result of a combination of surgical, anaesthetic, and procedural improvements over time, rather than due to the introduction of a dedicated hip fracture unit[53] (Figure 3). Future work is required to determine the optimal way to describe retrospective time-series: How many linear segments should be used, and how to best model binary outcomes.

Figure 3
Figure 3 Time to surgery for neck of femur fractures. The vertical dashed line marks the onset of a dedicated hip fracture unit. The line-plateau is the best-fitting linear model for the entire period: the line has equation y = −0.0414t + 40.1868; plateau at y = 24.7033 reached after 375 d. The initial drop may be related to the introduction of the Best Practice Tariff. The hip fracture unit did not significantly affect time to surgery[53].
CONCLUSION

Learning and change are integral to quality improvement and surgical education, and strongly influence the development of our specialty. The orthopaedic community has seen several improvements in PROMs research, learning curve analysis, randomised trial design, and time-series analysis.

Future work is required to improve and standardise learning variables and formally implement simulation in orthopaedic residency education. Global collaborative research networks are developing but integrating randomised trials with Big Data on an international scale to improve orthopaedics will require a concerted effort.

Footnotes

Manuscript source: Invited manuscript

Specialty type: Orthopedicst

Country of origin: United Kingdom

Peer-review report classification

Grade A (Excellent): 0

Grade B (Very good): B

Grade C (Good): 0

Grade D (Fair): 0

Grade E (Poor): 0

P-Reviewer: Tangtrakulwanich B S-Editor: Gong ZM L-Editor: Wang TQ E-Editor: Liu MY

References
1.  Scally G, Donaldson LJ. The NHS's 50 anniversary. Clinical governance and the drive for quality improvement in the new NHS in England. BMJ. 1998;317:61-65.  [PubMed]  [DOI]
2.  Barratt H, Turner S, Hutchings A, Pizzo E, Hudson E, Briggs T, Hurd R, Day J, Yates R, Gikas P, Morris S, Fulop NJ, Raine R. Mixed methods evaluation of the Getting it Right First Time programme - improvements to NHS orthopaedic care in England: study protocol. BMC Health Serv Res. 2017;17:71.  [PubMed]  [DOI]
3.  Nousiainen MT, Mironova P, Hynes M, Glover Takahashi S, Reznick R, Kraemer W, Alman B, Ferguson P, Safir O, Sonnadara R, Murnaghan J, Ogilvie-Harris D, Theodoropoulos J, Hall J, Syed K, Howard A, Ford M, Daniels T, Dwyer T, Veillette C, Wadey V, Narayanan U, Yee A, Whyne C. Eight-year outcomes of a competency-based residency training program in orthopedic surgery. Med Teach. 2018;1-13.  [PubMed]  [DOI]
4.  Sedrakyan A, Paxton EW, Phillips C, Namba R, Funahashi T, Barber T, Sculco T, Padgett D, Wright T, Marinac-Dabic D. The International Consortium of Orthopaedic Registries: overview and summary. J Bone Joint Surg Am. 2011;93 Suppl 3:1-12.  [PubMed]  [DOI]
5.  Wolfstadt JI, Ward SE, Kim S, Bell CM. Improving Care in Orthopaedics: How to Incorporate Quality Improvement Techniques into Surgical Practice. J Bone Joint Surg Am. 2018;100:1791-1799.  [PubMed]  [DOI]
6.  Berwick DM. A primer on leading the improvement of systems. BMJ. 1996;312:619-622.  [PubMed]  [DOI]
7.  Brydges R, Butler D. A reflective analysis of medical education research on self-regulation in learning and practice. Med Educ. 2012;46:71-79.  [PubMed]  [DOI]
8.  Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37:830-837.  [PubMed]  [DOI]
9.  Ramsay CR, Grant AM, Wallace SA, Garthwaite PH, Monk AF, Russell IT. Assessment of the learning curve in health technologies. A systematic review. Int J Technol Assess Health Care. 2000;16:1095-1108.  [PubMed]  [DOI]
10.  Alvand A, Logishetty K, Middleton R, Khan T, Jackson WF, Price AJ, Rees JL. Validating a global rating scale to monitor individual resident learning curves during arthroscopic knee meniscal repair. Arthroscopy. 2013;29:906-912.  [PubMed]  [DOI]
11.  Chang J, Banaszek DC, Gambrel J, Bardana D. Global Rating Scales and Motion Analysis Are Valid Proficiency Metrics in Virtual and Benchtop Knee Arthroscopy Simulators. Clin Orthop Relat Res. 2016;474:956-964.  [PubMed]  [DOI]
12.  Pusic MV, Kessler D, Szyld D, Kalet A, Pecaric M, Boutis K. Experience curves as an organizing framework for deliberate practice in emergency medicine learning. Acad Emerg Med. 2012;19:1476-1480.  [PubMed]  [DOI]
13.  Singer JD, Willett JB. Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. 1st ed New York; Oxford University Press. 2003. .  [PubMed]  [DOI]
14.  Ramsay CR, Grant AM, Wallace SA, Garthwaite PH, Monk AF, Russell IT. Statistical assessment of the learning curves of health technologies. Health Technol Assess. 2001;5:1-79.  [PubMed]  [DOI]
15.  Ekhtiari S, Horner NS, Bedi A, Ayeni OR, Khan M. The Learning Curve for the Latarjet Procedure: A Systematic Review. Orthop J Sports Med. 2018;6:2325967118786930.  [PubMed]  [DOI]
16.  Valsamis EM, Chouari T, O'Dowd-Booth C, Rogers B, Ricketts D. Learning curves in surgery: variables, analysis and applications. Postgrad Med J. 2018;94:525-530.  [PubMed]  [DOI]
17.  Valsamis EM, Golubic R, Glover TE, Husband H, Hussain A, Jenabzadeh AR. Modeling Learning in Surgical Practice. J Surg Educ. 2018;75:78-87.  [PubMed]  [DOI]
18.  Valsamis EM, Ricketts D, Hussain A, Jenabzadeh AR. Imageless navigation total hip arthroplasty - an evaluation of operative time. SICOT J. 2018;4:18.  [PubMed]  [DOI]
19.  Hay S, Kulkarni R, Watts A, Stanley D, Trail I, Van Rensburg L, Little C, Samdanis V, Jenkins P, Eames M, Phadnis J, Ali A, Rangan A, Drew S, Amirfeyz R, Conboy V, Clark D, Brownson P, Connor C, Jones V, Tennent D, Falworth M, Thomas M, Rees J. The Provision of Primary and Revision Elbow Replacement Surgery in the NHS. Shoulder Elbow. 2018;10:S5-S12.  [PubMed]  [DOI]
20.  Atesok K, Mabrey JD, Jazrawi LM, Egol KA. Surgical simulation in orthopaedic skills training. J Am Acad Orthop Surg. 2012;20:410-422.  [PubMed]  [DOI]
21.  Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in General Surgery.  Available from: https://www.unitypoint.org/desmoines/filesimages/Residency/Surgery/general_surgery_program_requirements.pdf.  [PubMed]  [DOI]
22.  Franzeck FM, Rosenthal R, Muller MK, Nocito A, Wittich F, Maurus C, Dindo D, Clavien PA, Hahnloser D. Prospective randomized controlled trial of simulator-based versus traditional in-surgery laparoscopic camera navigation training. Surg Endosc. 2012;26:235-241.  [PubMed]  [DOI]
23.  Middleton RM, Alvand A, Garfjeld Roberts P, Hargrove C, Kirby G, Rees JL. Simulation-Based Training Platforms for Arthroscopy: A Randomized Comparison of Virtual Reality Learning to Benchtop Learning. Arthroscopy. 2017;33:996-1003.  [PubMed]  [DOI]
24.  Bhattacharyya R, Sugand K, Al-Obaidi B, Sinha I, Bhattacharya R, Gupte CM. Trauma simulation training: a randomized controlled trial -evaluating the effectiveness of the Imperial Femoral Intramedullary Nailing Cognitive Task Analysis (IFINCTA) tool. Acta Orthop. 2018;89:689-695.  [PubMed]  [DOI]
25.  Morgan M, Aydin A, Salih A, Robati S, Ahmed K. Current Status of Simulation-based Training Tools in Orthopedic Surgery: A Systematic Review. J Surg Educ. 2017;74:698-716.  [PubMed]  [DOI]
26.  Kellet C. The Role of Simulation. J Trauma Orthop. 2018;6:78-79.  [PubMed]  [DOI]
27.  Solberg LI, Mosser G, McDonald S. The three faces of performance measurement: improvement, accountability, and research. Jt Comm J Qual Improv. 1997;23:135-147.  [PubMed]  [DOI]
28.  Royal College of Physicians. National Hip Fracture Database (NHFD) Annual Report 2017.  Available from: https://nhfd.co.uk/files/2017ReportFiles/NHFD-AnnualReport2017.pdf.  [PubMed]  [DOI]
29.  Costa ML, Griffin XL, Achten J, Metcalfe D, Judge A, Pinedo-Villanueva R, Parsons N. World Hip Trauma Evaluation (WHiTE): framework for embedded comprehensive cohort studies. BMJ Open. 2016;6:e011679.  [PubMed]  [DOI]
30.  Lübbeke A. Research methodology for orthopaedic surgeons, with a focus on outcome. EFORT Open Rev. 2018;3:160-167.  [PubMed]  [DOI]
31.  Joeris A, Knoll C, Kalampoki V, Blumenthal A, Gaskell G. Patient-reported outcome measurements in clinical routine of trauma, spine and craniomaxillofacial surgeons: between expectations and reality: a survey among 1212 surgeons. BMJ Open. 2018;8:e020629.  [PubMed]  [DOI]
32.  Ulvik A, Kvåle R, Wentzel-Larsen T, Flaatten H. Quality of life 2-7 years after major trauma. Acta Anaesthesiol Scand. 2008;52:195-201.  [PubMed]  [DOI]
33.  Fernandez MA, Arnel L, Gould J, McGibbon A, Grant R, Bell P, White S, Baxter M, Griffin X, Chesser T, Keene D, Kearney RS, White C, Costa ML. Research priorities in fragility fractures of the lower limb and pelvis: a UK priority setting partnership with the James Lind Alliance. BMJ Open. 2018;8:e023301.  [PubMed]  [DOI]
34.  National Institute for Health Research. Going the extra mile: improving the nation’s health and wellbeing through public involvement in research. 2015.  Available from: https://www.nihr.ac.uk/patients-and-public/documents/Going-the-Extra-Mile.pdf.  [PubMed]  [DOI]
35.  Costa M. The Price to Pay. J Trauma Orthop. 2018;6:60-61.  [PubMed]  [DOI]
36.  Chiu YJ, Chung HH, Yeh CH, Cheng JT, Lo SH. Improvement of insulin resistance by Chlorella in fructose-rich chow-fed rats. Phytother Res. 2011;25:1306-1312.  [PubMed]  [DOI]
37.  Howick J, Chalmers I, Glasziou P, Greenhalgh P, Heneghan C, Liberati A. The Oxford 2011 Levels of Evidence. 2011. .  [PubMed]  [DOI]
38.  Sabharwal S, Patel NK, Holloway I, Athanasiou T. Sample size calculations in orthopaedics randomised controlled trials: revisiting research practices. Acta Orthop Belg. 2015;81:115-122.  [PubMed]  [DOI]
39.  Cook JA, Julious SA, Sones W, Hampson LV, Hewitt C, Berlin JA, Ashby D, Emsley R, Fergusson DA, Walters SJ, Wilson ECF, Maclennan G, Stallard N, Rothwell JC, Bland M, Brown L, Ramsay CR, Cook A, Armstrong D, Altman D, Vale LD. DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial. Trials. 2018;19:606.  [PubMed]  [DOI]
40.  Roberts I, Shakur H, Coats T, Hunt B, Balogun E, Barnetson L, Cook L, Kawahara T, Perel P, Prieto-Merino D, Ramos M, Cairns J, Guerriero C. The CRASH-2 trial: a randomised controlled trial and economic evaluation of the effects of tranexamic acid on death, vascular occlusive events and transfusion requirement in bleeding trauma patients. Health Technol Assess. 2013;17:1-79.  [PubMed]  [DOI]
41.  Fixation using Alternative Implants for the Treatment of Hip fractures (FAITH) Investigators. Fracture fixation in the operative management of hip fractures (FAITH): an international, multicentre, randomised controlled trial. Lancet. 2017;389:1519-1527.  [PubMed]  [DOI]
42.  Costa ML, Achten J, Parsons NR, Rangan A, Edlin RP, Brown J, Lamb SE. UK DRAFFT - a randomised controlled trial of percutaneous fixation with kirschner wires versus volar locking-plate fixation in the treatment of adult patients with a dorsally displaced fracture of the distal radius. BMC Musculoskelet Disord. 2011;12:201.  [PubMed]  [DOI]
43.  Hip Fracture Accelerated Surgical Treatment and Care Track (HIP ATTACK) Investigators. Accelerated care versus standard care among patients with hip fracture: the HIP ATTACK pilot trial. CMAJ. 2014;186:E52-E60.  [PubMed]  [DOI]
44.  Perry D. Big data – what it does, and what it doesn’t. J Trauma Orthop. 2018;6:52-53.  [PubMed]  [DOI]
45.  Ollivere B. Equipoise, ethics, and offering patients participation in studies. Bone Jt 360. 2018;.  [PubMed]  [DOI]
46.  Mohsin M. Anti-smoking campaign in Multan, Pakistan. East Mediterr Health J. 2005;11:1110-1114.  [PubMed]  [DOI]
47.  Fowkes FG. Medical audit cycle. A review of methods and research in clinical practice. Med Educ. 1982;16:228-238.  [PubMed]  [DOI]
48.  Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2017;46:348-355.  [PubMed]  [DOI]
49.  Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011.  Available from: http://handbook-5-1.cochrane.org/.  [PubMed]  [DOI]
50.  McKirdy A, Imbuldeniya AM. The clinical and cost effectiveness of a virtual fracture clinic service: An interrupted time series analysis and before-and-after comparison. Bone Joint Res. 2017;6:259-269.  [PubMed]  [DOI]
51.  Matharu GS, Hunt LP, Murray DW, Howard P, Pandit HG, Blom AW, Bolland B, Judge A. Is the rate of revision of 36 mm metal-on-metal total hip arthroplasties with Pinnacle acetabular components related to the year of the initial operation? an interrupted time-series analysis using data from the National Joint Registry for England and Wales. Bone Joint J. 2018;100-B:33-41.  [PubMed]  [DOI]
52.  Fretheim A, Tomic O. Statistical process control and interrupted time series: a golden opportunity for impact evaluation in quality improvement. BMJ Qual Saf. 2015;24:748-752.  [PubMed]  [DOI]
53.  Valsamis EM, Ricketts D, Husband H, Rogers BA. Segmented Linear Regression Models for Assessing Change in Retrospective Studies in Healthcare. Comput Math Methods Med. 2019;2019:9810675.  [PubMed]  [DOI]