Minireviews Open Access
Copyright ©The Author(s) 2020. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Orthop. Dec 18, 2020; 11(12): 584-594
Published online Dec 18, 2020. doi: 10.5312/wjo.v11.i12.584
Technological developments enable measuring and using patient-reported outcomes data in orthopaedic clinical practice
David F Hamilton, Johannes M Giesinger, Karlmeinrad Giesinger
David F Hamilton, School of Health and Social Care, Edinburgh Napier University, Edinburgh EH114BN, United Kingdom
Johannes M Giesinger, University Hospital of Psychiatry II, Medical University of Innsbruck, Innsbruck 6020, Austria
Karlmeinrad Giesinger, Department of Orthopaedics and Traumatology, Kantonsspital St Gallen, St Gallen 9000, Switzerland
ORCID number: David F Hamilton (0000-0001-9060-9255); Johannes M Giesinger (0000-0003-2744-5950); Karlmeinrad Giesinger (0000-0002-2707-0020).
Author contributions: All authors contributed to determining the scope and purpose of this review. Hamilton DF drafted the initial manuscript which was then developed by all authors. All authors critically revised the text, and all approved the final submission.
Conflict-of-interest statement: The authors declare no conflicts of interest.
Open-Access: This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/Licenses/by-nc/4.0/
Corresponding author: David F Hamilton, PhD, Lecturer, School of Health and Social Care, Edinburgh Napier University, Sighthill Campus, Edinburgh EH114BN, United Kingdom. d.hamilton@napier.ac.uk
Received: April 28, 2020
Peer-review started: April 28, 2020
First decision: September 11, 2020
Revised: October 30, 2020
Accepted: November 11, 2020
Article in press: November 11, 2020
Published online: December 18, 2020

Abstract

Patient-reported outcomes measures form the backbone of outcomes evaluation in orthopaedics, with most of the literature now relying on these scoring tools to measure change in patient health status. This patient-reported information is increasingly collected routinely by orthopaedic providers but use of the data is typically restricted to academic research. Developments in electronic data capture and the outcome tools themselves now allow use of this data as part of the clinical consultation. This review evaluates the role of patient reported outcomes data as a tool to enhance daily orthopaedic clinical practice, and documents how develop-ments in electronic outcome measures, computer-adaptive questionnaire design and instant graphical display of questionnaire can facilitate enhanced patient-clinician shared decision making.

Key Words: Patient reported outcome measures, Electronic patient-reported outcome, Computer adaptive design, Outcomes evaluation

Core Tip: Utilising modern information technology, data collection, processing and intuitive graphical data display in real-time, electronic patient-reported outcome assessment can be implemented in daily clinical practice.



INTRODUCTION

Patient-reported outcome measures (PROMs) are widely used in orthopaedic research as they provide important and detailed information on patients' perception of symptoms and function in everyday life[1]. These metrics are central to evaluating success in orthopaedics as factors such as pain and satisfaction are only accessible by self-report. Most orthopaedic units collect these to some degree and clinical staff will be familiar with these scores from the literature if not their personal practice. In the main, outcomes are assessed via pen-and-paper questionnaires, but electronic PRO questionnaire administration is increasingly employed in clinical studies and more recently also in daily clinical practice[2].

PROMs allow for objective measurement of the patient’s subjective view of their health status[3] which complement other clinical or image-based evaluations. In clinical practice, numerous benefits have been cited for using PROM data, including enhanced screening, diagnosis, and longitudinal monitoring of conditions, along with the promotion of patient centred care[4-6]. Greenhalgh et al[7] suggest that collecting and using PROM data, along with clear feedback and dissemination of this information, will stimulate and incentivise health professionals and ultimately healthcare providers to provide better care[7]. The challenge though is how to take the mean outcome score from a patient questionnaire and modify care delivery.

Today’s patients certainly want greater involvement in decisions regarding their care[8]. Shared decision-making i.e. the conversation that happens between a patient and health professional to reach a healthcare choice together, has been advocated and been embedded in clinical practice for some time. Patients should always make the ultimate decision - especially in elective surgery - about their care but can only do so if they are fully informed about the options and are encouraged to participate[9]. In orthopaedics, surveys suggest that surgeons are strong advocates of shared decision-making, but also that they have concerns regarding logistics, practicality and in particular burden to the clinical workflow relating to engaging the patient and providing in-depth personalised feedback and discussion[10]. Being able to interpret how well an individual is performing relative to the wider knee replacement population can form a meaningful part of the consultation process, and potentially influence the overall experience of the surgical pathway[11].

Use of patient reported outcomes data as part of clinical care

To date, the integration of PROM data in routine orthopaedic care is less evolved than in other specialties such as oncology. Here several clinical studies have emphasized the positive effects of regular collection and use of this data in patient management[12-14]. Regular collection of health data has demonstrated a positive impact on symptom control, overall quality of life and emotional well-being without burdening clinical management. Physicians report that over time the clinical use of PROM information improved their ability to recognize symptoms and health issues and enabled crucial conversation with the patient[15-17]. These positive effects should be similarly replicable in the orthopaedic field.

The routine collection of PRO data in daily practice can provide valuable information for research purposes. The collection of pre- and post-operation PRO questionnaire data is routine for many orthopaedic interventions in many healthcare systems. In fact, combined PRO assessments of general health and joint-specific outcomes have been integrated with national registries in the United Kingdom, Sweden, and the Netherlands and numerous local or regional registries[18]. This data that evaluates all patients at scale, can add valuable real-world context to help interpret the results of tightly controlled studies that may, for example, not encompass the typical case-mix presentation in routine practice. However, these routinely collected registry data are typically not directly available for the clinical teams, but only used for later analysis for research purposes or quality assurance. Thus, analyses of collected PROMs is generally at the population level and published primarily for the benefit of academic readers. Only summed information is occasionally fed back into clinical practice at the clinical governance level in the Figure 1. Therefore, the time and resources spent to collect PROM data do not help inform the individual patient or clinician and the full potential of these data is poorly exploited. Clinical assessment is separately based on the clinician’s individual discussion with the patient as to their symptomology and medical history and does not incorporate the standardized information provided by patients via PROMs. The latter is particularly useful for the longitudinal evaluation of symptoms and functional levels, as patients may have difficulties to provide valid information on the degree of deterioration or improve-ment of symptoms over time and clinical documentation may capture such change insufficiently; depending on the individual viewpoints, expectations and experience of the treating clinician. Alongside a physical analysis, imaging and potentially lab tests, PRO data may therefore have a key role in driving clinical management.

Figure 1
Figure 1 Current usage of patient information data flows.

PROM data can be used to promote patient centred care[19] and to facilitate patient and clinician’s understanding of how different treatments affect patient functioning and wellbeing over time, informing treatment decision making and, most importantly, improving expectation management[20]. Feeding back PRO data at the level of the individual patient during the consultation combines these two, usually separate streams of information in the Figure 2.

Figure 2
Figure 2 Potential usage of patient information data flows.

We are increasingly using prediction models for estimating eventual post-operative scores as part of pre-operative surgical expectation management. Unlike paper-pencil assessments, electronic data capture provides immediate availability of all collected data in a database that may be used for establishing and constant updating of regression models that allow for a prognosis of treatment outcome for individual patients based on baseline PRO assessments and patient characteristics, such as age, body mass index, or comorbidities[21,22]. Such models based on the data collected at an individual centre or taken from the literature may be helpful when dealing with interpreting individual PROM scores. Prediction models based on the routine data of a specific centre, may be more precise for predicting scores of new patients at that centre than models from the literature, as in such a scenario the sample on which the model is based is more similar to the patients for which predictions are desired.

Contextualising PROM data for the patient in terms of that individual’s position within the wider expected levels for people of their specific demographics, at a particular point in time, makes the PROM data relevant to the presenting complaint and planned management discussion. It facilitates the patient-clinician discussion of symptomology, expected trajectories and clinical management options; directly feeding into clinical management and outcomes. The use of these reports during the consultation can help to identify important health issues that might otherwise not be disclosed during the consultation[12]. Engaging the patient has the further benefit of enhancing inclusion and shared decision making contributes to patient satisfaction with clinical services.

While incorporating patient outcome scores in this manner as part of the clinical consultation has been an attractive idea for quite some time, it has only recently become a feasible proposition with a transformative step change in the way we can collect and use information technology in the clinic.

Electronic proms

As noted, PROMs are traditionally pen and paper-based, collected en-masse and manually uploaded for processing at the national, institutional or project level. Logistics can be challenging, and time consuming, often delayed, and manual data entry opens further issues with imputation errors.

Electronic PROMs (ePROMs) collected via a tablet computer connected to a local network to collate, process and store the data, clearly, offers several advantages; electronic administration of outcome questionnaires using tablet PCs or smartphones improved feasibility of data collection, decreased responder burden and sophisticated survey administration with real-time processing[23-25]. The International Society for Pharmacoeconomics and Outcomes Research PRO Mixed Modes Task Force[26] highlights additional advantages of using electronic data collection including avoidance of secondary data entry errors, date and time stamping, edit checks, and more accurate and complete data. Further, electronic systems allow remote questionnaire completion (e.g., at home), and automatic messages that remind patients of due assessments[27].

But most importantly, ePROM collection is necessary for using these data in the clinical consultation with the individual patient, as data collected on paper can hardly be made available timely in an accessible format without electronic means. In addition, regular PRO monitoring from home allows patients to observe their recovery process independently, to assess how they compare with a similar sample population and to remain in constant contact with surgical and therapy centres between outpatient appointments via increasingly available institutional web portals. This supports self-management by helping patients to better assess and contextualise their state of health/recovery. In a next step, if appropriate, patients can be invited to attend for radiographs or clinical review based on changes in their remote data presentation.

The successful use of PROMs in routine care in this way clearly requires careful planning and adequate logistics[28-30]. Implementation is supported by international guidelines such as the “User’s Guide for Implementing Patient-Reported Outcomes Assessment in Clinical Practice” published by the International Society for Quality of Life Research[31]. In detail, this process includes tasks such as: development of training and information materials regarding the use of PRO data during the clinical encounter, identifying the appropriate PRO measure that allows valid and reliable assessment of the relevant health issues, definition of time points during the treatment and disease trajectory when assessments should take place, and establishing pathways for linking clinically relevant scores to interventions and referrals.

Computer-adaptive measures and ‘intelligent’ questionnaires

The technological move to electronic data collection also facilitates a wider change in the use of PROMs, offering enhanced “intelligent” questionnaires. Traditionally, patient reported outcomes are “static” questionnaires, where all patients complete a defined set of the same fixed questions[32]. The questionnaire tools themselves can be lengthy, as many questions are required to allow accurate measurement[33]. For example, the well-used Knee injury and Osteoarthritis Outcome Score comprises 42 questions[34]. Lengthy questionnaires can cause high drop-off rates, leading in turn to fewer complete PROMs[35,36]. This static “one-size fits all” structure is suboptimal for multiple reasons. The questionnaire length can pose a burden to the patient as these measures require a considerable number of questions to cover the whole measurement range of the outcome parameter of interest (pain, functional limitation etc.). Following this, patients often find themselves confronted with questions that are repetitive or that are not appropriate to their current condition (e.g., inquiring as to sporting activities they could clearly not partake in). Such inappropriate questions can be irritating to complete and impair compliance. Frustratingly, these “inappropriate” questions provide no or little additional information to the clinician nor the researcher interpreting the questionnaire (e.g., if a patient reports barely being able to walk, further questions on various sports activities provide little or no further information).

A major step forwards in addressing these issues is the development of computer-adaptive test (CAT) measures[37-39]. Computer-adaptive testing uses large sets (or item banks) of potential questions that cover the spectrum of the issue in question and an algorithm for tailoring individual sets of questions. Based on the response to the initial item the CAT algorithm calculates a first estimate and confidence interval of the outcome score and selects the next most appropriate item (that offers the highest item information in the range of that estimated score) to be administered to specific the patient. This procedure continues until a predefined measurement precision has been reached or a maximum number of items have been asked. In this way, patients only need to answer a fraction of the possible questions – and not necessarily the same questions that others will complete. Thanks to the underlying mathematical model, the scores calculated by different questions within the same item bank are perfectly comparable.

The underlying probabilistic model of a CAT tool is based on item response theory (IRT). An essential prerequisite for running these models is that the outcome questionnaire is based on a strictly unidimensional item sets (i.e. questions that measure the same thing; be that pain or function or general health)[40]. IRT frameworks have been developed in various fields such as cancer and respiratory medicine[41-44]. Since internal consistency (unidimensionality) of orthopaedic questionnaires tends to be high they naturally lend themselves to IRT model application and CAT designs[45]. Although still a relatively new field in orthopaedics, a few studies have applied IRT modelling approaches to orthopaedic outcome measures for functional status, pain, and rehabilitation outcome[46-51] including studies on various aspects of CAT measures[32,52,53].

The largest initiative on the development of item banks for health outcomes measurement is the United States-led Patient-reported Outcome Measure Information System (PROMIS) group. PROMIS has released a substantial number of item banks, including an item bank for the assessment of physical functioning in all types of patient groups[54-56]. The PROMIS measures have been tested and validated in large reference populations, making them suitable for research on different health conditions[57]. For orthopaedic outcome research PROMIS provides useful measures for physical function (including versions for upper extremity, mobility, and mobility aid users)[54,58]. To date there is no joint-specific PROMIS tool available. The computer-adaptive pilot version of the FJS-12[32] for hip and knee assessment is the first computer-adaptive joint-specific measure for orthopaedics. However, because of its limited item bank, this pilot version represents primarily a proof of principle rather than an elaborate CAT instrument with an extensive item bank.

Clearly, CAT based outcome measures require infrastructure; hardware such as a computer tablet interface and a software package to run the questionnaires electronically, to manage item banks and to employ the CAT algorithm for item administration. As IT infrastructure for electronic data capture is increasingly available in hospital settings, the use of CAT measures becomes more feasible and efficient[59].

Compared to static versions, CAT questionnaires take a fraction of the time to complete and, because of the large item banks in the background, can be even more reliable, valid and sensitive to change compared to their paper-based counterparts. Advantages of the CAT approach include instant score calculation and no missing data. The beneficial use of this technology is perhaps most readily seen in clinical trials, where efficiency is enhanced by the direct entry of data by the patient, reducing transcription error rates in processing but most importantly in the enhanced accuracy in terms of confidence intervals facilitating reduction in required sample sizes and costs[60,61].

Scoring, interpretation, and feedback

For PROMs to contribute meaningfully to clinical care, patients and clinicians must be able to readily understand and interpret these data. Unfortunately, interpretation of the results presented by these instruments is not always easy, especially when different instruments are used to measure similar things, and each instrument is scored and scaled differently. For example, the score range of the Oxford Hip Score[62] ranges from 0 to 48 points, whereas the FJS-12[63] is reported on a 0-100-point scale. Scale direction is not consistent; some outcome tools report higher scores as “better” health whereas others indicate problems. The Western Ontario MacMaster score[64] is an example of a tool that is inversely scored, as it is a measure of symptom burden. This can make it difficult to compare information across questionnaires in daily practice. Both patients and clinicians report that variation in PROM scoring, scaling and presentation poses obstacles to interpretation and application[4].

Questionnaire scores taken at isolated time points are somewhat abstract and additional information is required from the literature or from manuals to interpret these. Various approaches have been taken to solve this; wider comparison can be straightforward by simply contrasting the individual score to the population mean or converting the score to percentiles. A further method is to convert scores to a standardized T-score metric with a fixed mean of 50 points and a standard deviation of 10 points. The T-score of an individual patient informs directly to what degree (measured in standard deviation units) the patient deviates from the mean of the reference population[11].

With today's widespread electronic data collection and processing, this data can be visualized immediately after the questionnaire has been completed. Using electronic data capture and display, patients can be presented with a normalised graph (targeted for their specific demographics) showing the expected change and standard deviations around the mean change following surgical interventions such as knee arthroplasty.

PROs are usually assessed repeatedly, prior to and following intervention, across multiple outpatient visits. As such it makes sense to present results longitudinally. This allows a quick overview as to the course of symptoms or functional impairments over time. Few studies have evaluated how to present outcome score results graphically[65,66]. These have shown that longitudinal line charts may be optimal for presenting individual patient data and that it is important to clearly indicate the scale direction (i.e., if high scores indicate poor or good health). Using color-coding or specific percentiles from reference populations can be integrated into charts to guide interpretation[67,68].

The transformation to T-values enables both patients and clinicians to understand the individual score results more easily, as results can be presented in the context of the results reported by others of a similar demographic and intervention. This feedback somewhat akin to the population height and weight growth charts that are routinely used in primary care, allows the patient to see what change would be expected for them in terms of outcomes that matter to them in Figure 3, and monitor their progress against their direct peers. Presenting the typical pre-operative scores also highlights the relative improvement that can be made from the patient’s individual starting point. This data should be captured sequentially, as it can be difficult to accurately deduce symptom burden retrospectively from memory and non-standardised clinic notes.

Figure 3
Figure 3 OKS T-scores feedback model stratified to patient demographics . Figure showing the longitudinal course of expected patient outcomes. The exemplary course (green line) of an individual patient undergoing total knee arthroplasty at 3 different points prior to and following surgery is presented in terms of t-scores against a background of an age and gender matched population. The dark grey area is the range for +/- 1 standard deviation of the appropriate matched population and the light grey area the range for +/- 2 standard deviation.
Challenges in moving to ePROMs

As with all new developments there is a reluctance to embrace new technology[3] and there are real challenges to address. There are issues to consider in terms of data security and privacy[23], as well as the feasibility of developing the local infrastructure required[26]. With regard to software there are essentially two possibilities, the necessary features can either be provided by an extension of the clinical information system, or stand-alone software developed specifically for the purpose of routine PRO monitoring may be used. Whereas the first option has the advantage of easier integration of PRO results into medical charts and linkage of clinical and PRO data, the latter option usually has more refined features for facilitating questionnaire administration and a more sophisticated presentation of results. Currently, it is also unclear whether the computer platforms and servers required increase the provider costs compared to employing a pen-and-paper model with manual transcription and upload. This may also depend on the individual institutional situation.

Electronic questionnaire administration requires the patient to have a basic level computer literacy. However, the last decade has seen more and more elderly people engaging with technology such as using smartphones and social media on a day-to-day basis. It has been suggested that vulnerable patients and elderly groups may struggle to fully engage with this technology. However, these patients currently engage in PROM surveys and in principle should be perfectly able to contribute data[69]. While the orthopaedic literature on implementation studies is still scarce, there are encouraging results, such as the findings by Slover et al[70] who reported question-naire completion rates of about 95% in osteoarthritis patients using a web portal. Further optimization of the graphical user interfaces may be required to promote wide uptake and to promote efficient usage of electronic questionnaires as in the early days of smartphones with full-field touchscreens. In practice, a person well-integrated within the clinic team could be designated as the local ePROM facilitator for introducing patients to the electronic assessments, responding to questions, and discussing patient concerns[71].

When migrating a PROM to an electronic format, great care needs to be taken that standardization of the questionnaire is maintained and no major changes in wording and formatting are introduced. Guidance on such migration and the assessment of equivalence of paper and electronic questionnaires has been developed by an ISPOR task force[72], which also highlights the type of evidence needed to demonstrate equivalence of both formats.

CONCLUSION

Utilising modern information technology, data collection, processing and intuitive graphical data display in real-time, electronic PRO assessment can be implemented in daily clinical practice. With home monitoring, the individual recovery process of the patients can be observed via an easy-to-use web portal and, if necessary, focused appointments can be made for clinically important parameters. With this transition to daily practice, PROMs expand from their original homeland of clinical studies and data registries and enter routine outpatient appointments to facilitate shared decision making, help manage patient expectations and complement follow-up.

Footnotes

Manuscript source: Invited manuscript

Specialty type: Orthopedics

Country/Territory of origin: United Kingdom

Peer-review report’s scientific quality classification

Grade A (Excellent): A

Grade B (Very good): 0

Grade C (Good): C

Grade D (Fair): D

Grade E (Poor): 0

P-Reviewer: Azimi P, Kocazeybek B, Widmer KH S-Editor: Zhang L L-Editor: A P-Editor: Xing YX

References
1.  Wright RW. Knee injury outcomes measures. J Am Acad Orthop Surg. 2009;17:31-39.  [PubMed]  [DOI]
2.  Dixon S, Bunker T, Chan D. Outcome scores collected by touchscreen: medical audit as it should be in the 21st century? Ann R Coll Surg Engl. 2007;89:689-691.  [PubMed]  [DOI]
3.  Hamilton DF, Giesinger JM, Giesinger K. It is merely subjective opinion that patient-reported outcome measures are not objective tools. Bone Joint Res. 2017;6:665-666.  [PubMed]  [DOI]
4.  Porter I, Gonçalves-Bradley D, Ricci-Cabello I, Gibbons C, Gangannagaripalli J, Fitzpatrick R, Black N, Greenhalgh J, Valderas JM. Framework and guidance for implementing patient-reported outcomes in clinical practice: evidence, challenges and opportunities. J Comp Eff Res. 2016;5:507-519.  [PubMed]  [DOI]
5.  Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract. 1999;5:401-416.  [PubMed]  [DOI]
6.  Valderas JM, Kotzeva A, Espallargues M, Guyatt G, Ferrans CE, Halyard MY, Revicki DA, Symonds T, Parada A, Alonso J. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res. 2008;17:179-193.  [PubMed]  [DOI]
7.  Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measures in routine clinical practice: lack of impact or lack of theory? Soc Sci Med. 2005;60:833-843.  [PubMed]  [DOI]
8.  Coulter A, Jenkinson C. European patients' views on the responsiveness of health systems and healthcare providers. Eur J Public Health. 2005;15:355-360.  [PubMed]  [DOI]
9.  Jayadev C, Khan T, Coulter A, Beard DJ, Price AJ. Patient decision aids in knee replacement surgery. Knee. 2012;19:746-750.  [PubMed]  [DOI]
10.  Adam JA, Khaw FM, Thomson RG, Gregg PJ, Llewellyn-Thomas HA. Patient decision aids in joint replacement surgery: a literature review and an opinion survey of consultant orthopaedic surgeons. Ann R Coll Surg Engl. 2008;90:198-207.  [PubMed]  [DOI]
11.  Hamilton DF, Giesinger JM, Patton JT, MacDonald DJ, Simpson AH, Howie CR, Giesinger K. Making the Oxford Hip and Knee Scores meaningful at the patient level through normative scoring and registry data. Bone Joint Res. 2015;4:137-144.  [PubMed]  [DOI]
12.  Rotenstein LS, Huckman RS, Wagle NW. Making Patients and Doctors Happier - The Potential of Patient-Reported Outcomes. N Engl J Med. 2017;377:1309-1312.  [PubMed]  [DOI]
13.  Howell D, Molloy S, Wilkinson K, Green E, Orchard K, Wang K, Liberty J. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann Oncol. 2015;26:1846-1858.  [PubMed]  [DOI]
14.  Basch E. Patient-Reported Outcomes - Harnessing Patients' Voices to Improve Clinical Care. N Engl J Med. 2017;376:105-108.  [PubMed]  [DOI]
15.  Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, Selby PJ. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol. 2004;22:714-724.  [PubMed]  [DOI]
16.  Kotronoulas G, Kearney N, Maguire R, Harrow A, Di Domenico D, Croy S, MacGillivray S. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? J Clin Oncol. 2014;32:1480-1501.  [PubMed]  [DOI]
17.  Detmar SB, Muller MJ, Schornagel JH, Wever LD, Aaronson NK. Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial. JAMA. 2002;288:3027-3034.  [PubMed]  [DOI]
18.  Rolfson O, Eresian Chenok K, Bohm E, Lübbeke A, Denissen G, Dunn J, Lyman S, Franklin P, Dunbar M, Overgaard S, Garellick G, Dawson J; Patient-Reported Outcome Measures Working Group of the International Society of Arthroplasty Registries. Patient-reported outcome measures in arthroplasty registries. Acta Orthop. 2016;87 Suppl 1:3-8.  [PubMed]  [DOI]
19.  Brundage MD, Smith KC, Little EA, Bantug ET, Snyder CF; PRO Data Presentation Stakeholder Advisory Board. Communicating patient-reported outcome scores using graphic formats: results from a mixed-methods evaluation. Qual Life Res. 2015;24:2457-2472.  [PubMed]  [DOI]
20.  Brundage M, Bass B, Jolie R, Foley K. A knowledge translation challenge: clinical use of quality of life data from cancer clinical trials. Qual Life Res. 2011;20:979-985.  [PubMed]  [DOI]
21.  Giesinger JM, Giesinger K, Federico B, Howie CD, Hamilton DF. Differences in case mix and outcomes between Swiss and Scottish total knee arthroplasty patients. Knee Surg Sports Traumatol Arthrosc. 2020;28:1797-1804.  [PubMed]  [DOI]
22.  Loth FL, Giesinger JM, Giesinger K, MacDonald DJ, Simpson AHRW, Howie CR, Hamilton DF. Impact of Comorbidities on Outcome after Total Hip Arthroplasty. J Arthroplasty. 2017;32:2755-2761.  [PubMed]  [DOI]
23.  Jones JB, Snyder CF, Wu AW. Issues in the design of Internet-based systems for collecting patient-reported outcomes. Qual Life Res. 2007;16:1407-1417.  [PubMed]  [DOI]
24.  Bennett AV, Jensen RE, Basch E. Electronic patient-reported outcome systems in oncology clinical practice. CA Cancer J Clin. 2012;62:337-347.  [PubMed]  [DOI]
25.  Snyder CF, Herman JM, White SM, Luber BS, Blackford AL, Carducci MA, Wu AW. When using patient-reported outcomes in clinical practice, the measure matters: a randomized controlled trial. J Oncol Pract. 2014;10:e299-e306.  [PubMed]  [DOI]
26.  Eremenco S, Coons SJ, Paty J, Coyne K, Bennett AV, McEntegart D; ISPOR PRO Mixed Modes Task Force. PRO data collection in clinical trials using mixed modes: report of the ISPOR PRO mixed modes good research practices task force. Value Health. 2014;17:501-516.  [PubMed]  [DOI]
27.  Girgis A, Durcinoska I, Levesque JV, Gerges M, Sandell T, Arnold A, Delaney GP; PROMPT-Care Program Group. eHealth System for Collecting and Utilizing Patient Reported Outcome Measures for Personalized Treatment and Care (PROMPT-Care) Among Cancer Patients: Mixed Methods Approach to Evaluate Feasibility and Acceptability. J Med Internet Res. 2017;19:e330.  [PubMed]  [DOI]
28.  Möhler R, Köpke S, Meyer G. Criteria for Reporting the Development and Evaluation of Complex Interventions in healthcare: revised guideline (CReDECI 2). Trials. 2015;16:204.  [PubMed]  [DOI]
29.  Basch E, Barbera L, Kerrigan CL, Velikova G. Implementation of Patient-Reported Outcomes in Routine Medical Care. Am Soc Clin Oncol Educ Book. 2018;38:122-134.  [PubMed]  [DOI]
30.  Holch P, Warrington L, Bamforth LCA, Keding A, Ziegler LE, Absolom K, Hector C, Harley C, Johnson O, Hall G, Morris C, Velikova G. Development of an integrated electronic platform for patient self-report and management of adverse events during cancer treatment. Ann Oncol. 2017;28:2305-2311.  [PubMed]  [DOI]
31.  International Society for Quality of Life Research  User’s guide to implementing patient-reported outcomes assessment in clinical practice. Version 2. 2015. Available from: https://www.isoqol.org/wp-content/uploads/2019/09/2015UsersGuide-Version2.pdf.  [PubMed]  [DOI]
32.  Giesinger JM, Kuster MS, Holzner B, Giesinger K. Development of a computer-adaptive version of the forgotten joint score. J Arthroplasty. 2013;28:418-422.  [PubMed]  [DOI]
33.  Kingsley C, Patel S. Patient-reported outcome measures and patient-reported experience measures. BJA Educ. 2017;17:137-144.  [PubMed]  [DOI]
34.  Roos EM, Lohmander LS. The Knee injury and Osteoarthritis Outcome Score (KOOS): from joint injury to osteoarthritis. Health Qual Life Outcomes. 2003;1:64.  [PubMed]  [DOI]
35.  Hamilton DF, Burnett R, Patton JT, MacPherson GJ, Simpson AHRW, Howie CR, Gaston P. Reduction in patient outcomes but implant-derived preservation of function following total knee arthroplasty: longitudinal follow-up of a randomized controlled trial. Bone Joint J. 2020;102-B:434-441.  [PubMed]  [DOI]
36.  De Faoite D. The advantages of electronic patient-reported measures and an example digital platform to collect ePROs after total knee arthroplasty. Medicine Access @ Point of Care. 2018.  [PubMed]  [DOI]
37.  Bjorner JB, Chang CH, Thissen D, Reeve BB. Developing tailored instruments: item banking and computerized adaptive assessment. Qual Life Res. 2007;16 Suppl 1:95-108.  [PubMed]  [DOI]
38.  Cook KF, O'Malley KJ, Roddey TS. Dynamic assessment of health outcomes: time to let the CAT out of the bag? Health Serv Res. 2005;40:1694-1711.  [PubMed]  [DOI]
39.   The PROMIS network Patient-Reported Outcomes Measurement Information System (PROMIS) 2011. Available from: http://www.nihpromis.org.  [PubMed]  [DOI]
40.  Hambleton R, Swaminathan H, Rogers H.   Fundamentals of item response theory. Newbury Park: Sage Publications, 1991.  [PubMed]  [DOI]
41.  Petersen MA, Groenvold M, Aaronson NK, Chie WC, Conroy T, Costantini A, Fayers P, Helbostad J, Holzner B, Kaasa S, Singer S, Velikova G, Young T; EORTC Quality of Life Group. Development of computerised adaptive testing (CAT) for the EORTC QLQ-C30 dimensions - general approach and initial results for physical functioning. Eur J Cancer. 2010;46:1352-1358.  [PubMed]  [DOI]
42.  Reeve BB, Hays RD, Bjorner JB, Cook KF, Crane PK, Teresi JA, Thissen D, Revicki DA, Weiss DJ, Hambleton RK, Liu H, Gershon R, Reise SP, Lai JS, Cella D; PROMIS Cooperative Group. Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Med Care. 2007;45:S22-S31.  [PubMed]  [DOI]
43.  Mitchell AJ, Smith AB, Al-salihy Z, Rahim TA, Mahmud MQ, Muhyaldin AS. Redefining diagnostic symptoms of depression using Rasch analysis: testing an item bank suitable for DSM-V and computer adaptive testing. Aust N Z J Psychiatry. 2011;45:846-852.  [PubMed]  [DOI]
44.  Stucky BD, Edelen MO, Sherbourne CD, Eberhart NK, Lara M. Developing an item bank and short forms that assess the impact of asthma on quality of life. Respir Med. 2014;108:252-263.  [PubMed]  [DOI]
45.  Impellizzeri FM, Mannion AF, Leunig M, Bizzini M, Naal FD. Comparison of the reliability, responsiveness, and construct validity of 4 different questionnaires for evaluating outcomes after total knee arthroplasty. J Arthroplasty. 2011;26:861-869.  [PubMed]  [DOI]
46.  Jette AM, Haley SM, Ni P, Moed R. Adaptive short forms for outpatient rehabilitation outcome assessment. Am J Phys Med Rehabil. 2008;87:842-852.  [PubMed]  [DOI]
47.  Ryser L, Wright BD, Aeschlimann A, Mariacher-Gehler S, Stucki G. A new look at the Western Ontario and McMaster Universities Osteoarthritis Index using Rasch analysis. Arthritis Care Res. 1999;12:331-335.  [PubMed]  [DOI]
48.  Wolfe F, Kong SX. Rasch analysis of the Western Ontario MacMaster questionnaire (WOMAC) in 2205 patients with osteoarthritis, rheumatoid arthritis, and fibromyalgia. Ann Rheum Dis. 1999;58:563-568.  [PubMed]  [DOI]
49.  Goetz C, Ecosse E, Rat AC, Pouchot J, Coste J, Guillemin F. Measurement properties of the osteoarthritis of knee and hip quality of life OAKHQOL questionnaire: an item response theory analysis. Rheumatology (Oxford). 2011;50:500-505.  [PubMed]  [DOI]
50.  Pollard B, Dixon D, Dieppe P, Johnston M. Measuring the ICF components of impairment, activity limitation and participation restriction: an item analysis using classical test theory and item response theory. Health Qual Life Outcomes. 2009;7:41.  [PubMed]  [DOI]
51.  Cook C, Hegedus E, Goode A, Mina C, Pietrobon R, Higgins LD. Relative validity of the modified American Shoulder and Elbow Surgeons (M-ASES) questionnaire using item response theory. Rheumatol Int. 2008;28:217-223.  [PubMed]  [DOI]
52.  Haley SM, Gandek B, Siebens H, Black-Schaffer RM, Sinclair SJ, Tao W, Coster WJ, Ni P, Jette AM. Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: II. Participation outcomes. Arch Phys Med Rehabil. 2008;89:275-283.  [PubMed]  [DOI]
53.  Hart DL, Mioduski JE, Stratford PW. Simulated computerized adaptive tests for measuring functional status were efficient with good discriminant validity in patients with hip, knee, or foot/ankle impairments. J Clin Epidemiol. 2005;58:629-638.  [PubMed]  [DOI]
54.  Hung M, Clegg DO, Greene T, Saltzman CL. Evaluation of the PROMIS physical function item bank in orthopaedic patients. J Orthop Res. 2011;29:947-953.  [PubMed]  [DOI]
55.  Rose M, Bjorner JB, Becker J, Fries JF, Ware JE. Evaluation of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS). J Clin Epidemiol. 2008;61:17-33.  [PubMed]  [DOI]
56.  Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, Amtmann D, Bode R, Buysse D, Choi S, Cook K, Devellis R, DeWalt D, Fries JF, Gershon R, Hahn EA, Lai JS, Pilkonis P, Revicki D, Rose M, Weinfurt K, Hays R; PROMIS Cooperative Group. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. J Clin Epidemiol. 2010;63:1179-1194.  [PubMed]  [DOI]
57.  Bingham CO 3rd, Bartlett SJ, Merkel PA, Mielenz TJ, Pilkonis PA, Edmundson L, Moore E, Sabharwal RK. Using patient-reported outcomes and PROMIS in research and clinical applications: experiences from the PCORI pilot projects. Qual Life Res. 2016;25:2109-2116.  [PubMed]  [DOI]
58.  Shim J, Hamilton DF. Comparative responsiveness of the PROMIS-10 Global Health and EQ-5D questionnaires in patients undergoing total knee arthroplasty. Bone Joint J. 2019;101-B:832-837.  [PubMed]  [DOI]
59.  Holzner B, Giesinger JM, Pinggera J, Zugal S, Schöpf F, Oberguggenberger AS, Gamper EM, Zabernigg A, Weber B, Rumpold G. The Computer-based Health Evaluation Software (CHES): a software for electronic patient-reported outcome monitoring. BMC Med Inform Decis Mak. 2012;12:126.  [PubMed]  [DOI]
60.  Fries JF, Krishnan E, Rose M, Lingala B, Bruce B. Improved responsiveness and reduced sample size requirements of PROMIS physical function scales with item response theory. Arthritis Res Ther. 2011;13:R147.  [PubMed]  [DOI]
61.  Giesinger JM, Kesterke N, Hamilton DF, Holzner B, Jost B, Giesinger K. Development of an item list to assess the forgotten joint concept in shoulder patients. BMC Musculoskelet Disord. 2015;16:67.  [PubMed]  [DOI]
62.  Dawson J, Fitzpatrick R, Murray D, Carr A. Questionnaire on the perceptions of patients about total knee replacement. J Bone Joint Surg Br. 1998;80:63-69.  [PubMed]  [DOI]
63.  Behrend H, Giesinger K, Giesinger JM, Kuster MS. The "forgotten joint" as the ultimate goal in joint arthroplasty: validation of a new patient-reported outcome measure. J Arthroplasty 2012; 27: 430-436. e1.  [PubMed]  [DOI]
64.  Bellamy N, Buchanan WW, Goldsmith CH, Campbell J, Stitt LW. Validation study of WOMAC: a health status instrument for measuring clinically important patient relevant outcomes to antirheumatic drug therapy in patients with osteoarthritis of the hip or knee. J Rheumatol. 1988;15:1833-1840.  [PubMed]  [DOI]
65.  Kuijpers W, Giesinger JM, Zabernigg A, Young T, Friend E, Tomaszewska IM, Aaronson NK, Holzner B. Patients' and health professionals' understanding of and preferences for graphical presentation styles for individual-level EORTC QLQ-C30 scores. Qual Life Res. 2016;25:595-604.  [PubMed]  [DOI]
66.  Bantug ET, Coles T, Smith KC, Snyder CF, Rouette J, Brundage MD; PRO Data Presentation Stakeholder Advisory Board. Graphical displays of patient-reported outcomes (PRO) for use in clinical practice: What makes a pro picture worth a thousand words? Patient Educ Couns. 2016;99:483-490.  [PubMed]  [DOI]
67.  Snyder CF, Smith KC, Bantug ET, Tolbert EE, Blackford AL, Brundage MD; PRO Data Presentation Stakeholder Advisory Board. What do these scores mean? Cancer. 2017;123:1848-1859.  [PubMed]  [DOI]
68.  Tucker G, Adams R, Wilson D. The case for using country-specific scoring coefficients for scoring the SF-12, with scoring implications for the SF-36. Qual Life Res. 2016;25:267-274.  [PubMed]  [DOI]
69.  Kesterke N, Egeter J, Erhardt JB, Jost B, Giesinger K. Patient-reported outcome assessment after total joint replacement: comparison of questionnaire completion times on paper and tablet computer. Arch Orthop Trauma Surg. 2015;135:935-941.  [PubMed]  [DOI]
70.  Slover JD, Karia RJ, Hauer C, Gelber Z, Band PA, Graham J. Feasibility of integrating standardized patient-reported outcomes in orthopedic care. Am J Manag Care. 2015;21:e494-e500.  [PubMed]  [DOI]
71.  Tieu L, Sarkar U, Schillinger D, Ralston JD, Ratanawongsa N, Pasick R, Lyles CR. Barriers and Facilitators to Online Portal Use Among Patients and Caregivers in a Safety Net Health Care System: A Qualitative Study. J Med Internet Res. 2015;17:e275.  [PubMed]  [DOI]
72.  Coons SJ, Gwaltney CJ, Hays RD, Lundy JJ, Sloan JA, Revicki DA, Lenderking WR, Cella D, Basch E; ISPOR ePRO Task Force. Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force report. Value Health. 2009;12:419-429.  [PubMed]  [DOI]