Minireviews
Copyright ©The Author(s) 2017. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Meta-Anal. Aug 26, 2017; 5(4): 80-84
Published online Aug 26, 2017. doi: 10.13105/wjma.v5.i4.80
Assessing the quality of studies in meta-analyses: Advantages and limitations of the Newcastle Ottawa Scale
Claudio Luchini, Brendon Stubbs, Marco Solmi, Nicola Veronese
Claudio Luchini, Department of Diagnostics and Public Health, University and Hospital Trust of Verona, 37134 Verona, Italy
Claudio Luchini, Surgical Pathology Unit, Santa Chiara Hospital, 38122 Trento, Italy
Brendon Stubbs, Physiotherapy Department, South London and Maudsley NHS Foundation Trust, Denmark Hill, London SE5 8AZ, United Kingdom
Brendon Stubbs, Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience King's College London, De Crespigny Park, London Box SE5 8AF, United Kingdom
Brendon Stubbs, Faculty of Health, Social Care and Education, Anglia Ruskin University, Chelmsford CM1 1SQ, United Kingdom
Marco Solmi, Department of Neurosciences, University of Padova, 35122 Padova, Italy
Marco Solmi, Nicola Veronese, Institute for Clinical Research and Education in Medicine, IREM, 35122 Padova, Italy
Nicola Veronese, National Research Council, Neuroscience Institute, Aging Branch, 35131 Padova, Italy
Author contributions: Luchini C and Solmi M wrote the manuscript; Stubbs B and Veronese N critically evaluated the manuscript; all the authors approved the final form and the submission.
Conflict-of-interest statement: None to declare.
Open-Access: This article is an open-access article which was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Correspondence to: Claudio Luchini, MD, Department of Diagnostics and Public Health, University and Hospital Trust of Verona, 37134 Verona, Italy. claudio.luchini@univr.it
Telephone: +39-45-8124842 Fax: +39-45-8027136
Received: January 13, 2017
Peer-review started: January 16, 2017
First decision: February 17, 2017
Revised: May 18, 2017
Accepted: June 6, 2017
Article in press: June 8, 2017
Published online: August 26, 2017

Abstract

One of the most important points in the meta-analyses is certainly represented by the assessment of the quality of the studies included in such research. The meta-analyses are considered the highest level of evidence in science. Also for this reason, the quality of the studies included should be accurately evaluated by standardized tools. The overall results of the meta-analysis depend indeed also on a rigorous evaluation of the studies quality. Among all the possible tools for this complex evaluation, the Newcastle Ottawa Scale (NOS) is one of the most used worldwide, above all for observational studies. In this review, we will discuss the strengths and limitation of the NOS, also on the basis of the branch of science in which it has been applied.

Key Words: Quality, Meta-analysis, Newcastle Ottawa Scale

Core tip: To assess the quality of a meta-analysis is a remarkable point. In this review, we summarize the current evidence regarding the use of the Newcastle Ottawa Scale, one of the most used tool for evaluating quality in meta-analyses of observational studies. Taking also our works as example, we found that, even standardized and quick in its application, it suffers from some limitations, particularly when evaluating cross-sectional studies.



INTRODUCTION

The quality assessment of studies included in systematic reviews and meta analyses is essential to enable a clear understanding of the evidence base. There are several sources of biases in meta-analyses including inaccurate selection of participants, data collection, analysis and selective reporting of study results[1]. Many of these biases derive directly from the studies which are included in the meta-analysis. However, since systematic reviews and meta-analyses are considered the highest level of evidence in science[2], the quality of the studies included should be accurately evaluated by validated and standardized tools.

For randomized controlled trials (RCTs), numerous tools are available to assess the risk of bias and methodological quality. Among the most commonly used, the Cochrane Collaboration’s tool[3] seems one of the most accurate, since it accounts for the main features of the RCTs. However, other tools (such as the Jadad’s scale[4]) are commonly used.

There are less methodological quality assessment tools available for the meta-analyses of observational studies. Some authors (including our group[5,6]) have used reporting checklists for detailing the quality of included studies (such as STROBE[7-9]). Whilst this method has several strengths, it may be seen as a simple reporting checklist. Such tools are not validated for assessing the quality of studies included in meta-analyses[1]. For this reason, other tools are commonly used for assessing quality and risk of bias in observational (both cross-sectional and longitudinal) studies. Among these, the Newcastle Ottawa Scale (NOS)[10] is one of the most used worldwide.

Given the rising number of meta analyses of observational studies in the scientific literature, it is mandatory that the tools used to assess study quality in such endeavors are appraised. In this review, we will discuss the strengths and limitation of the NOS and its application, taking as example the branches of pathology and psychiatry, also with reference to some meta-analyses from our group of research.

REVIEW AND DISCUSSION
NOS: Definition

The NOS can be used for both case-control and longitudinal (prospective studies). Typically, cross-sectional studies are evaluated as case control studies. The NOS evaluates three quality parameters (selection, comparability, and outcome) divided across eight specific items, which slightly differ when scoring case control and longitudinal studies[10]. Each item on the scale is scored from one point, except for comparability, which can be adapted to the specific topic of interest to score up to two points. Thus, the maximum for each study is 9, with studies having less than 5 points being identified as representing at high risk of bias[11].

In order to minimize the subjective interpretation of bias from scoring the NOS typically two independent authors should score each paper. However, in our opinion, the most important as well as critical point in the NOS scoring and filling in is certainly represented by the specific field in which the meta-analysis has been conducted. Each field of science has indeed intrinsic aspects with consequent implications: Here we present these differences and both the advantages and the limitations of NOS scale in some of the most important branches of science.

NOS for meta-analyses in pathology

The pathologists’ role in the era of modern medicine is based on performing an accurate as well as precise diagnosis, using standardized parameters, fixed cut-offs and thresholds. This standard strategy should be applied from the gross sampling to the pathology report[12,13]. The perfect tool for modern surgical pathologists to reach a consensus on the parameters to be reported in the diagnosis (when, how and why) is certainly represented by meta-analysis. With this statistical method the best standard and significant parameters, that can guide the pathologists during the diagnostic activity, can be documented. The meta-analysis can be thus applied to three of the main aspects of surgical pathology: (1) the prognostic impact of the mutation status of particular genes in cancer[14,15]; (2) the prognostic role of macro- or microscopic features of cancers[16-25]; and (3) the diagnostic utility of some morphological, immunohistochemical and/or molecular parameters[26]. Regarding the specific points of NOS scale for pathologists, an important topic is represented by the selection of the right method for ascertainment of exposure. If the meta-analysis regards a morphological aspect or an immunohistochemical staining, the classical microscopic exam should be preferred. Conversely, if the investigation regards a molecular aspect, the best standard molecular approach for the specific parameter should be applied, knowing that two of the most important are Sanger Sequencing[27] and Next Generation Sequencing[28-30]. For the outcome of interest, it is important specifying that this point is very subjective and may vary more than other points among different meta-analyses. In prognostic meta-analyses, however, we suggest to consider the disease-specific survival or the recurrence-free survival as the right parameter, being the overall survival the most common parameter in prognostic studies and thus not an index of quality. The point of NOS scale represented by the control for important factor or additional factor is obviously very important, since it can give two stars, determining a significant part of the quality’s evaluation. In this case the choice of the right parameters it is even more important. Knowing that data from multivariate analysis are more reliable, this merit should be ack-nowledge using this index, for example giving a star to a study that presents hazard ratios and an additional star if this data are obtained considering at least two or more potential confounders. For meta-analysis in pathology, at last, it has to be highlighted that a standard length for the follow-up of patients is 60 mo (5 years). A point of strength of this scale is that these parameters are not fixed and adaptable on the basis of the specific analysis. For example, a meta-analysis on the survival of patients with glioblastomas, a tumor with a very poor prognosis generally no longer than 2 years, will reach a star for smaller period (18 mo for example) than the classical 60 mo. The adaptability of NOS scale in this sense represents surely a point of strength. A limitation of this scale in pathology is that it may be very difficult, or even impossible, considering every possible source of bias in this scale, or highlighting every point of strength of the analyzed studies with this multi-stars system. The expertise and a consensus meeting among the authors is the best way to choose the right parameters for this scale.

NOS for meta-analyses in psychiatry

Psychiatry is different from pathology and other branches of medicine, since diagnoses, response, and remission are defined exclusively on clinical evaluation. Thus, the selection and exposure or outcome NOS items, which assess whether cases are diagnosed through reliable and independent validation or through self-report instead, is fundamental in establishing research quality. Moreover, among the main diagnostic systems, namely DSM-V[31] and ICD, some differences are evident[32] and there remains a great debate about diagnosis in general. Also, a control group defined as free from a specific mental disease in the general population, has a lower odds of having other psychiatric comorbidity compared to a control population of inpatients, due to frequent medical comorbidity among patients with severe psychiatric conditions[33,34]. Furthermore, remission and response definitions need specific psychopathologic scales’ cut-offs to be defined, and self-report or no description of such criteria need to be accounted for from a quality assessment scale, which is the case of NOS. Finally, treatment adherence is a substantial problem in psychiatry, and results are often affected by rates of completers and subjects lost at follow-up; again outcome NOS items account for such a variable. Thus authors encourage the use of NOS scale in the field of psychiatry, as has already been done in both observational[35,36] and interventional[37] studies’ meta-analyses.

Strengths of the NOS

The NOS is one of the most known scale for assessing quality and risk of bias in observational studies for several reasons[38], as reported in Table 1. The first one is that this tool is relatively quick to do, although it requires the right attention. Second, as already explained, the adaptability of its indexes on the basis of the investigated topic is very important. Furthermore, differently from other checklists and tools, it is validated for case-control and longitudinal studies[10]. Finally, differently from other tools, NOS gives a score between 0 and 9 and so it is possible to use it as potential moderator in meta-regression analyses[11,39].

Table 1 Strengths and weaknesses of the Newcastle Ottawa Scale.
StrengthsWeaknesses
Quick and adaptableNot validation for cross-sectional studies
ValidatedPoor agreement
ModeratorLack of comprehensive manuals
Weaknesses of the NOS

The NOS suffers from several weaknesses, however (Table 1). First, some domains are not univocal and one author should usually adapt this scale modifying some items. It is particularly true for cross-sectional studies for which the NOS should be adapted from the scheme of the case-control studies. Regarding the longitudinal investigations, the points usually adapted by the authors are the number and type of adjustments in the multivariate analyses, the duration of follow-up (not univocal follow-up is in fact given) and the outcome of interest not present at the baseline.

Another point of weakness is the low agreement between two independent reviewers in making the NOS. In the work proposed by Hartling et al[40], the agreement between the two reviewers was moderate/poor as shown by the k-value (< 0.50 for eight of the nine questions on the NOS). This is particularly true in case of low experience by the authors in meta-analysis/systematic reviews[41] suggesting that a training with a more expert author is needed.

Finally, the lack of comprehensive manuals could be interpreted as another limitation[42].

CONCLUSION

The NOS is a tool commonly used in medicine for the assessment of quality. Although it is a validated instrument and with a long history of reliability it suffers from several limitations. Other tools (tailored for cross-sectional studies and with more univocal items for other observational studies) are probably needed.

Footnotes

Manuscript source: Invited manuscript

Specialty type: Medicine, research and experimental

Country of origin: Italy

Peer-review report classification

Grade A (Excellent): A

Grade B (Very good): B

Grade C (Good): 0

Grade D (Fair): 0

Grade E (Poor): 0

P- Reviewer: Francisco G, Treglia G S- Editor: Kong JX L- Editor: A E- Editor: Lu YJ

References
1.  Dreier M. Quality Assessment in Meta-analysis. Methods of Clinical Epidemiology. Berlin, Heidelberg: Springer Berlin Heidelberg 2013; 213-228.  [PubMed]  [DOI]
2.  Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg. 2011;128:305-310.  [PubMed]  [DOI]
3.  Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA; Cochrane Bias Methods Group; Cochrane Statistical Methods Group. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.  [PubMed]  [DOI]
4.  Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17:1-12.  [PubMed]  [DOI]
5.  Solmi M, Veronese N, Correll CU, Favaro A, Santonastaso P, Caregaro L, Vancampfort D, Luchini C, De Hert M, Stubbs B. Bone mineral density, osteoporosis, and fractures among people with eating disorders: a systematic review and meta-analysis. Acta Psychiatr Scand. 2016;133:341-351.  [PubMed]  [DOI]
6.  Veronese N, Bano G, Bertozzo G, Granziera S, Solmi M, Manzato E, Sergi G, Cohen AT, Correll CU. Vitamin K antagonists’ use and fracture risk: results from a systematic review and meta-analysis. J Thromb Haemost. 2015;13:1665-1675.  [PubMed]  [DOI]
7.  von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61:344-349.  [PubMed]  [DOI]
8.  Solmi M, Veronese N, Sergi G, Luchini C, Favaro A, Santonastaso P, Vancampfort D, Correll CU, Ussher M, Thapa-Chhetri N. The association between smoking prevalence and eating disorders: a systematic review and meta-analysis. Addiction. 2016;111:1914-1922.  [PubMed]  [DOI]
9.  Solmi M, Veronese N, Favaro A, Santonastaso P, Manzato E, Sergi G, Correll CU. Inflammatory cytokines and anorexia nervosa: A meta-analysis of cross-sectional and longitudinal studies. Psychoneuroendocrinology. 2015;51:237-252.  [PubMed]  [DOI]
10.  Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, Tugwell P. The Newcastle-Ottawa Scale (NOS) for assessing the quality if nonrandomized studies in meta-analyses, 2012.  Available from: http://wwwohrica/programs/clinical_epidemiology/oxfordasp.  [PubMed]  [DOI]
11.  Veronese N, Cereda E, Solmi M, Fowler SA, Manzato E, Maggi S, Manu P, Abe E, Hayashi K, Allard JP. Inverse relationship between body mass index and mortality in older nursing home residents: a meta-analysis of 19,538 elderly subjects. Obes Rev. 2015;16:1001-1015.  [PubMed]  [DOI]
12.  Luchini C, Parcesepe P, Nottegar A, Parolini C, Mafficini A, Remo A, Chilosi M, Manfrin E. CD71 in Gestational Pathology: A Versatile Immunohistochemical Marker With New Possible Applications. Appl Immunohistochem Mol Morphol. 2016;24:215-220.  [PubMed]  [DOI]
13.  Luchini C, Parcesepe P, Mafficini A, Nottegar A, Parolini C, Veronese N, Remo A, Manfrin E. Specific expression patterns of epithelial to mesenchymal transition factors in gestational molar disease. Placenta. 2015;36:1318-1324.  [PubMed]  [DOI]
14.  Luchini C, Veronese N, Solmi M, Cho H, Kim JH, Chou A, Gill AJ, Faraj SF, Chaux A, Netto GJ. Prognostic role and implications of mutation status of tumor suppressor gene ARID1A in cancer: a systematic review and meta-analysis. Oncotarget. 2015;6:39088-39097.  [PubMed]  [DOI]
15.  Luchini C, Veronese N, Yachida S, Cheng L, Nottegar A, Stubbs B, Solmi M, Capelli P, Pea A, Barbareschi M. Different prognostic roles of tumor suppressor gene BAP1 in cancer: A systematic review with meta-analysis. Genes Chromosomes Cancer. 2016;55:741-749.  [PubMed]  [DOI]
16.  Luchini C, Nottegar A, Pea A, Solmi M, Stubbs B, Capelli P, Sergi G, Manzato E, Fassan M, Wood LD. Significance of the prognostic stratification of extranodal extension in colorectal cancer. Ann Oncol. 2016;27:1647.  [PubMed]  [DOI]
17.  Luchini C, Nottegar A, Pea A, Solmi M, Stubbs B, Capelli P, Sergi G, Manzato E, Fassan M, Wood LD. Extranodal extension is an important prognostic parameter for both colonic and rectal cancer. Ann Oncol. 2016;27:955-956.  [PubMed]  [DOI]
18.  Luchini C, Nottegar A, Solmi M, Sergi G, Manzato E, Capelli P, Scarpa A, Veronese N. Prognostic implications of extranodal extension in node-positive squamous cell carcinoma of the vulva: A systematic review and meta-analysis. Surg Oncol. 2016;25:60-65.  [PubMed]  [DOI]
19.  Luchini C, Veronese N, Pea A, Sergi G, Manzato E, Nottegar A, Solmi M, Capelli P, Scarpa A. Extranodal extension in N1-adenocarcinoma of the pancreas and papilla of Vater: a systematic review and meta-analysis of its prognostic significance. Eur J Gastroenterol Hepatol. 2016;28:205-209.  [PubMed]  [DOI]
20.  Luchini C, Wood LD, Cheng L, Nottegar A, Stubbs B, Solmi M, Capelli P, Pea A, Sergi G, Manzato E. Extranodal extension of lymph node metastasis is a marker of poor prognosis in oesophageal cancer: a systematic review with meta-analysis. J Clin Pathol. 2016;Jul 7; epub ahead of print.  [PubMed]  [DOI]
21.  Nottegar A, Veronese N, Senthil M, Roumen RM, Stubbs B, Choi AH, Verheuvel NC, Solmi M, Pea A, Capelli P. Extra-nodal extension of sentinel lymph node metastasis is a marker of poor prognosis in breast cancer patients: A systematic review and an exploratory meta-analysis. Eur J Surg Oncol. 2016;42:919-925.  [PubMed]  [DOI]
22.  Veronese N, Fassan M, Wood LD, Stubbs B, Solmi M, Capelli P, Pea A, Nottegar A, Sergi G, Manzato E. Extranodal Extension of Nodal Metastases Is a Poor Prognostic Indicator in Gastric Cancer: a Systematic Review and Meta-analysis. J Gastrointest Surg. 2016;20:1692-1698.  [PubMed]  [DOI]
23.  Veronese N, Luchini C, Nottegar A, Kaneko T, Sergi G, Manzato E, Solmi M, Scarpa A. Prognostic impact of extra-nodal extension in thyroid cancer: A meta-analysis. J Surg Oncol. 2015;112:828-833.  [PubMed]  [DOI]
24.  Veronese N, Nottegar A, Pea A, Solmi M, Stubbs B, Capelli P, Sergi G, Manzato E, Fassan M, Wood LD. Prognostic impact and implications of extracapsular lymph node involvement in colorectal cancer: a systematic review with meta-analysis. Ann Oncol. 2016;27:42-48.  [PubMed]  [DOI]
25.  Nottegar A, Veronese N, Solmi M, Luchini C. Tumor suppressor gene ARID1A in cancer: recent advances and future perspective. J Carinog Mutagene. 2016;7:255.  [PubMed]  [DOI]
26.  Luchini C, Nottegar A, Solmi M, Veronese N. Meta-Analysis in Pathology. J Pathol & Microbiol. 2016;1:1002-1003.  [PubMed]  [DOI]
27.  Simbolo M, Mafficini A, Agostini M, Pedrazzani C, Bedin C, Urso ED, Nitti D, Turri G, Scardoni M, Fassan M. Next-generation sequencing for genetic testing of familial colorectal cancer syndromes. Hered Cancer Clin Pract. 2015;13:18.  [PubMed]  [DOI]
28.  Mafficini A, Simbolo M, Parisi A, Rusev B, Luchini C, Cataldo I, Piazzola E, Sperandio N, Turri G, Franchi M. BRCA somatic and germline mutation detection in paraffin embedded ovarian cancers by next-generation sequencing. Oncotarget. 2016;7:1076-1083.  [PubMed]  [DOI]
29.  Yachida S, Wood LD, Suzuki M, Takai E, Totoki Y, Kato M, Luchini C, Arai Y, Nakamura H, Hama N. Genomic Sequencing Identifies ELF3 as a Driver of Ampullary Carcinoma. Cancer Cell. 2016;29:229-240.  [PubMed]  [DOI]
30.  Luchini C, Capelli P, Fassan M, Simbolo M, Mafficini A, Pedica F, Ruzzenente A, Guglielmi A, Corbo V, Scarpa A. Next-generation histopathologic diagnosis: a lesson from a hepatic carcinosarcoma. J Clin Oncol. 2014;32:e63-e66.  [PubMed]  [DOI]
31.  Chan CB, Abe M, Hashimoto N, Hao C, Ifor R, Brewster AS, Wang G, Yu X, William B, Brewster AS. Correction for Chan et al., Mice lacking asparaginyl endopeptidase develop disorders resembling hemophagocytic syndrome. Proc Natl Acad Sci USA. 2009;106:6023.  [PubMed]  [DOI]
32.  Hyland P, Shevlin M, McNally S, Murphy J, Hansen M, Elklit A. Exploring differences between the ICD-11 and DSM-5 models of PTSD: Does it matter which model is used? J Anxiety Disord. 2016;37:48-53.  [PubMed]  [DOI]
33.  Vancampfort D, Stubbs B, Mitchell AJ, De Hert M, Wampers M, Ward PB, Rosenbaum S, Correll CU. Risk of metabolic syndrome and its components in people with schizophrenia and related psychotic disorders, bipolar disorder and major depressive disorder: a systematic review and meta-analysis. World Psychiatry. 2015;14:339-347.  [PubMed]  [DOI]
34.  Stubbs B, Vancampfort D, De Hert M, Mitchell AJ. The prevalence and predictors of type two diabetes mellitus in people with schizophrenia: a systematic review and comparative meta-analysis. Acta Psychiatr Scand. 2015;132:144-157.  [PubMed]  [DOI]
35.  Zaninotto L, Solmi M, Toffanin T, Veronese N, Cloninger CR, Correll CU. A meta-analysis of temperament and character dimensions in patients with mood disorders: Comparison to healthy controls and unaffected siblings. J Affect Disord. 2016;194:84-97.  [PubMed]  [DOI]
36.  Solmi M, Zaninotto L, Toffanin T, Veronese N, Lin K, Stubbs B, Fornaro M, Correll CU. A comparative meta-analysis of TEMPS scores across mood disorder patients, their first-degree relatives, healthy controls, and other psychiatric disorders. J Affect Disord. 2016;196:32-46.  [PubMed]  [DOI]
37.  Fornaro M, Stubbs B, De Berardis D, Perna G, Valchera A, Veronese N, Solmi M, Ganança L. Atypical Antipsychotics in the Treatment of Acute Bipolar Depression with Mixed Features: A Systematic Review and Exploratory Meta-Analysis of Placebo-Controlled Clinical Trials. Int J Mol Sci. 2016;17:241.  [PubMed]  [DOI]
38.  Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25:603-605.  [PubMed]  [DOI]
39.  Veronese N, Carraro S, Bano G, Trevisan C, Solmi M, Luchini C, Manzato E, Caccialanza R, Sergi G, Nicetto D. Hyperuricemia protects against low bone mineral density, osteoporosis and fractures: a systematic review and meta-analysis. Eur J Clin Invest. 2016;46:920-930.  [PubMed]  [DOI]
40.  Hartling L, Milne A, Hamm MP, Vandermeer B, Ansari M, Tsertsvadze A, Dryden DM. Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers. J Clin Epidemiol. 2013;66:982-993.  [PubMed]  [DOI]
41.  Oremus M, Oremus C, Hall GB, McKinnon MC; ECT & Cognition Systematic Review Team. Inter-rater and test-retest reliability of quality assessments by novice student raters using the Jadad and Newcastle-Ottawa Scales. BMJ Open. 2012;2:e001368.  [PubMed]  [DOI]
42.  Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.  [PubMed]  [DOI]