Systematic Reviews
Copyright ©The Author(s) 2015. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Meta-Anal. Jun 26, 2015; 3(3): 142-150
Published online Jun 26, 2015. doi: 10.13105/wjma.v3.i3.142
Development of the Documentation and Appraisal Review Tool for systematic reviews
Rebecca L Diekemper, Belinda K Ireland, Liana R Merz
Rebecca L Diekemper, American College of Chest Physicians, CHEST, Glenview, IL 60026, United States
Belinda K Ireland, The EvidenceDoc, Pacific, MO 63069, United States
Liana R Merz, Center for Clinical Excellence, BJC HealthCare, Saint Louis, MO 63108, United States
Author contributions: Diekemper RL was the primary developer of the tool and she participated in the testing of the tool and drafting parts of the paper; Ireland BK came up with the concept of developing the tool and was a co-developer of the tool and participated in the testing of the tool and drafting parts of the paper; Merz LR was a co-developer of the tool and participated in the testing of the tool and drafting parts of the paper.
Conflict-of-interest: All of the authors report that they receive no financial compensation for DART. Diekemper RL uses DART for assessing the quality of systematic reviews used to inform guideline recommendations for CHEST guidelines. Due to her role as a developer of DART, the tool has been adopted by CHEST for use in guideline development. Ireland BK reports that as a consultant who frequently conducts systematic reviews and overviews of reviews, she is interested in an effective and efficient tool for evaluating the quality of systematic reviews. Merz LR has no conflicts of interest to disclose.
Data sharing: No additional data are available.
Open-Access: This article is an open-access article which was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Correspondence to: Rebecca L Diekemper, MPH, American College of Chest Physicians, 2595 Patriot Blvd, Glenview, IL 60026, United States. rdiekemper@chestnet.org
Telephone: +1-314-5319325
Received: January 27, 2015
Peer-review started: February 5, 2015
First decision: March 6, 2015
Revised: April 2, 2015
Accepted: April 27, 2015
Article in press: April 29, 2015
Published online: June 26, 2015
Abstract

AIM: To develop a tool to more explicitly assess and document the quality of systematic reviews.

METHODS: We developed the Documentation and Appraisal Review Tool (DART) using epidemiologic principles of study design and the following resources: the modified Overview Quality Assessment Questionnaire (modified OQAQ), Assessment of Multiple Systematic Reviews (AMSTAR), the Cochrane Handbook, and the standards promoted by the Agency for Healthcare Research and Quality, and the Institutes of Medicine (IOM). We designed the DART tool to include the following: more detail to provide guidance and improve standardization of use, an approach to assess quality of systematic reviews addressing a variety of research designs, and additional space for recording notes to facilitate recall. DART underwent multiple rounds of testing with methodologists of varying levels of training and experience. Based on the results of six phases of pilot testing, we revised DART to improve performance, clarity and consistency. Pilot testing also included comparisons between DART, and the two most commonly used tools to evaluate the quality of systematic reviews, the modified OQAQ and AMSTAR.

RESULTS: Compared to AMSTAR and modified OQAQ, DART includes two unique questions and several questions covered by modified OQAQ or AMSTAR but not both. Modified OQAQ and DART had the highest reporting consistency. Four AMSTAR questions were unclear and elicited inconsistent responses. Identifying reviewer rationale was most difficult using the modified OQAQ tool, and easiest using DART. DART allows for documentation of reviewer rationale, facilitating reconciliation between reviewers and documentation for future updates. DART also provides a comprehensive, systematic approach for reviewers with limited experience with systematic review methodology, to critically analyze systematic reviews. In addition, DART is the only one of the three tools to explicitly include quality review for biases specific to observational studies. This is now more widely recognized as important for assessing risk in order to generate recommendations that balance benefit to harm. The tool also includes the assessment of standards recommended by the March 2011 IOM Standards for Systematic Review.

CONCLUSION: This comprehensive tool improves upon existing tools for assessing the quality of systematic reviews and guides reviewers through critically analyzing a systematic review.

Keywords: Quality assessment tool, Methodology, Healthcare research, Systematic review, Meta-analysis, Guidelines

Core tip: Systematic reviews and meta-analyses are commonly used to inform the recommendations presented in evidence-based clinical practice guidelines. The purpose of this study was to evaluate the Documentation and Appraisal Review Tool (DART) for its comprehensiveness, identify areas addressed by DART that were not addressed by two other validated tools [Overview Quality Assessment Questionnaire (OQAQ) and Assessment of Multiple Systematic Reviews (AMSTAR)], and to test its performance in eliciting consistent responses. We found that our tool was more comprehensive and included several questions not included in the other tools. We also found that DART elicited the most consistent responses when compared to OQAQ and AMSTAR.