Editorial
Copyright ©The Author(s) 2017. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Methodol. Dec 26, 2017; 7(4): 112-116
Published online Dec 26, 2017. doi: 10.5662/wjm.v7.i4.112
Predictive power of statistical significance
Thomas F Heston, Jackson M King
Thomas F Heston, Department of Family Medicine, University of Washington, Seattle, WA 98195-6340, United States
Thomas F Heston, Jackson M King, Department of Medical Education and Clinical Sciences, Elson S. Floyd College of Medicine, Washington State University, Spokane, WA 99210-1495, United States
Author contributions: Heston TF and King JM made substantial contributions to this article, drafted the manuscript and approved the final version of the article.
Conflict-of-interest statement: The authors have no conflict of interest to declare.
Open-Access: This article is an open-access article which was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Correspondence to: Thomas F Heston, MD, Associate Professor, Department of Medical Education and Clinical Sciences, Elson S. Floyd College of Medicine, Washington State University, PO Box 1495, Spokane, WA 99210-1495, United States. tom.heston@wsu.edu
Telephone: +1-509-3587944 Fax: +1-815-5508922
Received: October 28, 2017
Peer-review started: October 29, 2017
First decision: November 20, 2017
Revised: November 23, 2017
Accepted: December 3, 2017
Article in press: December 3, 2017
Published online: December 26, 2017
Abstract

A statistically significant research finding should not be defined as a P-value of 0.05 or less, because this definition does not take into account study power. Statistical significance was originally defined by Fisher RA as a P-value of 0.05 or less. According to Fisher, any finding that is likely to occur by random variation no more than 1 in 20 times is considered significant. Neyman J and Pearson ES subsequently argued that Fisher’s definition was incomplete. They proposed that statistical significance could only be determined by analyzing the chance of incorrectly considering a study finding was significant (a Type I error) or incorrectly considering a study finding was insignificant (a Type II error). Their definition of statistical significance is also incomplete because the error rates are considered separately, not together. A better definition of statistical significance is the positive predictive value of a P-value, which is equal to the power divided by the sum of power and the P-value. This definition is more complete and relevant than Fisher’s or Neyman-Peason’s definitions, because it takes into account both concepts of statistical significance. Using this definition, a statistically significant finding requires a P-value of 0.05 or less when the power is at least 95%, and a P-value of 0.032 or less when the power is 60%. To achieve statistical significance, P-values must be adjusted downward as the study power decreases.

Keywords: Statistical significance, Positive predictive value, Biostatistics, Clinical significance, Power

Core tip: Statistical significance is currently defined as a P-value of 0.05 or less, however, this definition is inadequate because of the effect of study power. A better definition of statistical significance is based upon the P-value’s positive predictive value. To achieve statistical significance using this definition, the power divided by the sum of power plus the P-value must be 95% or greater.