Concerns about widespread misunderstanding and misuse of p-values in science have prompted the American Statistical Association (ASA) to issue its first-ever policy statement about the proper use of the statistical tool. On March 7, the organization released a set of six principles on the power and limitations of the p-value.
For instance, determining policy or making scientific conclusions should not be based on a p-value alone. “Practices that reduce data analysis or scientific inference to mechanical ‘bright-line’ rules (such as ‘p < 0.05’) for justifying scientific claims or conclusions can lead to erroneous beliefs and poor decision-making,” according to the ASA’s statement. “A conclusion does not immediately become ‘true’ on one side of the divide and ‘false’ on the other.”
Rather, complementing p-values with other statistics, such as confidence intervals, may better address the validity of a hypothesis.
In a commentary on the statement, Stanford University’s John Ioannidis wrote that adding more statistical layers does not solve the problems of “hidden multiplicity and selective reporting biases.” Transparency—another of the ASA’s principles—is essential. “Efforts to promote transparency in study design, conduct and reporting may have more to offer in this setting than blaming p-values,” Ioannidis wrote.
The ASA’s statement also points out what is arguably the biggest misconception about p-values. As FiveThirtyEight explained: “A common misconception among nonstatisticians is that p-values can tell you the probability that a result occurred by chance. . . . The p-value only tells you something about the probability of seeing your results given a particular hypothetical explanation—it cannot tell you the probability that the results are true or whether they’re due to random chance.”
Giovanni Parmigiani, a biostatistician at the Dana Farber Cancer Institute, told Nature News that guidance on proper p-value use has been needed. “Surely if this happened twenty years ago, biomedical research could be in a better place now.