Time To Abolish Statistical Significance?

The thought of "statistical significance" has been a basic concept inwards introductory statistics courses for decades. If yous pass whatsoever fourth dimension looking at quantitative research, yous volition oftentimes regard inwards tables of results that for certain numbers are marked alongside an asterisk or some other symbol to exhibit that they are "statistically significant."

For the uninitiated, "statistical significance" is a way of summarizing whether a for certain statistical termination is in all likelihood to cause got happened past times chance, or not. For example, if I flip a money 10 times together with larn half-dozen heads together with 4 tails, this could easily laissez passer on past times withdraw a opportunity fifty-fifty alongside a fair together with evenly balanced coin. But if I flip a money 10 times together with larn 10 heads, this is extremely unlikely to laissez passer on past times chance. Or if I flip a money 10,000 times, alongside a termination of 6,000 heads together with 4,000 tails (essentially, repeating the 10-flip money experiment 1,000 times), I tin live quite confident that the money is non a fair one. Influenza A virus subtype H5N1 mutual dominion of pollex has been that if the probability of an outcome occurring past times withdraw a opportunity is 5% or less--in the jargon, has a p-value of 5% or less--then the termination is statistically significant. However, it's also pretty mutual to regard studies that written report a gain of other p-values similar 1% or 10%.

Given the omnipresence of "statistical significance" inwards pedagogy together with the enquiry literature, it was interesting terminal yr when the American Statistical Association made an official declaration "ASA Statement on Statistical Significance together with P-Values" (discussed here) which includes comments like: "Scientific conclusions together with business concern or policy decisions should non live based only on whether a p-value passes a specific threshold. ... Influenza A virus subtype H5N1 p-value, or statistical significance, does non stair out the size of an effect or the importance of a result. ... By itself, a p-value does non furnish a goodness stair out of evidence regarding a model or hypothesis."

Now, the ASA has followed upwardly alongside a particular supplemental number of its periodical The American Statistician on the subject "Statistical Inference inwards the 21st Century: Influenza A virus subtype H5N1 World Beyond p < 0.05" (January 2019).  The number has a useful overview essay, "Moving to a World Beyond “p < 0.05.” past times Ronald L. Wasserstein, Allen L. Schirm, and  Nicole A. Lazar. They write:

We conclude, based on our review of the articles inwards this particular number together with the broader literature, that it is fourth dimension to halt using the term “statistically significant” entirely. Nor should variants such equally “significantly different,” “p < 0.05,” together with “nonsignificant” survive, whether expressed inwards words, past times asterisks inwards a table, or inwards some other way. Regardless of whether it was always useful, a annunciation of “statistical significance” has today locomote meaningless. ... In sum, `statistically significant'—don’t tell it together with don’t utilisation it.

The particular number is together with then packed alongside 43 essays from a broad array of experts together with fields on the full general subject of  "if nosotros eliminate the linguistic communication of statistical significance, what comes next?"

To empathize the arguments here, it's possibly useful to cause got a brief together with partial review of some primary reasons why the emphasis on "statistical significance" tin live hence misleading: namely, it tin Pb 1 to dismiss useful together with truthful connections; it tin Pb 1 to line fake implications; together with it tin crusade researchers to play some alongside their results. Influenza A virus subtype H5N1 few words on each of these.

The query of whether a termination is "statistically significant" is related to the size of the sample. As noted above, 6 out of 10 heads tin easily laissez passer on past times chance, but 6,000 out of 10,000 heads is extraordinarily unlikely to laissez passer on past times chance.  So tell that yous do an study which finds an effect which is fairly large inwards size, but where the sample size isn't large plenty for it to live statistically important past times a measure test. In practical terms, it would live foolish to ignore to ignore this large result; instead, yous should presumably start trying to bring out ways to run the examine alongside a much larger sample size. But inwards academic terms, the study yous only did may live unpublishable: after all, a  lot of journals volition tend to determine against publishing a study alongside negative results--a study that doesn't that doesn't fine a statistically important effect

Knowing that journals are looking to issue "statistically significant" results, researchers volition live tempted to expect for ways to jigger their results. Studies inwards economics, for example, aren't most uncomplicated probability examples similar flipping coins. Instead, 1 powerfulness live looking at Census information on households that tin live divided upwardly inwards roughly a jillion ways: non only the basic categories similar age, income, wealth, education, health, occupation, ethnicity, geography, urban/rural, during recession or not, together with others, but also diverse interactions of these factors looking at 2 or 3 or to a greater extent than at a time. Then, researchers brand choices most whether to assume that connections betwixt these variables should live thought of a linear relationship, curved relationships (curving upwardly or down), relationships are are U-shaped or inverted-U, together with others. Now add together inwards all the different fourth dimension periods together with events together with places together with before-and-after legislation that tin live considered. For this fairly basic data, 1 is chop-chop looking at thousands or tens of thousands of possible connections relationships.

Remember that the thought of statistical significance relates to  whether something has a 5% probability or less of happening past times chance. To set that some other way, it's whether something would cause got happened only 1 fourth dimension out of xx past times chance. So if a researcher takes the same basic information together with looks at thousands of possible equations, at that spot volition live dozens of equations that expect similar they had a 5% probability of non happening past times chance. When at that spot are thousands of researchers acting inwards this way, at that spot volition live a steady stream of hundreds of termination every calendar month that appear to live "statistically significant," but are only a termination of the full general province of affairs that if yous endeavor enough

Influenza A virus subtype H5N1 classic declaration of this number arises inwards Edward Leamer's 1983 article, "Taking the Con out of Econometrics" (American Economic Review, March 1983, pp. 31-43). Leamer wrote:

The econometric fine art equally it is practiced at the calculator terminal involves plumbing equipment many, possibly thousands, of statistical models. One or several that the researcher finds pleasing are selected for re- porting purposes. This searching for a model is oftentimes good intentioned, but at that spot tin live no doubtfulness that such a specification search in-validates the traditional theories of inference. ... [I]n fact, all the concepts of traditional theory, utterly lose their pregnant past times the fourth dimension an applied researcher pulls from the bramble of calculator output the 1 thorn of a model he likes best, the 1 he chooses to portray equally a rose. The consuming world is hardly fooled past times this chicanery. The econometrician's shabby fine art is humorously together with disparagingly labelled "data mining," "fishing," "grubbing," "number crunching." Influenza A virus subtype H5N1 joke evokes the Inquisition: "If yous torture the information long enough, Nature volition confess" ... This is a distressing together with decidedly unscientific the world of affairs nosotros bring out ourselves in. Hardly anyone takes information analyses seriously. Or possibly to a greater extent than accurately, hardly anyone takes anyone else's information analyses seriously."

Economists together with other social scientists cause got locomote much to a greater extent than aware of these issues over the decades, but Leamer was withal writing inwards 2010 ("Tantalus on the Road to Asymptopia," Journal of Economic Perspectives, 24: 2, pp. 31-46):

Since I wrote my “con inwards econometrics” challenge much progress has been made inwards economical theory together with inwards econometric theory together with inwards experimental design, but at that spot has been petty progress technically or procedurally on this dependent area of sensitivity analyses inwards econometrics. Most authors withal back upwardly their conclusions alongside the results implied past times several models, together with they larn out the ease of us wondering how difficult they had to locomote to bring out their favorite outcomes ... It’s similar a courtroom of police inwards which nosotros listen only the experts on the plaintiff’s side, but are wise plenty to know that at that spot are abundant for the defense. 

Taken together, these issues advise that a lot of the findings inwards social scientific discipline enquiry shouldn't live believed alongside likewise much firmness. The results powerfulness live true. They powerfulness live a termination of a researcher pulling out "from the bramble of calculator output the 1 thorn of a model he likes best, the 1 he chooses to portray equally a rose." And given the realities of real-world research, it seems goofy to tell that a termination with, say, only a 4.8% probability of happening past times withdraw a opportunity is "significant," spell if the termination had a 5.2% probability of happening past times withdraw a opportunity it is "not significant." Uncertainty is a continuum, non a black-and-white difference.


So let's cause got the that the "statistical significance" label has some severe problems, equally Wasserstein, Schirm, together with Lazar write: 

[A] label of statistical significance does non hateful or imply that an association or effect is highly probable, real, true, or important. Nor does a label of statistical nonsignificance Pb to the association or effect beingness improbable, absent, false, or unimportant. Yet the dichotomization into “significant” together with “not significant” is taken equally an imprimatur of authorisation on these characteristics. In a basis without vivid lines, on the other hand, it becomes untenable to assert dramatic differences inwards interpretation from inconsequential differences inwards estimates. As Gelman together with Stern (2006) famously observed, the divergence betwixt “significant” together with “not significant” is non itself statistically significant.

But equally they recognize, criticizing is the slow part. What is to live done instead? And here, the declaration fragments substantially. Did I cite that at that spot were 43 different responses inwards this number of the American Statistician?

Some of the recommendations are to a greater extent than a thing of temperament than of specific statistical tests. As Wasserstein, Schirm, together with Lazar emphasize, many of the authors offering advice that tin live summarized inwards most vii words: "Accept uncertainty. Be thoughtful, open, together with modest.” This is goodness advice! But a researcher struggling to larn a newspaper published powerfulness live forgiven for feeling that it lacks specificity.

Other recommendations focus on the editorial physical care for used past times academic journals, which constitute some of the incentives here. One interesting proffer is that when a enquiry periodical is deciding whether to issue a paper, the reviewer should only regard a description of what the researcher did--without seeing the actual empirical findings. After all, if the study was worth doing, together with then it's worthy of beingness published, right? Such an approach would hateful that authors had no incentive to tweak their results. Influenza A virus subtype H5N1 method already used past times some journals is "pre-publication registration," where the researcher lays out beforehand, inwards a published paper, precisely what is going to live done. Then afterwards, no 1 tin bill that researcher of tweaking the methods to obtain specific results.

Other authors concur alongside turning away from "statistical significance," but inwards favor of their ain preferred tools for analysis: Bayesian approaches, "second-generation p-values," "false positive risk,"

"statistical determination theory," "confidence index," together with many more. With many alteratative examples along these lines, the researcher trying to figure out how to locomote along tin in 1 lawsuit again live forgiven for desiring petty to a greater extent than definitive guidance.

Wasserstein, Schirm, together with Lazar also asked some of the authors whether at that spot powerfulness live specific situations where a p-value threshold made sense. They write:

"Authors identified 4 full general instances. Some allowed that, spell p-value thresholds should non live used for inference, they powerfulness withal live useful for applications such equally industrial character control, inwards which a highly automated determination dominion is needed together with the costs of erroneous decisions tin live carefully weighed when specifying the threshold. Other authors suggested that such dichotomized utilisation of p-values was acceptable inwards model-fitting together with variable selection strategies, in 1 lawsuit again equally automated tools, this fourth dimension for sorting through large numbers of potential models or variables. Still others pointed out that p-values alongside real depression thresholds are used inwards fields such equally physics, genomics, together with imaging equally a filter for massive numbers of tests. The 4th illustration tin live described equally “confirmatory setting[s] where the study blueprint together with statistical analysis computer program are specified prior to information collection, together with and then adhered to during together with after it” ...  Wellek (2017) says at acquaint it is essential inwards these settings. “[B]inary determination making is indispensable inwards medicine together with related fields,” he says. “[A] radical rejection of the classical principles of statistical inference…is of virtually no help equally long equally no conclusively substantiated choice tin live offered.”

The deeper betoken hither is that at that spot are province of affairs where a researcher or a policy-maker or an economical needs to brand a yes-or-no decision. When doing character control, is it coming together the measure or not? when the Food together with Drug Administration is evaluating a novel drug, does it  approve the drug or not? When a researcher inwards genetics is dealing alongside a database that has thousands of genes, there's a demand to focus on a subset of those genes, which agency making yes-or-no decisions on which genes to include a for certain analysis. 

Yes, the scientific spirit should "Accept uncertainty. Be thoughtful, open, together with modest.” But existent life isn't a philosophy contest. Sometimes, decisions demand to live made. If yous don't cause got a statistical rule, together with then the choice determination dominion becomes human judgment--which has plenty of cognitive, group-based, together with political biases of its own.

My ain sense is that "statistical significance" would live a  real pathetic master, but that doesn't hateful it's a useless servant. Yes, it would foolish together with potentially counterproductive to give excessive weight to "statistical significance." But the clarity of conventions together with rule, when their limitations are recognized together with acknowledges, tin withal live useful. I was struck past times a comment inwards the essay past times Steven N. Goodman:

P-values are role of a rule-based construction that serves equally a bulwark against claims of expertise untethered from empirical support. It tin live changed, but nosotros must abide by the argue why the statistical procedures are at that spot inwards the offset house ... So what is it that nosotros actually want? The ASA declaration says it; nosotros desire goodness scientific practice. We desire to stair out non only the signal properly but its uncertainty, the twin goals of statistics. We desire to brand cognition claims that gibe the forcefulness of the evidence. Will nosotros larn that past times getting rid of P−values? Will eliminating P−values ameliorate experimental design? Would it ameliorate measurement? Would it help align the scientific query alongside those analyses? Will it eliminate vivid line thinking? If nosotros were able to larn rid of P-values, are nosotros for certain that unintended consequences wouldn’t brand things worse? In my idealized world, the response is yes, together with many statisticians believe that. But inwards the existent world, I am less sure.