Good news for psychology research: publications with “fragile” statistical results are decreased, and the champions are more numerous.
Ten years after the study that raised the lid on the crisis of reproducibility of research in the psychological field (according to which one of two research, among those of social and cognitive psychogy, was difficult to replicate), signs of an improvement in the precision of the scientific articles published in this field arrive.
Based on an analysis of 240,355 scientific publications in the field of psychology, the “fragile” statistical results have significantly decreased between 2004 and 2024. And with them – presumably – also the methods of research of doubtful value, or samples of subjects too restricted to represent reality. The research was recently published in the magazine Advances in Methods and Practices in Psychological Science.
Not always reliable literature
Between November 2011 and December 2014, the International Group of the Open Access Reproducibility Project project: Psychology had tried to reproduce the results of a hundred psychological studies published in 2008 on three different magazines in the sector, without going into the merits of the research, but only to verify the penalty with which they had been performed.
The results had been disappointing: only 36% of the replicated studies had given the same results. In addition, the size of the effects found was less than half of that reported by the original studies.
From that work, coordinated by Briana Nosek, a psychologist of the University of Virginia, an important collective reflection on How to improve the reliability of research in psychologyfor example by describing the research plan for extensive before taking a study, or making the data and codes used for analyzes public.
Progress in significance
Paul Bogdan, Duke University psychologist (North Carolina), has decided to measure the level of statistical significance of the studies of its sector, that is, the threshold that determines if a result can be considered statistically significant. He did it through the P-Value o Value P, which helps to understand if the difference between the observed result and the hypothesized one is due to the randomness introduced by the sampling, or if that difference is statistically significant.
Bogdan wrote a code to extract the P-Value from 240,355 studies from various sectors of psychology published in the last 20 years. In scientific research, a result is considered significant if the P-Value is less than 0.05. In 2004, 32% of psychology studies fell into what the scientist called a “fragile interval”, that is, just above the conventional threshold of significance. In 2024, the percentage of studies in this interval had fallen importantly, taking just above 26%.
The value provided for by the simple “case” – signal of the fact that Methodic errors in this field of study had greatly reduced.
More correct behaviors
It means that psychologists They are leading more rigorous scientific studieswith minor behaviors that go to the detriment of the sector such as the selective selection of results to publish only the most flashy ones (“Cherry Picking”). Progress has not been uniform in every field: clinical and development psychology still report a higher number of fragile studies from a statistical point of view compared to social and cognitive psychology.
Also The size of the samples of tested subjects have increased over timewith most of the studies that in 2004 included less than a hundred participants on average and in 2024 about 250. This growth is due for example to the use of online platforms to recruit volunteers or the practice of rewarding the time donated to research with small incentives.
