The Price of Psychological Science


Recent focus on replicability in psychology has brought renewed attention to the problem of underpowered, small sample experiments. Researchers are expected to increase the amount of evidence they collect, in the form of more studies (e.g., replications) and larger samples. Collecting more evidence, however, requires investment of limited financial resources. How do researchers trade off monetary and scientific considerations? We study this problem by leveraging the popularity and transparency of online experimental platforms. We show how a permanent increase in the cost of using Amazon Mechanical Turk decreased the number of studies conducted by half. In a separate field experiment, we show how a 15% discount in the cost of using Prolific more than doubled sample sizes. These results highlight the need to inform current metascientific debates with evidence of how scientists work. Financial considerations play a critical role and should be part of these discussions. We highlight some of the strategies available to researchers to help them meet the often financially demanding requirements of behavioral science.