Consumers of Experimental Observations: Understanding How Experimental Costs Affect Sample Size and Composition


Experimental samples that are too small to detect true effects have plagued the behavioral sciences for years. At best, these so-called underpowered studies provide weak evidence for measured effects. At worst, they add false positive results to the literature and are a waste of limited research money. Obtaining bigger samples is expensive, and yet previous empirical research has not considered the role of experimental costs. This study shows how much costs matter by analyzing a novel data set describing all experiments at a behavioral lab over many years. Demand for paid participants at this lab is inelastic: a 10% percent higher reimbursement rate corresponds with a 6% smaller sample. Studies reimbursing students with course credit rather than money have significantly larger samples, but incentivizing students with credit creates a subtle and important selection problem. These and other results show how researchers conduct science with limited resources, leading to new insights relevant to how we fund, incentivize, and reform behavioral research.