This project is about how experimenters choose sample sizes. It grew out of conversations I had with colleagues about instances of early stage work in which an expected effect was not detected in a lab study. Sometimes it seemed to me like the effect was likely to be real, but the sample was simply too small to measure it. I would often (naively) ask why they didn’t include more people in the study, and the answer was often a variant of “it’s too expensive.” Because studies in my group are very well funded (financially), it was clear that they didn’t mean the money cost of the experiment. Rather, they were referring to the time cost of gathering observations, and in particular, the idea that “more participants would mean another day or week in the lab.”
I thought this was an interesting phenomenon to study, and set out to obtain study meta data from our lab, spanning many years. Like much of my work, early drafts of the manuscript included both descriptive evidence (showing time and money costs affect sample size choices), and a structure model (to allow a counterfactual analysis of what would happen if we paid experimenters to use bigger samples). This 2017 version is a good representation of what the manuscript was like at that point.
I submitted the manuscript to a wide range of outlets where it was usually (but not always) desk rejected. I eventually decided to try my luck at the Journal of Consumer Research (JCR) because most of the lab data were generated by researchers working in the marketing subfield of consumer behavior (CB). To my surprise, the journal wanted to see a revision.
The review team at JCR provided very detailed advice about how to improve the manuscript for their audience: 1) expand the descriptive analysis, and 2) replace the structural model with something simpler, such as an IV regression. This 2018 version was the result of these changes. Upon receiving the revision, the JCR review team had two reactions: 1) “Wow, thank you for doing everything we asked!” and 2) “This manuscript doesn’t make enough of a contribution to warrant publication in JCR.”
Even in spite of knowing about JCR’s reputation for bad behavior during the review process, I was nevertheless shocked. The manuscript, which previously had a somewhat broad focus, had been completely rewritten for a CB audience. At that point, I contacted the editor of another CB journal, explained the situation, and asked if they would be interested in a paper like this. The reply was equivocal—saying they would want to see the structural model put back in the manuscript, and (as you would expect) making no promises about whether they would be favorable to publishing it.
I was reluctant to revise the manuscript again. While working on the JCR revision, I came to realize I didn’t have a lot of confidence in the structural model (I thought it was overly constrained by the utility function and the small sample size [ironically]). Further, at this point in my career, I was starting to understand more about the limits of simple IV estimation, and was losing confidence in those results as well. I decided to table the project for a while to work on a few other things, and now here it is over 5 years later.