Choosing the Next Experiment: Tradition, Innovation, and Efficiency in the Selection of Scientific Ideas (Innovation and Digital Scholarship Lecture Series)

What factors affect a scientist’s choice of research problem? Qualitative research in the history, philosophy, and sociology of science suggests that the choice is shaped by an ‘essential tension’ between a professional demand for productivity and a conflicting drive toward risky innovation.

We examine this tension empirically in the context of biomedical chemistry. We use complex networks to represent the evolving state of scientific knowledge, as expressed in digital publications. We then build measurements and a model of scientific discovery informed by key properties of this network.

Measuring such choices in aggregate, we find that the distribution of strategies remains remarkably stable, even as chemical knowledge grows dramatically. High-risk strategies, which explore new chemical relationships, are less prevalent in the literature, reflecting a growing focus on established knowledge at the expense of new opportunities. Research following a risky strategy is more likely to be ignored but also more likely to achieve high impact and recognition. While the outcome of a risky strategy has a higher expected reward than the outcome of a conservative strategy, the additional reward is insufficient to compensate for the additional risk.

By studying the winners of major prizes, we show that the occasional “gamble” for extraordinary impact is the most plausible explanation for observed levels of risk-taking. To examine efficiency in scientific search, we build a model of scientific discovery informed by key properties of this network, namely node degree and inter-node distance. We infer the typical research strategy in biomedical chemistry from 30 years of publications and patents and compare its efficiency with thousands of alternatives. Strategies of chemical discovery are similar in articles and patents, conservative in their neglect of low-degree, distant or disconnected chemicals, and efficient only for initial exploration of the network of chemical relationships. We identify much more efficient strategies for maturing fields.