More on scientific reproducibility. Four researchers from Idaho University argue that the relationship between reproducibility and other desirable properties of scientific discovery, such as early discovery of truth, persistence on truth once it is discovered, and time spent on truth in a long-term scientific inquiry, is not well understood. They have built a mathematical model of scientific discovery and conclude that the scientific process may not converge to truth even if scientific results are reproducible and that irreproducible results do not necessarily imply untrue results.

The proportion of different research strategies represented in the scientific population, scientists’ choice of methodology, the complexity of truth, and the strength of signal contribute to this counter-intuitive finding. Important insights include that innovative research speeds up the discovery of scientific truth by facilitating the exploration of model space and epistemic diversity optimizes across desirable properties of scientific discovery.

Devezer, Berna, et al. ‘Scientific Discovery in a Model-Centric Framework: Reproducibility, Innovation, and Epistemic Diversity’. PLOS ONE, vol. 14, no. 5, May 2019, p. e0216125. PLoS Journals, doi:10.1371/journal.pone.0216125.

Mathematical models might be critical to understand scientific discovery and progress:

I find it ironic that we have models for how political systems work (…) but we pretty much know nothing about the dynamics of knowledge discovery. We don’t know how the academic system works, for how people develop their ideas, for how these ideas get selected, for how these ideas proliferate. We don’t have any good understanding of how that works.

Looking in the Wrong Places, A Conversation With Sabine Hossenfelder

Sabine Hossenfelder is concerned with stagnation in physics, and she is very clear on the cause: “It’s not institutional pressure that creates this resistance, it’s that scientists themselves don’t want to move their butts”. No one likes change.

The Idaho team also identifies this problem with their model:

our results also advise against homogeneity in scientific practice. We find that a diversity of strategies in the scientific population optimizes across desirable properties of scientific discovery—a finding consistent with the cognitive division of labor literature.

A problem which can only worsen with ageing societies:

Two critical features of this [U.S. science] system are now threatened: the support of young people and their unique potential to take risks and explore promising new ideas; and a merit-based selection of scientists and engineers to populate academia and industry, viewing everyone as equal, regardless of the nation in which they were born.

Bruce Alberts1, Venkatesh Narayanamurti ‘Two threats to U.S. science’, Science 17 May 2019: Vol. 364, Issue 6441, pp. 613 doi: 10.1126/science.aax9846

In another recent paper, Richard Shiffrin at. al. review an increasing number of challenges well beyond reproducibility and factors that make scientific success difficult. They are more optimistic and believe that science is producing outstanding new results and theories at a rapid rate, but they recognise that:

Although there is good reason to believe in the rapidity of scientific progress, we know of no feasible methods to produce a quantitative metric, either across science or within the field

Shiffrin, Richard M., et al. ‘Scientific Progress despite Irreproducibility: A Seeming Paradox’. Proceedings of the National Academy of Sciences, vol. 115, no. 11, Mar. 2018, pp. 2632–39. http://www.pnas.org, doi:10.1073/pnas.1711786114.

Another seeming paradox is that we know so little about our most reliable way of gaining knowledge.