A hidden peak behind the decline in scientific disruption
A new study, published in Nature, shows that the previously reported decline in scientific ‘disruption’ was overblown. Researchers from Vrije Universiteit Brussel (VUB) and KU Leuven found that missing reference lists inflate the scores of older papers. VSC’s high memory CPU nodes helped the team calculate disruption scores and test the finding on large citation networks.
In January 2023, a widely cited Nature cover article claimed that papers and patents were becoming less disruptive over time. The finding was based on citation analysis of almost 45 million scientific papers and 3.9 million patents. It suggested that science was increasingly building on existing knowledge instead of opening new directions.
But what if the apparent decline partly reflected the quality of the databases used to measure it?
A hidden peak
“Shortly after its publication, I attempted to reproduce the study using open data,” says doctoral researcher Vincent Holst, first author of the new study. “I discovered a large peak of papers receiving the maximal disruption score. This was strange, since the histograms published in Nature showed no such peak.”
Together with his colleagues, Holst traced the disparity to a bug in the plotting library seaborn. The software had silently omitted the largest values from the histograms. In the Web of Science data alone, correcting the plot revealed 972,161 papers with the maximum disruption score. While invisible in those graphs, these papers remained included in the analysis behind the headline finding.
“I discovered a large peak of papers receiving the maximal disruption score.” — Vincent Holst, doctoral researcher
When missing references look like breakthroughs
The study used the CD index, a measure that aims to distinguish research that consolidates earlier work from research that breaks with it. It looks at whether later papers cite a publication together with the work in its reference list, or whether they cite the publication without citing its predecessors.
If a database records no references for a paper, the index automatically assigns it the maximum score of +1 as soon as it receives at least one citation within five years. In the open SciSciNet database, 97% of papers with a maximum score had no recorded references. The Belgian team checked 100 randomly selected papers with maximum scores and zero recorded references against their source documents. Of those, 93 contained references, exposing the hidden outliers as dataset artefacts.
Older database records are more likely to have incomplete reference data. As coverage improved, these false maximum scores became less common, creating the appearance of declining disruption. When maximum-score papers were excluded, the decline largely disappeared across the scientific databases examined. In Web of Science, this reduced the reported decline between 1945 and 2010 by 93%.
How a tiny rounding error hid nearly a million papers
When drawing a histogram, seaborn 0.11.2 used NumPy’s arange function to calculate bin edges when binwidth was specified. A floating-point rounding error could place the final edge just below +1, silently excluding all papers with that score. Using bins instead of binwidth already produced the correct histogram in the old version, as shown below. The bug was fixed in seaborn 0.13.0.
Following 780 million citations with VSC
VSC provided the high memory CPU nodes needed to calculate the CD index and run a random rewiring algorithm on citation networks. The cluster jobs each requested roughly a terabyte of memory.
“The VSC, with its Tier-2 infrastructure at the VUB, allowed us to calculate disruption scores for millions of papers and run several random rewiring experiments in parallel.” — Dr. Floriano Tori
One analysis used raw SciSciNet journal data: around 48 million articles published between 1800 and 2016, connected by roughly 780 million citations. The team calculated five-year disruption scores and analysed trends for nearly 31 million papers published between 1944 and 2011.
The team also tested ten randomly rewired versions of the network, reshuffling who cites whom while preserving each paper’s numbers of references and incoming citations. Recalculating disruption scores on these networks helped explain why the original robustness check had failed to catch the consequences of the hidden outliers.
“This project required not only pure computing power, but also large amounts of memory to load in the citation graphs we analysed. The VSC, with its Tier-2 infrastructure at the VUB, allowed us to calculate disruption scores for millions of papers and run several random rewiring experiments in parallel by providing access to their high memory CPU nodes,” says Dr. Floriano Tori, who set up the large-scale computations on the VSC infrastructure.
From headlines to science policy
The original claim reached hundreds of news stories and informed science-policy discussions. The Belgian team’s conclusion is that the citation-based evidence behind that claim does not provide a sound basis for policy. “Based on the used measure, we simply cannot make conclusive statements about whether science is getting more or less disruptive,” says Professor Vincent Ginis, who led the study.
“Based on the used measure, we simply cannot make conclusive statements about whether science is getting more or less disruptive.” — Prof. Vincent Ginis
Publication: Vincent Holst, Andres Algaba, Floriano Tori, Sylvia Wenmackers and Vincent Ginis. Dataset artefacts can partially drive the measured decline in disruption. Nature 656, E7–E13 (2026).
Open data and code: https://github.com/VincentHolst/reanalysis_declining_disruption. Contact: Vincent Holst, vincent.thorge.holst@vub.be; Vincent Ginis, vincent.ginis@vub.be.
🔍 Your Research Matters — Let’s Share It!
Have you used VSC’s computing power in your research? Did our infrastructure support your simulations, data analysis, or workflow?
We’d love to hear about it!
Take part in our #ShareYourSuccess campaign and show how VSC helped move your research forward. Whether it’s a publication, a project highlight, or a visual from your work, your story can inspire others.
🖥️ Be featured on our website and social media. Show the impact of your work. Help grow our research community
📬 Submit your story: https://www.vscentrum.be/sys
