科技科学

Dataset artefacts can partially drive the measured decline in disruption

Nature作者：Vincent Holst2026年8月12日正文已收录本站

arising from: M. Park et al. Nature https://doi.org/10.1038/s41586-022-05543-x (2023).

Park et al.1 applied the CD index2, a measure of disruption in citation networks, to databases containing almost 45 million papers and 3.9 million patents. They reported a decline in the relative amount of disruptive scientific papers and technological patents in the past decades. Our re-examination, including a thorough assessment of the robustness checks conducted by Park et al.1, shows that a part of the reported decline in disruptiveness can be attributed to a relative decline of database entries with zero references, which have a maximum CD index of 1. Moreover, closer inspection of the source documents corresponding to these entries with zero references reveals that most of them do in fact contain references, indicating that the main effect on the measured disruptiveness seems to be weakened by artefacts in the dataset.

Data availability

The Web of Science and the PatentsView data for the study were retrieved from the public repository (https://doi.org/10.5281/zenodo.7258379 (ref. 10)) made publicly available by Park et al.1. The SciSciNet data source4 (https://doi.org/10.6084/m9.figshare.c.6076908 (ref. 11)) and the DBLP citation network v146 (https://www.aminer.cn/open/article?id=655db2202ab17a072284bc0c) are publicly available to download. The data for the reanalysis that we performed on these two datasets are publicly available at https://github.com/VincentHolst/reanalysis_declining_disruption (ref. 12). The data used to recreate Supplementary Fig. 17 were obtained either as source data from ref. 1 or by digitizing figures from refs. 13,14 using WebPlotDigitizer (https://automeris.io), and are publicly available alongside the other reanalysis data.

Code availability

The reanalysis code for this publication is publicly available at https://github.com/VincentHolst/reanalysis_declining_disruption (ref. 12). It is based on the original analysis code (https://doi.org/10.5281/zenodo.7258379 (ref. 10)) made publicly available by Park et al.1 and uses the same software packages as Park et al.1: pandas v1.4.3, numpy v1.23.1, matplotlib v3.5.2 and seaborn v0.11.2. To replicate the regression table, we used StataMP v18.0 (reghdfe v6.12.3). To digitize figure data for Supplementary Fig. 17, we used WebPlotDigitizer v5.2 (https://automeris.io).

References

  1. Park, M., Leahey, E. & Funk, R. J. Papers and patents are becoming less disruptive over time. Nature 613, 138–144 (2023).

    Article  ADS  CAS  PubMed  Google Scholar 

  2. Funk, R. J. & Owen-Smith, J. A dynamic network measure of technological change. Manage. Sci. 63, 791–817 (2017).

    Article  Google Scholar 

  3. Waskom, M. Treat binwidth as approximate to avoid dropping outermost datapoints. GitHub https://github.com/mwaskom/seaborn/pull/3489 (2023).

  4. Lin, Z., Yin, Y., Liu, L. & Wang, D. SciSciNet: a large-scale open data lake for the science of science research. Sci. Data 10, 315 (2023).

    Article  PubMed  PubMed Central  Google Scholar 

  5. Uzzi, B., Mukherjee, S., Stringer, M. & Jones, B. Atypical combinations and scientific impact. Science 342, 468–472 (2013).

    Article  ADS  CAS  PubMed  Google Scholar 

  6. Tang, J. et al. Arnetminer: extraction and mining of academic social networks, KDD ’08, 990–998 (ACM, 2008).

  7. Macher, J. T., Rutzer, C. & Weder, R. Is there a secular decline in disruptive patents? Correcting for measurement bias. Research Policy 53, 104992 (2024).

    Article  Google Scholar 

  8. Wu, L., Wang, D. & Evans, J. A. Large teams develop and small teams disrupt science and technology. Nature 566, 378–382 (2019).

    Article  ADS  CAS  PubMed  Google Scholar 

  9. Lin, Y., Frey, C. B. & Wu, L. Remote collaboration fuses fewer breakthrough ideas. Nature 623, 987–991 (2023).

    Article  ADS  CAS  PubMed  Google Scholar 

  10. Funk, R. J., Park, M. & Leahey, E. Papers and patents are becoming less disruptive over time (1.0) [Data set]. Zenodo https://doi.org/10.5281/zenodo.7258379 (2022).

  11. Yin, Y. SciSciNet: a large-scale open data lake for the science of science research. Figshare https://doi.org/10.6084/m9.figshare.c.6076908.v1 (2023).

  12. Holst, V. & Algaba, A. Dataset artefacts are the hidden drivers of the declining disruptiveness in science. GitHub https://github.com/VincentHolst/reanalysis_declining_disruption (2026).

  13. Funk, R. J. X https://x.com/russellfunk/status/1761950261671526618 (retrieved 30 August 2024).

  14. Park, M., Leahey, E. & Funk, R. J. The decline of disruptive science and technology. Preprint at https://doi.org/10.48550/arXiv.2106.11184 (2021).

  15. Hofmann, H., Wickham, H. & Kafadar, K. value plots: boxplots for large data. J. Comput. Graph. Stat. 26, 469–477 (2017).

    Article  MathSciNet  Google Scholar 

  16. Ruan, X., Lyu, D., Gong, K., Cheng, Y. & Li, J. Rethinking the disruption index as a measure of scientific and technological advances. Technol. Forecast. Soc. Change 172, 121071 (2021).

    Article  Google Scholar 

Download references

Acknowledgements

We thank M. Park, E. Leahey and R. J. Funk for making their code and source material public, and for responding to a previous version of this Comment, highlighting the different robustness checks in their manuscript. We thank S. J. Klein for helpful advice and M. Waskom for maintaining the open source seaborn library, which enabled us to quickly identify the bug in the plotting of the histograms.

Funding

Work was supported by the Research Council (OZR) of the VUB.

Author information

Authors and Affiliations

  1. Data Analytics Laboratory, Vrije Universiteit Brussel, Brussels, Belgium

    Vincent Holst, Andres Algaba, Floriano Tori & Vincent Ginis

  2. Centre for Logic and Philosophy of Science (CLPS), KU Leuven, Leuven, Belgium

    Sylvia Wenmackers

  3. School of Engineering and Applied Sciences, Harvard University, Boston, MA, USA

    Vincent Ginis

Authors

  1. Vincent Holst
  2. Andres Algaba
  3. Floriano Tori
  4. Sylvia Wenmackers
  5. Vincent Ginis

Contributions

V.H. and V.G. were responsible for the main idea of the study. V.H. identified the mistake in the original analysis (hidden outliers). A.A. identified the reason for the mistake (seaborn update). V.H., A.A. and V.G. designed the analysis. A.A. replicated the regression analysis. V.H. and F.T. implemented the random rewiring algorithm. V.H. and F.T. tested the robustness of the results with additional data and analyses. V.H. and F.T. designed the final figures. S.W. independently reviewed the results. All authors discussed the results and collaboratively drafted and revised the manuscript.

Corresponding authors

Correspondence to Vincent Holst or Vincent Ginis.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 Distribution of the CD5 index with vs without the hidden outliers and its impact on the disruptiveness for the SciSciNet data source.

This figure replicates the observation that papers with CD5 = 1 are driving the decline in disruptive science for the SciSciNet data source4 (with 39,888,199 papers between 1944 and 2011), which originated from the Microsoft Academic Graph. a, The distribution of the CD5 index for SciSciNet, created using the binwidth parameter in seaborn 0.11.2. Here again, the largest data points are hidden. b, The correct histogram of the underlying dataset. A peak at CD5 = 1 is revealed, corresponding to 8,861,343 additional papers. c, The time evolution of the average CD5 index. When dropping the outliers with CD5 = 1, the decline in disruptiveness reduces by 97%, from −0.31 with outliers (top curve, 2011 vs. 1944) to −0.01 without (bottom curve, 2011 vs. 1944). Excluding papers with zero references impacts the data similarly (Fig. 2 and Extended Data Fig. 2), reducing the decline by 87%, to −0.04 (dashed curve, 2011 vs. 1944). The shaded bands correspond to 95% confidence intervals. Moreover, the curve with papers with CD5 = 1 omitted is the curve corresponding to the histogram (a).

Extended Data Fig. 2 Papers and patents with CD5 = 1 predominantly make zero references.

This figure displays that most papers in the SciSciNet data source4 (n = 39,888,199) and most patents in the PatentsView data source (n = 2,926,923) with CD5 = 1 have zero references. a, Our analysis shows that PatentsView contains 142,362 patents with CD5 = 1 between 1980 and 2010, of which 78% appear in the database with zero references. b, Within the category of patents with CD5 = 1, the relative frequency of patents with zero references is stable between 1980 and 2010. c, The relative frequency of patents with CD5 index exactly equal to one and zero references is decreasing over time. Therefore, a substantial part of the reported decline in the disruptiveness of technological knowledge over time can be attributed to a relatively increasing metadata quality over time. It is also intriguing to note how well the shape of this curve resembles the shape of the top curve shown in Fig. 1f. d, SciSciNet4 shows a similar behaviour with 8,861,343 papers having CD5 = 1 between 1944 and 2011, of which 97% appear in the database with zero references. e, Within the category of papers with CD5 = 1, the relative frequency of papers with zero references is stable between 1944 and 2011. f, The relative frequency of papers with CD5 index exactly equal to one and zero references is decreasing over time. Therefore, a substantial part of the observed decline in the disruptiveness of scientific knowledge over time can be attributed to a relatively increasing metadata quality over time. It is also intriguing to note how well the shape of this curve resembles the shape of the top curve shown in Extended Data Fig. 1c.

Extended Data Fig. 3 Across various data sources and within different categories, papers and patents with CD5 = 1 can partially drive the measured decline in disruption over time.

This figure displays the average CD5 index over time for six data sources and five different patent categories. The data sources are JSTOR (1,588,088 papers), the American Physical Society corpus (461,359 papers), Microsoft Academic Graph (random sample of 1,000,000 papers), and PubMed (1,563,211 papers). For reference, the Web of Science (22,479,429 papers) and PatentsView (2,926,923 patents) data sources are also included. The patent categories are Chemical (517,964 patents), Computers and communications (748,849 patents), Drugs and medical (321,449 patents), Electrical and electronic (734,769 patents), and Mechanical (603,892 patents). Shaded bands correspond to 95% confidence intervals. a, The temporal evolution of the average CD5 index for different data sources as presented in Park et al.1 (Extended Data Fig. 6 in1). b, When dropping the outliers with CD5 = 1, the decline in disruptiveness reduces by 100%, 107%, 100%, and 61% for JSTOR, the American Physical Society corpus, Microsoft Academic Graph, and PubMed, from −0.16, −0.27, −0.31, and −0.18 with outliers (panel a, 2006 vs. 1930 for JSTOR, 2010 vs. 1930 otherwise) to −0.00, +0.02, −0.00, and −0.07 without (panel b, 2006 vs. 1930 for JSTOR, 2010 vs. 1930 otherwise), respectively. c, The time evolution of the average CD5 index for different patent categories as presented in Park et al.1 (Fig. 2b in1). d, When dropping the outliers with CD5 = 1, the decline in disruptiveness reduces by 70%, 58%, 69%, 63 % and 71% for Chemical, Computers and communications, Drugs and medical, Electrical and electronic, and Mechanical patents, from −0.33, −0.24, −0.35, −0.32, and −0.38 with outliers (panel c, 2010 vs. 1980) to −0.10, −0.10, −0.11, −0.12, and −0.11 without (panel d, 2010 vs. 1980), respectively.

Supplementary information

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Holst, V., Algaba, A., Tori, F. et al. Dataset artefacts can partially drive the measured decline in disruption. Nature 656, E7–E13 (2026). https://doi.org/10.1038/s41586-026-10787-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1038/s41586-026-10787-y