What is Metascience?
Casrai provides a definition of
Metascience:
“The empirical, quantitative study of science itself:
applying research methods to research methods, reporting practices, evaluation
systems, and incentive structures, in order to describe how research is
actually conducted and to test interventions that make it more reliable and
efficient. Something counts as metascience when its object of study is the
research process across a field — study design, statistical power, reporting
completeness, peer review, replication rates, or how funding and career incentives
shape researcher behavior — rather than a first-order scientific question
within a specific discipline. Also called meta-research or, informally,
"the science of science."
David Peterson, a professor of
sociology at Purdue University, defines it roughly and simply as “the
practice applying scientific scrutiny to the work of science.”
While I am quite familiar with the
study of science itself and the scientific method from thinkers like Thomas
Kuhn and Karl Popper, and I have come across many “meta-studies” or
“meta-analyses” that look at many different studies simultaneously and compare
them, I have not really come across the term ‘metascience’ before. Perhaps I am
just late to the game. In any case, ‘meta-studies’ are defined by Microsoft
Copilot AI as follows:
“Metastudies” most commonly refers to meta‑analysis—an evidence‑synthesis method that
statistically combines results from multiple independent studies to produce a
single, more powerful estimate of effect size. It can also refer to broader
“meta‑studies”
in metatheory, which analyze methods, theories, and knowledge systems
themselves.”
Metascience for Innovation: Pros and Cons
Apparently, metascience is being
explored for its potential ability to spur innovation. One might see it as a
statistical study of a scientific subject, which may reveal things like where
most research dollars go or the feasibility of different theories and how much
effort and resources are going to each theory.
The Aventine Research Institute
notes in a recent article entitled ‘Can Metascience Supercharge Scientific
Discovery?’ that the National Science Foundation (NSF) and the National
Institutes of Health (NIH) recently formed offices within their domains devoted
to metascience.
“On September 10, the National Science Foundation (NSF)
proposed the creation of a new Office of Metascience. Part of a larger effort
to double the impact of each federal dollar spent on science by 2036, the new
office is designed to help the agency become a “self-improving institution”
that “tries new approaches” and “measures their effects.” The National
Institutes of Health (NIH) is creating a new division, too — the Office of
Research Economics, Planning, and Analysis — to carry out similar work.”
Thus, their goal is to make
scientific research more efficient. It is my own opinion that while the goal
may be noble, it could backfire if the focus becomes too much about research
efficiency and less about the research itself. Research takes time and requires
failure, or at least ruling out what doesn’t work. It is less amenable to
becoming efficient because it is overly inclusive, especially in the beginning,
and rigor takes time and resources. Prioritizing is, of course, a priority.
(See what I did there - is that a meta-statement?) Nobody wants to waste time,
money, and other resources on ideas that have little merit or on ones that are
not likely to have much of an effect on society. Thus, having people who work
exclusively in prioritizing and weeding out less impactful research may be a
good idea.
The article notes that the UK
established a metascience unit in 2024, Ireland and Japan are doing similarly,
and that “moonshot factories” such as NSF’s X-Labs initiative and the UK
government’s Advanced Research and Invention Agency can be seen as offshoots of
metascience.
Interestingly, the article notes
that scientific discoveries have been dropping for years and now require more
researchers to manifest.
“…our ability to generate scientific breakthroughs has
declined dramatically. Sustaining the rate of improvement in computer chip
advances now requires 18 times as many semiconductor researchers as it did in
the 1970s, for example. The number of new drugs approved per billion dollars
spent on R&D has halved roughly every nine years since 1950.”
“Metascience is meant to be corrective. It’s about “how
[we can] be more effective, more innovative, drive the pace of scientific
progress more quickly,” said Stuart Buck, a longtime supporter of metascience
who runs the Good Science Project, a nonprofit think tank with the stated
mission of improving the funding and practice of science.”
The article does mention that
there are risks to this approach. It’s untested, especially at scale. It may
take years or decades to know if it has been worthwhile. They emphasize that
success will be hard to measure. However, they also note some measurable
improvements. The UK’s Metascience Unit did an experiment where evaluating and
peer-reviewing grant applications was undertaken by more researchers, including
those applying for grants, rather than a small panel of experts. This resulted
in the measurable success of reducing the decision-making time for grant
approval or non-approval by three months. It really is hard to measure in the
sense that when one research path is chosen over another, it is simply hard to
know whether the other one would have yielded better and more useful results.
Another issue is that success at a small scale does not mean success when
scaled up. Scaling can kill projects as new technical challenges and expenses
arise. Another issue is that different sciences have much different standards.
“Standards of evidence, standards for justification,
data collection, replication — everything differs widely from field to field,”
said Peterson.
The article also notes some of the
history and recent events in metascience. They note that metascience began in
the 1930s or possibly earlier. The replication crisis of the 2010s in the field
of psychology was blamed on results that were not reproducible due to small
sample sizes and the urge to publish, especially to publish surprising results.
This led to calls for better apportionment of research dollars. They also cite
a 2019 article in The Atlantic by CEO Patrick Collison and economist Tyler Cowen
entitled: ‘We Need a New Science of Progress’ as being very influential.
Finally, the article asks whether
metascience is being used as a Trojan horse for budget cuts or whether it is
being used for needed disruption and reform of the processes. Stuart Buck, a supporter
of metascience who runs the Good Science Project, a nonprofit think tank with
the stated mission of improving the funding and practice of science, notes:
“There are a few people who are influential and whose
agenda is just cutting science,” he said, but there are others in the
administration who he believes have a “good faith intent to improve science.”
His hope, he added, is that “the folks with that view can prevail in the end.”
Collison and Cowen’s article in
The Atlantic suggests that scientific research is a major engine for progress.
They cite progress-minded 19th-century reformers who shaped some of the best
research universities in the U.S., including MIT, Caltech, Johns Hopkins,
Harvard, Stanford, and the University of Chicago. They argue that it could be
beneficial to study progress itself. They also cite hubs for R&D like
Silicon Valley in software and Switzerland’s Basel region in life sciences.
They call these cultural ecosystems that foster research. The article covers
optimizing talent and training and cultivating the most effective management
practices. They call for more investment and research funding and note examples
where it has led to innovation and discoveries. Thus, the current version of
the Trump administration’s gold standard science push, which includes attention
to metascience, while at the same time gutting research funding, does not seem
to accord with their ideas. Collison and Cowen make the case for progress
resulting from science applied to human well-being and their goal of
establishing progress studies. Before this article from 2019, there was and is
HumanProgress.org, which is associated with the libertarian-oriented Cato
Institute. They have featured in my blog posts several times. They post
articles, data, and ideas about human progress.
The 2010s were a pivotal time for
metascience. The replication crisis called for better oversight and methods.
Journals began to publish more meta-research. In 2015, a meta-research
conference was held for the first time. According to Wikipedia:
“On 8 July 2025, an editorial published in Nature
announced the birth of the Metascience Alliance, a coalition of more than 25
funders, academic groups, companies, and other institutions that pursue
metascience: the use of scientific methods to understand and improve science
itself. The initiative was attended in London by more than 830 participants
from around 65 countries. The editorial calls upon metascientists not to limit
themselves to studies within academia, but to address the broader social needs,
to communicate scientific uncertainty effectively, and rebuild public trust in
science.”
I think that metascience is both
the study and management of science and research.
Luke Fehilym in an article for The
Irish Times, suggests that metascience is resulting in better research outcomes
through better management of funding and grants.
Berna Devezer, a Professor
of Marketing at the University of Idaho and practitioner of metascience, wrote
an article in The Scientist that strongly criticizes the politicization of
metascience, which she thinks is more likely with metascience units and
offices, like the NSF and NIH plan to do. Michael Kratsios. Director of the
White House Office of Science and Technology Policy put out a big report in
July 2026 entitled Science: A New Golden Age, which has both good and bad
recommendations in my opinion. It recommends more private sector R&D, which
is good. However, private sector R&D is already ample. It is public or
government R&D that is good for getting things off the ground, taking on
the more difficult, riskier early stages of R&D. “Unleashing innovation” is
a goal of the report but it is hard to see that happening when budgets are
reduced by a half to two-thirds. It covers the replication crisis, grantmaking,
funding, publishing, bureaucracy, institutions, and accountability. I may read
and review the report later, now having just scanned a few parts. I will say
that the report itself is politicized, with sections where Trump and his team
strongly criticize the Biden approach to science. There is a chapter entitled
Securing U.S. Dominance in Critical and Emerging Technologies, which I think
both parties would support, although the Dems likely won’t keep the word
‘dominance’ that the Trump administration uses ubiquitously. One part I liked
is a short section on restoring permissionless innovation, having read and
considered Adam Therier’s book and writing on the subject. Below is a snippet
from the report:
“Restore Permissionless Innovation: American
regulators have grown skilled at weighing the risks of action, but blind to the
costs of inaction. Developing good rules require real-world evidence, and
building that evidence base only comes from letting innovators prototype and
experiment.”
I also agree with the idea of
strengthening public-private partnerships, which is a bipartisan goal. I
believe.
The report also talks about gold
standard science. What does that really mean? Devezer tears down the whole idea
in her article in The Scientist, noting that adopting the gold standard for
financial systems turned out to be wholly inadequate and likely contributed
strongly to the Great Depression. That idea has long been abandoned and is
widely seen as a failed idea among economists. She says that when
recommendations become standards, science fails. The heart of her takedown of
the idea is reproduced below:
“First, imposing externally fixed standards on
scientific fields without taking the specific needs, aims, and constraints of
the fields into account is unlikely to engender progress. Second, systemic,
institutional commitment to such external standards will render scientific
communities rigid and unable to correct course or adjust to emergent crises.
Failing to find the right standards is not my major concern. Top-down
imposition of any rigidly held standard would suffice to disrupt the scientific
process and derail progress.”
“Let me be blunt: There are no gold standards in science
and there have never been. This should not come as a surprise to working
scientists, or to anyone familiar with the history and philosophy of science.
One of the defining characteristics of science is that it keeps challenging its
own standards and redefining what it means to practice science at a given
moment, in a given field. As philosopher of science Paul Feyerabend captures in
a provocative slogan, “anything goes,” scientific progress happens when rigid
rules are actively broken and multiple, incompatible theories are allowed to
coexist and challenge one another simultaneously.”
She does think that it is good
that the government is interested in metascience but argues that funding is
essential and that taking funding away, as the administration has done,
undermines the whole process.
“To prioritize our understanding of science as a complex
system, NSF should fund both interdisciplinary and discipline-specific
metascience as independent research. To advance scientific understanding and to
provide solid grounding for applied research, it should also fund basic and
theoretical research. There are no gold standards in science. There are,
however, scientific communities who do the hard work, day in day out, and what
they need is support to build edifices.”
Jen Gold, who worked on the
development of the UK’s metascience unit, wrote in an article for Nature that
she thinks they have been successful with some funding reforms and with the
mentioned distributed peer review process, which led to much faster grant
evaluations. They also implemented a process of removing what they call
non-competitive grant applications. She recommends small teams as more
effective than larger ones. That is the last recommendation I got from the
article before the paywall hit.
An editorial published a couple of
days ago in Nature argues, as its title exemplifies, that “metascience
should be a source of evidence, not a means of control.” The article warns
against the misuse of metascience and notes that now there is a much larger
pool of researchers due to the expansion of universities and the globalization
of science.
“Funding agencies need to change how they work to adapt
as the world changes. That includes addressing the problem of falling success
rates resulting from the greater demand for grants.”
“By far the best way for policymakers to use metascience
to inform funding decisions is for them to work with researchers to create and
implement tools designed to measure quality and competitiveness, and — as in
science itself — to learn from experimentation and experience.”
Metascience can be quite useful, I
believe, but scientists will still need the slack to perfect their ideas. I
used to call this “tinker time,” or time to tinker around with ideas and test
them and come up with better ideas. I think that holding scientists to higher
efficiency standards will backfire. I also think that high levels of funding
are necessary for both the public and private sectors. Skimping on R&D
simply offers no advantages. Funding can be cut elsewhere. Funding science is
too important. Nonetheless, metascience can cut some waste and weed out some of
the less plausible and less potentially impactful research going on.
References:
Can
Metascience Supercharge Scientific Discovery? Aventine Research Institute. October
1, 2026. Can Metascience Supercharge
Scientific Discovery?
Metascience.
Casrai. (Dictionary). Metascience - Definition — CASRAI
Metascience.
Wikipedia. Metascience - Wikipedia
I
built the UK government’s metascience unit — here’s what countries need to know:
To find evidence-backed ways to improve grant-funding processes, metascientists
need freedom and money to experiment, as well as leaders who are willing to
make change. Jen Gold. Nature. October 30, 2026. I built the UK government’s
metascience unit — here’s what countries need to know | Nature
Holding
Science to a Gold Standard Will Hold It Back: A metascientist pushes back
against the unified and idealized vision of science embodied in NSF’s new
Office of Metascience. Opinion. Berna Devezer, PhD. The Scientist. October 29,
2026. Holding Science to a Gold Standard
Will Hold It Back | The Scientist
Rise
of metascience: Turning science on itself: Focus on the ‘science of science’ is
ensuring better research outcomes. Luke Fehily. The Irish Times. October 17, 2024. Rise of metascience: Turning science
on itself – The Irish Times
We
Need a New Science of Progress: Humanity needs to get better at knowing how to
get better. Patrick Collison and Tyler Cowen. The Atlantic. July 30, 2019. We
Need a New Science of Progress - The Atlantic
Science:
A New Golden Age. A REPORT TO THE PRESIDENT BY MICHAEL KRATSIOS, DIRECTOR OF
THE WHITE HOUSE OFFICE OF SCIENCE AND TECHNOLOGY POLICY. JULY 2026. Science-A-New-Golden-Age.pdf
Metascience
should be a source of evidence, not a means of control: The field has captured
the interest of politicians. There are positives — but governments must not
misuse the data. Editorial. Nature 659, 272. November 10, 2026. Metascience should be
a source of evidence, not a means of control | Nature
No comments:
Post a Comment