“06 November 2024 Editor’s Note: Readers are alerted
that the reliability of data and methodology presented in this manuscript is
currently in question. Appropriate editorial action will be taken once this
matter is resolved.”
The paper claimed that the
world was on track to lose 19% of GDP by 2050 due to impacts from climate
change. It also claimed that by 2100, we would lose 62% of GDP under a high
emissions scenario. Those predictions were triple the conclusions of other
analyses and papers. Perhaps that itself should have been a red flag that the
data should be examined more closely. One might ask why no other researchers
had predicted anything close to this. Instead, the paper was the second most
cited paper by the media in 2024, according to Carbon Brief. Shannon Osaka of
the Washington Post notes that the paper’s analysis and dataset have been used
for financial planning by the U.S. government, the World Bank, and other
institutions.
Apparently, the erroneous
data for one country, Uzbekistan, tainted the entire analysis. On August 6,
2025, a commentary paper describing the data error was published in Nature with
the following observations about the paper, known as KLW, referencing its
authors:
“(1) data anomalies arising from one country in KLW’s
underlying GDP dataset, Uzbekistan, substantially bias their predicted impacts
of climate change, (2) KLW underestimate statistical uncertainty in their
future projections of climate impacts, and (3) additional data-quality concerns
in KLW’s subnational GDP data warrant further investigation. When Uzbekistan’s
data are removed and statistical uncertainty is corrected to account for
spatial correlations, KLW’s central estimate aligns closely with previous
literature and their results are no longer statistically distinguishable from
mitigation costs at any time this century.”
Osaka notes:
“With Uzbekistan removed from the dataset, the
predictions dropped substantially — from 62 percent GDP loss in 2100 to 23
percent and from 19 percent by 2050 to 6 percent, said Solomon Hsiang, director
of the global policy laboratory at Stanford University and one of the authors
of the commentary.”
This puts the conclusions in
line with most other estimates. It was a case where a model was very strongly
influenced by a single incorrect model assumption. Those who found the data
error and wrote the commentary were surprised, noting that it was
counterintuitive that a single error could influence a model’s conclusions to
such a high degree.
The paper’s original authors
from the Potsdam Institute for Climate Impact Research, in Germany, however,
think that correcting the Uzbekistan error only reduces their conclusion by
about 2%, from a 19% GDP drop by 2050, to a 17% drop. They arrived at this
conclusion by changing the setup of the model. They claim that the paper’s
conclusions are still valid, according to a post on their web page. Others are
quite skeptical. One of the commentary authors, Solomon Hsiang, noted:
“Science doesn’t work by changing the setup of an
experiment to get the answer you want,” Hsiang said. “This approach is
antithetical to the scientific method.”
Osaka writes that during peer
review, according to a peer review document published by Nature, one anonymous
reviewer wrote:
“I find all of this well explained and fairly
convincing, yet, purely subjectively, I have a hard time in believing the
results, which seem unintuitively large given damages aren’t perfectly
persistent.”
After the paper was
published, the Potsdam Institute for Climate Impact Research put out a press
release claiming that there were $38 trillion in global climate impact damages
each year. That number seems pretty ridiculous.
According to climate impact
scientist Roger Pielke Jr., some in the media relished the paper’s original
conclusions:
“Some went far beyond the paper’s claims. The Washington
Post explained (incorrectly) that the paper showed why food prices were up last
summer:
“In March, a study from scientists at the European
Central Bank and the Potsdam Institute for Climate Impact Research found that
rising temperatures could add as much as 1.2 percentage points to annual global
inflation by 2035. The effects are taking shape already: Drought in Europe is
devastating olive harvests. Heavy rains and extreme heat in West Africa are
causing cocoa plants to rot. Wildfires, floods and more frequent weather
disasters are pushing insurance costs up, too.”
He also notes that the paper
found its way into some important policy groups, including the U.S.
Congressional Budget Office, the OECD, the World Bank, and the UK Office for
Budget Responsibility. In addition, the paper’s conclusion was accepted and
adopted into the Network for Greening the Financial System (NGFS), a consortium
of more than 100 central banks from around the world. It is a forum for the
world’s central banks and regulators to work on a set of climate risk principles.
The error prompted Greg
Hopper of the Bank Policy Institute to write about the paper’s erroneous
conclusions, calling it a “Flawed Analysis With Massive Economic Consequences”
in the subtitle. Hopper wrote his opinion in December 2024, after Nature
indicated that they were looking into the data, but before the source of the
error was found. He emphasized the arbitrary nature of the modeling and the
ability to derive very different conclusions from the same data:
“The statistical procedure used to justify the new
damage model is arbitrary and could easily have produced a damage function that
would have predicted much smaller losses of global real income. Those much
smaller loss projections should not be taken seriously either. In our
assessment of the damage function, we found very limited statistical evidence
for any causation between the climate variables and material economic damage
and no statistical evidence for the purported influence of the temperature variables
that drive almost all the economic damage.”
As a result of the paper, the NGFS
and affiliates, which may include the U.S. Federal Reserve, updated the ‘damage
function’ of their climate scenarios to reflect the paper’s original
conclusions. These should now be further updated, preferably without any
influence from the paper’s questionable re-modeled conclusions. The data issue,
which reveals a possible flaw in some models (where a single data point in a
large data set can drastically alter the conclusion, albeit counterintuitive)
and also shows the risks of climate risk modeling itself. Hopper describes the
current state of academic damage function research as highly uncertain.
Economic modeling is notoriously difficult and gets more difficult the further
it goes into the future. Hopper’s article goes into great detail about damage
function modeling and analysis. Below, Hopper explains how damage function
models work.
Source: The NGFS’s New Climate Damage Function: A Flawed Analysis With Massive Economic Consequences. Greg Hopper. Bank Policy Institute. December 9, 2024. The NGFS’s New Climate Damage Function: A Flawed Analysis With Massive Economic Consequences - Bank Policy Institute
Nordhaus noted that damage
function modeling was challenging. Hopper points out one potential source of
error in Nordhaus’s modeling that assumes global mean temperature equates to a
similar change everywhere, when the reality is that temperatures change by much
different magnitudes in different regions, such as the Arctic, where
temperatures have risen much more than in other regions. This would
theoretically skew the data.
The graph below shows how
much the conclusions changed when NGFS updated their damage function model to
that of the single new paper, ignoring all the other papers. The change is
pretty dramatic, and that in itself should have been a cause for concern or at
least triggered a desire to look further into the data.
Hopper wrote another article,
published just yesterday, after the source of the error was found, noting that
the NGFS should have seen that something was amiss in their damage function
model. He also critiques the new model put forth by the paper’s authors and
discredits it. It is worth reading his conclusion. Below that is a graph
showing what the conclusion would have been for the original model, had the
erroneous Uzbekistan data been removed. This alone shows that moving to another
model is really very likely an attempt to fit the data to pre-determined
conclusions.
Conclusion
In November 2024, central banks and other regulators across
the globe, under the auspices of the NGFS, put into their climate toolkit a
climate damage function that could justify more stringent climate targets for
banks, fines for banks that operate in Europe and potentially additional
capital. About a month after the new damage function was released, a BPI study
showed that the climate damage function had some serious statistical problems.
It should not have been terribly surprising then to learn the climate damage
function also had a serious data error that inflated its loss numbers by about
a factor of three.
What is surprising is the reaction of the NGFS. The
regulators in the NGFS could have, and should have, known about the data error
before putting the new model into their climate scenario toolkit. They also
could have investigated and acted when the academic paper the climate damage
function was based on was called into question by the academic journal that
published it. But they took no action and still take no action.
The revised climate damage model is even more flawed than
the original, since the statistical problems remain and it now appears that the
model update was cherry-picked to reach a pre-determined conclusion. The NGFS
should immediately retract the damage model from its climate scenario toolkit.
The NGFS must also review and correct its procedures for putting its climate
models into production to prevent a model with a serious data error from again
being adopted, especially when the error should have been caught. A general
review and update of NGFS model control procedures is necessary to restore
faith in their climate toolkit and data.
The more troubling questions remain unanswered: did any
central bank or other regulator follow up with Nature or Kotz et al. when it
was publicly revealed that there was a problem with the model they use to
regulate banks? If they did not, why not? Are regulators putting their thumb on
the scale, targeting a pre-determined conclusion rather than using the best
scientific evidence?
Source: The Flawed NGFS Damage Function Is Even More Flawed Than We Thought. Greg Hopper. Bank Policy Institute. August 14, 2025. The Flawed NGFS Damage Function Is Even More Flawed Than We Thought - Bank Policy Institute
References:
This
climate study made a big error. One piece of data was to blame. Shannon Osaka.
Washington Post. August 6, 2025. This climate study made a big error.
One piece of data was to blame.
Too
Big to Fail: A major new scandal in climate science. Roger Pielke Jr. The Honest
Broker. Substack. August 15, 2025. Too
Big to Fail - by Roger Pielke Jr. - The Honest Broker
The
economic commitment of climate change. Maximilian Kotz, Anders Levermann &
Leonie Wenz. Nature volume 628, pages 551–557 (April 17, 2024). The economic
commitment of climate change | Nature
Analysis:
The climate papers most featured in the media in 2024. Carbon Brief. January
15, 2025. Analysis:
The climate papers most featured in the media in 2024 - Carbon Brief
Data
anomalies and the economic commitment of climate change. Tom Bearpark, Dylan
Hogan & Solomon Hsiang. Nature volume 644, pages E7–E11 (August 6, 2025). Data anomalies and
the economic commitment of climate change | Nature
Peer
Review File. Manuscript Title: The economic commitment of climate change. 2024.
TPR1
38
trillion dollars in damages each year: World economy already committed to
income reduction of 19 % due to climate change. Potsdam Institute for Climate
Impact Research. April 17, 2024. 38
trillion dollars in damages each year: World economy already committed to
income reduction of 19 % due to climate change — Potsdam Institute for Climate
Impact Research
Nature
study on economic damages from climate change revised. Potsdam Institute for
Climate Impact Research. August 6, 2025. Nature
study on economic damages from climate change revised — Potsdam Institute for
Climate Impact Research
The
NGFS’s New Climate Damage Function: A Flawed Analysis With Massive Economic
Consequences. Greg Hopper. Bank Policy Institute. December 9, 2024. The
NGFS’s New Climate Damage Function: A Flawed Analysis With Massive Economic
Consequences - Bank Policy Institute
The Flawed NGFS Damage Function Is Even More Flawed Than We Thought. Greg Hopper. Bank Policy Institute. August 14, 2025. The Flawed NGFS Damage Function Is Even More Flawed Than We Thought - Bank Policy Institute
No comments:
Post a Comment