The purpose of this proposal is to begin to develop rules for the proper use of climate and other models in federal rulemaking. The use of wildly speculative and extreme modeling is spreading throughout the government, with potentially destructive consequences. This destructive growth needs to be opposed, but effective opposition requires the articulation of specific constraints on the use of speculative models in rulemaking.
The problem: federal use of speculative modeling is out of control.
For example, the extreme and entirely speculative “social cost of carbon” (SCC) modeling has been used to justify dozens of onerous and costly regulations, from multiple agencies. Moreover, a federal court has ruled that SCC must also be included in all NEPA determinations that involve CO2 emissions.
Arbitrary and capricious climate modeling is also being promoted in the planning realm. In coastal areas it is being used to project absurdly high sea level rise. FEMA has proposed requiring so-called climate change planning in its disaster preparedness planning program.
Extreme climate modeling is a green cancer that is spreading throughout the federal government rulemaking system. This destructive growth needs to be opposed, but effective opposition requires the articulation of specific constraints on the use of speculative modeling in rule making.
Solution: Rules to constrain the improper use of modeling
This proposal begins to articulate a set of rules for the proper use of models. The present constraints on rulemaking do no adequately address the arbitrary and capricious use of speculative modeling.
The goal is to develop principles, as well as specific guidance, that can be used to constrain the speculative use of models in federal rulemaking. Potential applications of these rules include their incorporation into legislation, executive orders, OMB and OSTP rulemaking guidance and public comments on proposed rules. These rules might also support litigation.
The central concept here is uncertainty. For example, climate science is extremely uncertain. In fact there is a large academic literature on this uncertainty, but this literature has failed to constrain the excessive use of speculative climate modeling in federal rulemaking. What is required is to translate this uncertainty into specific rules, including both principles and specific language.
To begin with, below are some candidate rules to be explored. Further suggestions are welcome.
The basic principle is that modeling results are usually merely speculative hypotheses. They need to be presented and treated as such. Model results are not scientific facts about the real world.
Candidate rules for the use of models in rulemaking
A. Computer modeling of regulatory costs and benefits often involves significant uncertainty. This uncertainty should be presented in detail.
B. In-house models versus external models. For external models present known peer criticisms. For internally developed models seek and present peer critiques. Peers should be true critics, not pals.
C. If many models are available then consider significantly different models and present their different results. These differences are part of the uncertainty.
D. Do not use worst case scenarios. These are typically unrealistic and they bias the results.
E. No guessing. Do not claim to model what is clearly unknowable.
F. State significant assumptions and simplifications.
G. Identify known unknowns.
H. Do not claim unwarranted accuracy.
I. If some models or runs give “no action” results then report this finding.
Comments on these proposed rules and suggestions for others are encouraged.
An important issue well spotted. Another approach to this comes from Paul Pfleiderer writing on the The Misuse of Theoretical Models in Finance and Economics. He coins the term “chameleon models” and explains:
“In this essay I discuss how theoretical models in finance and economics are used in ways that make them “chameleons” and how chameleons devalue the intellectual currency and muddy policy debates. A model becomes a chameleon when it is built on assumptions with dubious connections to the real world but nevertheless has conclusions that are uncritically (or not critically enough) applied to understanding our economy. I discuss how chameleons are created and nurtured by the mistaken notion that one should not judge a model by its assumptions, by the unfounded argument that models should have equal standing until definitive empirical tests are conducted, and by misplaced appeals to “as-if” arguments, mathematical elegance, subtlety, references to assumptions that are “standard in the literature,” and the need for tractability.”
Pfleiderer’s essay:
https://www.gsb.stanford.edu/sites/default/files/research/documents/Chameleons%20-The%20Misuse%20of%20Theoretical%20Models%20032614.pdf
My synopsis:
https://rclutz.wordpress.com/2016/05/28/cameleon-climate-models/
I should add here Pfleiderer’s questions to attest models:
“Whereas some theoretical models can be immensely useful in developing intuitions, in essence a theoretical model is nothing more than an argument that a set of conclusions follows from a given set of assumptions. Being logically correct may earn a place for a theoretical model on the bookshelf, but when a theoretical model is taken off the shelf and applied to the real world, it is important to question whether the model’s assumptions are in accord with what we know about the world. Is the story behind the model one that captures what is important or is it a fiction that has little connection to what we see in practice? Have important factors been omitted? Are economic agents assumed to be doing things that we have serious doubts they are able to do? These questions and others like them allow us to filter out models that are ill suited to give us genuine insights. To be taken seriously models should pass through the real world filter.”
Very well said, and thanks for the great reference, Ron. My way of putting it is that in science a model or model run is just an untested hypothesis. Playing with hypotheses is very useful, but it is not telling us much about the world. It is part of the theoretical side of science.
In climate there is an unfortunate confusion of untested climate models with proven weather models. Standard weather models are really engineering tools, not science. The models are tested day after day, year after year, so that users know well what they can and cannot do.
Climate models are nothing like this. Their forecasts are so distant as to be untestable in any meaningful sense.
This confusion has been carried to absurd lengths at NOAA, where there has been a push to create a National Climate Service, in parallel with the National Weather Service. So far this has not happened, but the proposals are still sitting there on the shelves.
Really would be interested in how you support
“… untested climate models…”