History as an Empirical Science
I weigh in on Noah Smith's argument with Bret Devereaux
Intro
It is apparently touchy subject whether or not History is an empirical science. There are a lot of strong opinions that even when Historians form beliefs about the future this is not based on a predictive model, it’s based on something different, which despite resulting in beliefs about the future, is definitely not a predictions.
Noah Smith wrote a great piece on this topic, where he used an article by Bret Devereaux as an example, where Bret insisted his analysis is not predictive, yet goes on to make claims about the future based on historical analogy. This is apparently not an uncommon belief, as he had enough support for it to become part of the twitter discourse.
I initially expanded on this, as well as Matthew Yglesias’ point in a thread, but I want to flesh it out more here.
While these points feel obvious to me, there are a lot of people who fundamentally disagree with the framework that learning from history is fundamentally empirical and predictive. The most common criticisms point out that there is this distinction between understanding the past and empirical social sciences. The Historian Noah was originally responding to, Bret, raises this in a subsequent follow-up post where he defends the idea that history represents a different type of science.
Qualitative and Quantitative Information
A substantial portion of the criticism against empiricism as the correct form of analysis to study History, focuses on the ways in which so-called qualitative and quantitative forms of social science analysis differ. The implication here is that empiricism is for numbers, whereas words are qualitative. While it’s true that some types of analysis focus more on words, and other more on numbers, from an information theoretic perspective, these are just different symbolic representations of the same underlying information set.
This is a common mistake, where certain types of scholars confuse the structural limitations of tools as reflective of the scientific methodology itself. For example, Bret claims that once you switch to quantitative analysis, you lose data, and says that “The conversion or compression of fuzzy, non-numerical evidence to data is thus not free, it is a ‘lossy‘ process.” — which can be true. But he continues to state that as a result this is a different epistemology. His conclusion is that the this therefore requires different types of science for qualitative and quantitative analysis.
It’s true that certain forms of qualitative analysis allow the researcher the ability to integrate large amounts of sparse often disparate data sources, ranging from the aesthetics of the time, the art scene, the political sensibilities, and other rich primary historical data sources. Computers aren’t that great at this stuff yet; the human brain is remarkably strong at bringing together unstructured information. This is why fundamental hedge funds, which would eventually go bankrupt if they couldn’t make money, still have a seat at the table and can compete against pure quant-funds.
On the other hand, quantitative information considers only the forms of information that can ultimately represent themselves as a matrix, loaded in the memory of a computer. This gives you the full power of Amazon Web Services computation at your finger tips, but you lose the dynamism of a lot of unstructured and higher-dimensional analysis.
It makes sense that different people will end up spending more time and being better at one form of analysis, it’s tough to master everything. But the fact that we must deal with structural limitations of the information in different ways shouldn’t imply there are multiple different types of science, or ways of ‘knowing things.’ Instead, It tells us that we currently lack the mechanical tools to treat different types of information as interchangeable. These methodologies have not been invented yet.
A vector of numbers representing historical information feels rigorous, when placed within the confines of mathematical or statistical jargon. Poetry from the past appears soft, and less rigorous. These feelings though don’t reflect the more fundamental information content, they reflect our tools. In both cases it’s information recorded from the past, which we can use to inform our belief about the future.
In fact, over the past decade we’ve learned that poetry can also be placed in a vector if we want, and theory aside, we can literally construct language models that can take in all the poetry from a past era, and simulate similar poems. Our forms of analysis will continue to slowly converge, because the true analysis of information doesn’t distinguish between numbers and words. Matrices or poems.
We can even look at another abstraction here: The type of researcher who prefers studying historical poetry to learn about the human condition probably has a different disposition and personality than the type of researcher who prefers to use mathematics to study the past. This may be reflected in their preferred forms of inferences, aesthetic preferences, political views, and who else know what it’s correlated with. But again: These don’t reflect a difference in the scientific approach, they reflect a difference in humans themselves.
Prediction
If we take qualitative and quantitative to be different forms of analysis on the same set of information, this also forces us to reduce what it means to form a prediction. The simplest definition of a prediction is using past information to define any structured belief, however uncertain, about the future.
What I think is happening here, is this reflects a misunderstanding of prediction, as this formalized, probably quantitative, model, that makes deterministic predictions about what will definitely happen in the future.
So from Bret’s perspective, he is trying to warn people of a specific way in which the future could unfold, because as a Historian he has probably read old books most of us haven’t. The uncertainty in his belief, and the fact that it doesn’t use numbers, means he doesn’t consider it a prediction. In his words “The result is not a prediction but rather an acknowledgement of possibility”.
This is just a misunderstanding of how we use probability as a tool. The language of probability lets us create predictions to represent arbitrary levels of uncertainty. In the quote above, if he has located an event in the past, defined it as an event that could happen in the future, and warned people this could happen with a non-zero probability. That’s still a prediction. Uncertainty only has to satisfy the Kolmogorov axioms, and uncertainty also reflects unknown unknowns.
But to take a different indirect path here: Wouldn’t it be absurd if the only way to represent certain forms of uncertainty was with wordy historical analysis? That would seem like a pretty glaring oversight for statistics as a field, if once things became too uncertain, you had to move on from probability theory, and instead write essays that vaguely hint at things. This would only be true if words and essays were somehow able to embed a way of representing beliefs that was off-limits to the language of probability and statistics — but I see absolutely no reason to believe this should be true.
My point here isn’t that writing down a probabilistic prediction is better than an essay, it’s often not, but rather that it’s still the correct framework within which to reason. When we speak of the possible futures, defining probability distributions over the future is something we’re doing implicitly with our words.
Historical Counterfactuals
In order to make a prediction, we need the ability to learn a model from the past, and this requires counterfactual reasoning, which is the primitive unit of historical inference.
To begin with what counter-factual inference is not: Bret writes
We tend to refuse to engage in counterfactual analysis because we look at the evidence and conclude that it cannot support the level of confidence we’d need to have. This is not a mindless, Luddite resistance but a considered position on the epistemic limits of knowing the past or predicting the future.
This response misunderstands counterfactual analysis. The framework of reasoning about the world in counterfactuals doesn’t only apply when you’re confident. In a lot of cases it’s true, we can’t learn much from the past, it’s too hard. So what? Wouldn’t it be strange if the only time you can do counter-factual analysis was you had a high level of confidence?
The framework for reasoning still works, even if uncertainty ruins our attempts to learn from the past. There aren’t different sciences you pick and choose from depending on your certainty. There is one scientific method, and sometimes we don’t have enough data to estimate anything with certainty. Or we only have enough data to suggest things are possible with deep uncertainty.
Nothing is wrong with trying to imagine what the world would look like if Germany won the second world war, but realistically we know it’s too complicated to have any real clue how the world would have unraveled over the coming years. That doesn’t mean it’s incorrect science. It’s a fine scientific question, we just don’t know the answer.
In order to understand counter-factual analysis, you need to understand the philosophy of science behind modern causal-inference. The approach is to take the treatment and control framework of randomized-control trials, and apply it to observational studies. Typically, this works by looking for some natural experiment and exploiting it.
From this perspective, we actually perform counter-factual analysis when we read history and try and learn from it, even if it’s not formalized. If you want to claim something is important, you’re effectively saying that it was a treatment at some point in time of the historical process, and history diverged as a result. Or as Adam Shapiro in an article on counter-factual history writes:
Tacitly, we historians make decisions like this all the time: when we say that a particular battle, a particular election, a particular act of human will matters—we imply that if these hadn’t happened, history would have been different. This is what it means to make claims about cause and effect,
We can, for example, cast Bret’s argument as a historical counter-factual — even though he claims it is not one, and it’s a different form of science:
His claim is that ancient Greece represents a comparative model for America. He notes that the greatest risks to Greece were from tyrants. Even though these tyrants failed, or seemed farcical, they kept trying to gain power, and when they finally got it, were ruthless.
In his model, there was the observed outcome, which is the tyrant finally gained power and “promptly set about using his position to butcher the best and brightest of Corinth and in so doing secured his reign for the rest of his life.” The unobserved outcome implicit in his argument, is that if the populace took the risk of him becoming a tyrant seriously and took some action (aka treatment effect), then this bad outcome may not have happened.
Of course, we don’t know what would have happened. Uncertainty remains in this counter-factual. But through his analysis, Bret is effectively estimating what he believes might have happened with some implicit probability. If the Greek populace had killed the tyrant when he kept trying to gain power, we can probably say with things would have been better, since him becoming tyrant seemed to be a pretty bad outcome.
What I want to stress here, is that from the perspective of our philosophy of science whether this counter-factual is estimated using a written essay, a regression, difference-in-difference, or machine learning, makes no difference. Estimation is not only true when it occurs in the hardware of your computer, it also applies to the hardware of your brain. Any computation device that tries to estimate this, is engaging in the same scientific act.
Bret then takes this simple counter-factual model, and pattern matches it to what he believes is happening today. This part involves pattern-matching past models to current events.
What Bret claims is this simple counter-factual model he built in the past, if we think it matches to the present, can provide us with some indication of what actions we may want to take now to prevent the worst outcome of our future, from looking like the worst outcome the Greek’s realized.
Despite all his claims that this is a different type of science, it certainly seems to fit within the empirical structure of building models and applying them to generate out-of-sample beliefs about the future.
Science
Borges wrote “Nothing is built on stone; All is built on sand, but we must build as if the sand were stone.“ I feel this same way about the scientific method. It’s a construct that allows us to make inferences about the world. We’re still developing the method, and learning about how the scientific method should be updated as we go.
What I deeply believe, is that there exists only one form of the scientific method. Physicists, Historians, and quantitative social scientists, don’t all use distinct forms of science. The structure we place on top of reality, and the framework we force between computation and data in order to learn about our world, is the same across all disciplines, even if our tools differ.
We’re all embedded within the same space, trying to understand facets of our world, and a core shared primitive in all areas of science is prediction: To know something about the process of reality well enough to predict where it will be in the future. You don’t get to side-step that one.