Update: Bad controls everywhere
Update: the authors have clarified that not including State Capacity or GDP related variables does not change the direction of the presidential variable. I'm truly puzzled. Trying to organize my thoughts around this at the moment.
I believe one of the greatest shortcomings in how we have been producing statistical papers is the lack of attention to what controls should be included. While the absence of a control is vigorously (and often rightly) attacked, the inclusion of a bad control rarely raises concerns. This gives rise to the problem Garett Jones calls "Everest regression": "controlling for height, Mount Everest isn’t that cold."
As Cinelli, Forney and Pearl put it (free here):
The problem of “bad controls” however, has not received systematic attention in the standard statistics and econometrics literature. While most of the widely adopted textbooks discuss the problem of omitting “relevant” variables, they do not provide guidance on deciding which variables are relevant, nor which variables, if included in the regression, could induce, or worsen existing biases. Researchers exposed only to this literature may get the impression that adding “more controls” to a regression model is always better. The few exceptions that do discuss the problem of “bad controls” unfortunately cover only a narrow aspect of the problem.
You should definitely check the full article, but let me try to summarize the issue. A good control will remove bias from an estimate, by not making the effect of an unspecified variable look like it is the effect of the independent variable you are examining. Classic example of a good control: if you run a correlation between the number of people who took their umbrella out of the house on a sunny day and the rain episodes, you would find that taking umbrellas would have a very large correlation with rain, and because it precedes it, you might infer umbrellas cause rain. If you control for the weather forecast on the day, however, the relationship would mostly disappear, as it should.
A bad control, on the other hand, introduces bias into the regression. Let's say we want to examine the effect of drinking alcohol the day before a test and test scores. We run a regression, find a huge negative effect. But assume that we can control for the quality of sleep the night before (using monitors, for example) and find out that the effect mostly vanishes. Is this uninteresting? Certainly not, it gives evidence (assumed evidence, to be sure, I don't have real information on this) that the main mechanism which makes students do worse on tests is through bad quality of sleep. The problem arises when one tries to use this as evidence that drinking alcohol will not affect test scores. It will, because it will affect your sleep, and that in turn will affect the test scores. So if you want to assess the effect of drinking alcohol on test scores (instead of assessing the mechanism by which alcohol affects test scores) you should not control for quality of sleep. In this case, quality of sleep is an intermediate outcome, one of the cases of bad controls.
Unfortunately, the inclusion of bad controls in the literature comparing presidential and parliamentary systems is all too common. A noteworthy example is the recent chapter by Hicken, Baltz, and Vasselai (HBV) - which they generously shared with me - in the new and impressive "Why Democracies Develop and Decline". HBV set out to compare how presidential and parliamentary systems compare in achieving democracy. The problem is that they include in their model state capacity.
State capacity, as they put it, is comprised of "coercive capacity, administrative capacity, and legitimacy." Coercive capacity is "the state’s monopoly on the legitimate use of force within its territory." Administrative capacity is " capacity of the bureaucracy to design and effectively implement public services and regulations across a country’s territory." Lastly, legitimacy "refers to agreement by the citizenry about the boundaries of the state and the rules for inclusion and exclusion."
When HBV run their regressions, they find that while state capacity has a huge effect on democracy levels, parliamentarism does not. In fact, parliamentarism has a negative effect.
But readers of the book will realize that state capacity is a clear outcome of parliamentarism. The main benefits of parliamentarism are directly associated with causing them, and the good outcomes listed in the book (economic growth, less inequality, less corruption) are derived from a monopoly on the legitimate use of force, a capacity to implement public services, and legitimacy. How could they not? How could parliamentarism have no effect on these variables yet still promote democracy? How can democracy be promoted when the State does not have those qualities?
As evidence that most people do agree that state capacity is clearly related to democracy levels, the huge relationship between state capacity and democracy levels is not the result of the chapter which seems to be getting attention, it is the negative result of parliamentarism (for example Democracy and Autocracy April 2021 and Justin Kempf's excellent Democracy Paradox podcast). This is probably because no one is surprised by the finding. In this case, what state capacity is doing in their models is serve as a control, which should not be there.
I suspect that this chapter will be hugely influential, being a part of a book which has every element of an instant classic. I would regret if this meant that people would conclude that parliamentarism is not a major promoter of democracy.