A study on corona measures is making waves. State lockdowns are said to have saved hardly any lives. But the paper by three economists should be treated with caution.
Pandemic, lockdowns, corona deaths. With these ingredients you can often be sure of public attention. Now a study has made it into the media limelight – initially bypassing scientific control. But that’s not the only strange thing about the paper.
Claim: Government measures taken during the pandemic have had little or no impact on the number of people who have died related to Corona. This is the result of a study by three economists published on the website of the Johns Hopkins Institute for Applied Economics.
Evaluation: The investigation raises an important question, but cannot answer it conclusively. Experts name several points of criticism.
Facts: It is undisputed that a reduction in contacts slows the spread of infectious diseases such as Covid-19. This also leads to fewer deaths. However, the question of whether additional state-imposed measures will help is the subject of the study by the three economists, which is currently attracting a lot of attention in the media.
What does the study examine – and what not?
Jonas Herby, Lars Jonung and Steve Hanke describe their paper as a so-called meta-study, which summarizes the data from individual studies and working papers as a kind of overview. The authors wanted to check: Is there evidence to suggest that government-imposed lockdowns had an additional effect on Covid-19 mortality – compared to the measures that the population voluntarily implemented anyway?
The study therefore makes no statement as to whether general measures such as reducing contact, wearing a mask or hand hygiene were superfluous. Information campaigns by the authorities or test capacities provided are also not considered.
What is the study not able to say anything about?
The results of the meta-study do not show how the government measures worked during the course of the pandemic and under other conditions – such as the availability of vaccines. Because the individual studies she examined only deal with the first wave of infections in spring 2020.
What is not considered at all in the study: “We exclude studies that use cases, hospital stays or other parameters,” it says explicitly. This means that the analysis cannot be used to make statements as to whether government measures influence the number of corona infections or the number of serious illnesses, for example. But the threat of overburdening in hospitals and care has always been the justification for very far-reaching regulations for politicians.
Which individual studies are included – which are not?
The authors want to have identified 18,590 studies that could potentially deal with their question. However, only 34 studies come into question, of which only 24 are ultimately included in the meta-analysis – in addition to studies reviewed by experts, there are also working papers that have not been evaluated.
There is a wealth of scientifically much higher quality studies, “but they were not considered on the basis of the selection criteria chosen by the authors,” said the head of the Institute for Health Services Research and Clinical Epidemiology at the University of Marburg, Max Geraedts, the German Press Agency (dpa) with.
There is also skepticism about the meta-analysis because the weighting of the studies used is not clearly comprehensible. Statistics professor Christoph Rothe from the University of Mannheim, for example, tweeted: “In the meta-analysis written by economists […] studies by non-social scientists (e.g. epidemiology) are automatically classified as “of lower quality”.»
The economist Andreas Backhaus from the Ludwig Maximilians University in Munich analyzed that some of the individual studies examined were “not overly convincing”. However, they received “a very high weight in the meta-analysis, so they drive the overall result”.
What is the concrete result of the meta-study?
In their paper, Herby and his colleagues come to the conclusion that state-regulated measures worldwide have had little effect compared to recommendations and voluntary changes in behavior by the population: In the first corona wave in spring 2020, the studies examined showed that the Covid -Death rate reduced by prescribed regulations by only 0.2 percent.
Herby explains specifically: “There were a total of around 300,000 Covid 19 deaths in Europe and the USA during the first wave of lockdowns,” he writes on Twitter. “Without lockdowns, according to scientific estimates, it would have been 300,601.”
In a detailed article on the meta-analysis, Herby explains that this does not necessarily lead to the conclusion that lockdowns have had no effect in any country in the world. If governments had found “the right timing” for their actions, the rules could have had a major impact.
The 0.2 percent refers to all state lockdown measures as a whole. On the other hand, the meta-analysis attributed a clear effect to individual regulations in terms of the number of deaths – such as wearing a mask at work or closed clubs and bars.
What does the study mean by lockdown?
Herby and colleagues define lockdown “as the imposition of at least one mandatory, non-pharmaceutical intervention”. These include state orders that affect people directly – such as restricting freedom of movement or banning cross-border travel.
The team led by mobility researcher Kai Nagel from the Technical University of Berlin does not consider the term lockdown to be very meaningful. “In Germany, this was primarily understood to mean a large-scale shutdown of public leisure, retail and schools,” the experts explain. On the other hand, there has been little intervention in the labor sector in Germany compared to other countries.
It is particularly problematic that the meta-analysis refers to the “Government Stringency Index” of the University of Oxford. “The studies on the Stringency Index show that lockdowns in Europe and the United States have reduced Covid-19 mortality by an average of only 0.2 percent,” the authors write.
But this index has a massive disadvantage. Because he only ever observes the strictest measures that apply no matter what administrative level in a country. Example Germany: If federal states or even individual districts temporarily have stricter rules than are provided nationwide, the Stringency Index treats this as if the measures applied nationwide.
“This index cannot objectively evaluate the actual efficiency of various measures taken,” said Nagel and colleagues. The epidemiologist Geraedts from Marburg finds “particularly problematic” that it is unclear to what extent these government measures “were actually enforced in the various countries considered”.
How is the work assessed in specialist groups?
The Berlin mobility researchers around Nagel see in the study by Herby and colleagues “first steps” to better understand the effect of official orders, as they write on dpa request. However, the team points out that the German population had already restricted their mobility during the first corona wave, “before the formal restrictions began” – and that people were on the move again before politicians officially ended the formal restrictions. A reaction of the population depends on the given rules.
One difficulty – also in this meta-analysis – is therefore to work out what part state regulations have in the actual behavior of people.
Geraedts also points out that the populations in the different countries have already adapted their behavior to the pandemic without mandatory measures. He accuses the authors of deliberately trying to prove the statement you want with an “eclectic compilation of literature”.
Who is actually behind the study?
The authors are not epidemiologists, virologists or other medical professionals. Steve Hanke is Professor of Applied Economics at Johns Hopkins University in Baltimore, Maryland. The paper by the three was published on his university server at the end of January. In the advertisement for the study, he wrote on Twitter at the time: “Lockdowns are for losers.” In addition, he has attracted attention in the past for publicly describing state corona measures in Italy or Germany, for example, as “fascist”.
In addition, Hanke is a senior scientist at the Cato Institute, a US economic policy think tank that claims to have “libertarian principles” that oppose state influence in business and society.
Jonas Herby is an advisor to the political-libertarian think tank Cepos in Copenhagen. According to his own statements, his main areas of focus are law and economics. The third author is Lars Jonung, a retired economics professor who taught at Lund University in Sweden.
Why is the path of publication strange?
The study was not published in a peer-reviewed journal, but on the website of one of the economics institutes at Johns Hopkins University at the end of January. “The good reputation of Johns Hopkins University was used to give this working paper a high level of credibility,” explains expert Geraedts from Marburg.
“The series of publications allows students, employees and former members of this institute to present their work for discussion,” says Geraedts. Quality criteria that working papers have to meet are not specified on the institute’s website.
The results of Herby and Co. were therefore not examined in a comprehensible test procedure before publication. “In this way, the authors avoid peer review, one of the most important quality assurance measures in science,” said virologist Friedemann Weber from the University of Giessen. “Self-publishing studies is absolutely unusual and unscientific.” Geraedts also accuses the authors of “consciously not having chosen the path” of having their methodology and the results and interpretations obtained from it checked by independent scientists.