The Cochrane review of covid vaccines under the microscope

The Cochrane review of covid vaccines has received little attention, in contrast to the global media attention for its review of face masks.

Since Cochrane reviews have been praised as “rigorous” and “trustworthy,” it is prudent to also look critically at the latest update on “Efficacy and Safety of Covid-19 Vaccines.”

Cochrane review results?

The review, published in December 2022, analyzed 41 randomized controlled trials of 12 different vaccines, involving more than 400,000 people without prior SARS-CoV-2 infection.

The review shows that most studies lasted no longer than two months and were conducted before the emergence of variants of concern such as Omicron (until November 2021).

Compared with placebo (mostly saline), the authors conclude that most of the available vaccines were able to reduce symptomatic covid with “high certainty” and in some studies also reduced severe or critical illness.

It also notes that there is “likely little or no difference between most vaccines and placebo in serious adverse events.”

Due to study exclusions, the authors acknowledged that the results could not be generalized to pregnant women, people already infected with SARS-CoV-2, or immunocompromised persons.

A flawed review?

A critique of the Cochrane review was recently published by researchers Peter Doshi, Joseph Fraiman, Juan Erviti, Mark Jones and Patrick Whelan.

The criticism expresses serious concerns about the conclusions drawn by the Cochrane authors.

Doshi et al. are the same researchers who reanalyzed the pivotal mRNA vaccine trials and found that one in 800 people vaccinated experienced a serious adverse event (SAE).

This contradicts the conclusion of the Cochrane review, which states that there is “little or no difference in SAEs compared with placebo.”

Doshi et al. point to significant flaws in the Cochrane review that could explain the conflicting results.

For example, the SAE tables in Moderna’s trial reports included efficacy data from people who had severe covid and who were almost exclusively in the placebo group. Cochrane did not remove this efficacy data before presenting its analysis of the Moderna vaccine’s SAEs, tabulating results that dilute the true rate of harm.

The Cochrane review also shows that the absolute difference in SAEs between most covid vaccines and placebo was “less than 5/1000 participants.” This equates to 1 in 200 people, which is not rare and contradicts Cochrane’s statement that there is “little or no difference“.

According to Doshi et al. another flaw in the Cochrane review is the use of a “composite endpoint” to categorize cases of “severe or critical covid”.

Doshi’s team points out that the endpoint includes many participants who were not “severe or critical.”

For example, in the Moderna study, 21 of 30 cases were not hospitalized.

There was also a problem with the counting of “covid cases” in the studies, and this problem was taken into account when synthesizing the evidence in the Cochrane review (garbage in = garbage out).

The study sponsors waited 1 week (Pfizer) or 2 weeks (Moderna) after dose 2 before counting covid cases.

This means that covid cases were only counted 4 and 6 weeks after dose 1 was administered (see table).

Doshi et al. consider the exclusion of all cases in the 4-6 weeks after dose 1 to be “particularly concerning” because the follow-up period was very short before the manufacturers decided to unblind the studies and offer the vaccine to the placebo group (median follow-up time 2 months after dose 2).

Another issue pointed out by Doshi et al. point out is the fact that the mRNA vaccines were very “reactive” and were likely to “unblind” the participants and distort the results of the study. Participants in the vaccine group took more fever-reducing medication after treatment than participants in the placebo group, making it easier to guess which treatment they had received.

Nevertheless, the Cochrane reviewers rated the risk of bias due to blinding in the study as “low”.

The Cochrane study also points out that it is unclear whether and how vaccination protection decreases over time and that the results of the study are current.

Doshi et al. however, find that this is incorrect. The Cochrane review contains two papers in which Pfizer actually showed that effectiveness against covid decreases over time – vaccine effectiveness had fallen to 84% ≥4 months after dose 2.

Finally, the studies analyzed in the Cochrane paper mainly looked at healthy people who had never been exposed to the virus, meaning that the effectiveness results no longer apply to most people who recover from one or more infections diseases. Likewise, the studies cannot be transferred to pregnant women or people with weakened immune systems. This was not highlighted by the Cochrane reviewers, although it is a significant limitation.

Cochrane’s answer?

A media request containing a questionnaire was sent to the Cochrane study’s lead author, Isabelle Boutron, professor of epidemiology at the Université Paris Cité and director of Cochrane France.

Boutron confirmed that her team followed Doshi et al.’s comments. They received the request to express their opinion, but did not respond to further questions, but only wrote an email:

“We are working on the answers and it will take time to review them and respond appropriately to each point. We will of course inform you when our answers are ready.”

There is no time frame for Cochrane authors to respond to criticism of their reviews – some have taken several years to do so. A critique of the Cochrane review of HPV vaccines by Doshi et al. from 2018 has not yet been answered.

If Cochrane accepts the criticism of Doshi et al. and the review is revised accordingly, the result will certainly be completely different from the one we have now.

 

yogaesoteric
December 4, 2023