Without Rigor and Transparency, There Can Be no Science: about Surgisphere and its Scientific Publications in High Impact Journals

[This text has been written by: Alberto García-Basteiro, Assistant Research Professor at ISGlobal and physician at the International Health Service of the Hospital Clínic; Joe Brew, data scientist; and Carlos Chaccour, Assistant Research Professor at ISGlobal and Chief Scientific Officer of BOHEMIA]

During this pandemic, there have been notable changes in the way scientific information is disseminated. In recent months there has been an exponential growth in the number of COVID-19  related publications, to the point where traditional scientific journals have collapsed due to the vast number of submissions. Due to this collapse, and motivated by the urgency of sharing information quickly during a health emergency, much of the research is now published through pre-prints, articles posted on the Internet without going through the usual peer review process. Many have highlighted the problems inherent in this phenomenon (errors, lack of rigor, methodological problems, etc.); but these problems are not exclusive to pre-prints. The speed and desire to make scientific results available in record time (by both authors and editors) has resulted in the publication of manuscripts of questionable quality in the world's most prestigious journals.

Recently, numerous concerns have arisen about a publication in The Lancet, which led to the temporary stop of multiple clinical trials evaluating hydroxychloroquine as a possible treatment for COVID-19, including part of the Solidarity Trial, a scientific project led by the World Health Organization comparing four treatments for hospitalized patients with severe COVID-19 (in Spain there are 20 clinical trials registered with hydroxychloroquine or chloroquine). The reason is that this publication from The Lancet associated higher mortality and risk of arrhythmias to patients who had taken chloroquine or hydroxychloroquine. The study reportedly includes more than 96,000 patients from 671 hospitals on all continents.

This is not a war about the usefulness of hydroxychloroquine for COVID-19, but about the credibility and transparency of scientific publications that have major public health implications

The authors claim to have used data from Surgisphere, a platform of "anonymized data obtained through automated extraction from electronic patient health records”. Therefore, it is assumed that there is some sort of collaborative agreement with hundreds of hospitals around the world that use electronic health records and allow this private company to obtain real-time and automated patient data from these facilities. If this were true, it would go directly against several points of the EU General Data Protection Regulation (GDPR), the US-EU PrivacyShield collaboration, as well as the national laws of many states, not to mention the ethical rules of research institutions.

Less attention has been paid so far to two other publications that use this same Surgisphere database, and in which several of the authors of The Lancet publication participate. The first is an article published the same month of May in the high impact New England Journal of Medicine (NEJM) on mortality associated with patients with cardiovascular pathology and COVID-19; the second is a preprint on the benefits of a drug, ivermectin (which is generally used as an antiparasitic), for the treatment of COVID-19. The latter, despite not having been published in a traditional journal or subjected to expert review, has contributed to the inclusion of this drug in the national therapeutic guidelines for COVID-19 in Peru and to the massive administration of this drug to 350,000 people for the treatment or prevention of COVID-19 in Bolivia.

Given the numerous irregularities that we have been able to observe in the three articles coming from this company, it is our position that the results and conclusions cannot be trusted

Given the numerous irregularities that we have been able to observe in the three articles coming from this company, it is our position that the results and conclusions cannot be trusted. Among them, inconsistencies in the numbers of patients included in some countries (including Spain) and/or continents with respect to the data made public by the countries and the World Health Organization itself, the dubious process of automated transfer of individualized data from many countries where electronic medical records do not usually exist, the absence of the obligatory ethical evaluation for this type of scientific study or the lack of transparency in the analysis (no code or anonymized base available to be contrasted). Many of these arguments are shared by respected researchers.

We also find it shocking that the supposedly largest and most sophisticated clinical database of hospitalized patients in the world is totally unknown to the international scientific community, that the processes of cleaning and analyzing huge amounts of data are done in just a few days (judging from the date of data collection to the date of publication), and that its first scientific publications are in two of the most prestigious medical journals that exist. This impressive debut, although infrequent and surprising, is no reason per se to suspect the rigor of the studies. However, these are not the only elements about the company that have sown so much disbelief. Although the work involved in creating, maintaining and cleaning up such a database is enormous, the articles are signed by 5 or 6 authors and, for alleged reasons of "data transfer agreements" with their clients, not one of the participating centers is mentioned. The statements of the CEO of Surgisphere and co-author of the articles questioning the need for clinical trials in the face of the evidence that emerges from the analysis of such a database is simply unacceptable and denotes a lack of scientific rigor and unreasonable boldness.

During emergencies, it is easy to forget the need for rigor in science, but it is now, more than ever, that evidence, and not mere opinions, should guide our decisions

In science, the threshold for determining what is "true" has to be high. Extraordinary claims (for example, having a database so powerful that clinical trials are no longer needed) require extraordinary evidence. Without it, the default position has to be doubt and skepticism. We find it extraordinary that Surgisphere has the data it claims to have. Not because we have anything against this company, but because this company has not provided enough information to exceed reasonable evidence thresholds. This is not a war about the usefulness of hydroxychloroquine for COVID-19, but about the credibility and transparency of scientific publications that have major public health implications.

Although we alerted the journals last week to the need to check the information contained in these articles, we are disappointed by the speed at which questionable results are disseminated and the slowness with which they are corrected. In our opinion, given the inconsistencies in these articles, it would be unwise to "trust" and more responsible to "doubt". Yesterday, finally, both The Lancet and NEJM published expressions of concern about the data base and conclusions of the studies. We are against any decision arising from results published by this private corporation as long as no further information is provided on the above elements or transparency on the database is increased. During emergencies, it is easy to forget the need for rigor in science, but it is now, more than ever, that evidence, and not mere opinions, should guide our decisions. We need evidence, rigor and the scrutiny of the scientific community always, but especially during a health emergency. In the case of Surgisphere, no elements have been provided to reproduce their analysis, and yet they have managed to publish in two of the most prestigious journals in the world in the same month. This episode is damaging the confidence in science when we need it the most.