doi: 10.56294/mw202454

ORIGINAL BRIEF

Availability of retracted Covid-19 papers on Internet research-sharing platforms

Disponibilidad de los artículos retractados de Covid-19 en plataformas de intercambio de investigación en Internet

Renan MVR Almeida¹ *, Aldo José Fontes-Pereira^2,3 *_,

¹Programa de Engenharia Biomédica COPPE/UFRJ. Brazil.

²Centro Universitário Serra dos Órgãos–Unifeso, Teresópolis, Rio de Janeiro, Brazil

³Universidade Federal do Rio de Janeiro, Faculdade de Fisioterapia, Rio de Janeiro, RJ, Brazil

Cite as: MVR Almeida R, Fontes-Pereira AJ. Availability of retracted Covid-19 papers on Internet research-sharing platforms. Seminars in Medical Writing and Education. 2024; 3:54. https://doi.org/10.56294/mw202454

Submitted: 11-10-2023          Revised: 06-12-2023          Accepted: 10-02-2024          Published: 11-02-2024

Editor: Dr. José Alejandro Rodríguez-Pérez

ABSTRACT

Introduction: this paper investigated the availability of retracted/with Expression of Concern Covid-19 papers on research-sharing platforms.

Methods: from the “Retraction Watch” (RW) list of Covid-19 retracted/with Expressions of Concern (EoC) papers, all articles pertaining to Covid-19 treatment were selected. After their identification, paper titles and authors were searched on the platforms: Research Gate and Academia (Academia.com). In case a retracted or EoC paper was identified as available, the presence of a warning note was ascertained (either as an attached note or as a direct warning on the paper). The citations that these papers received were then identified on the Google Scholar platform, and classified as prior to retraction date/posterior to retraction date.

Results: at first, a total of 44 papers were selected from the RW list. Out of these, 18 full papers could be obtained in the analyzed platforms (15 retractions, 3 EoCs). Fourteen of the identified papers concerned dubious, ineffective or “alternative” treatments. The most common countries of origin were India and Egypt. The median number of Google Scholar post-retraction citations was 29,5 and the mean 42,9 (range: 0 - 128).

Conclusion: research-sharing platforms should implement mechanisms to prevent non reliable research to be made available in them.

Keywords: Retractions; Research-sharing; Science Communication.

RESUMEN

Introducción: este trabajo investigó la disponibilidad de artículos retractados/con Expresión de Preocupación Covid-19 en plataformas de intercambio de investigación.

Métodos: se la lista “Retraction Watch” (RW) de documentos sobre Covid-19 retractados/con Expresiones de Preocupación (EoC), se seleccionaron todos los artículos relativos al tratamiento con Covid-19. Tras su identificación, se buscaron los títulos de los artículos y los autores en las plataformas Research Gate y Academia (Academia.com). En caso de que se identificara un artículo retractado o de EdC disponible, se comprobó la presencia de una nota de advertencia (bien como nota adjunta o como advertencia directa en el artículo). A continuación, se identificaron las citas que recibieron estos artículos en la plataforma Google Scholar y se clasificaron como anteriores o posteriores a la fecha de retractación.

Resultados: en un primer momento, se seleccionaron un total de 44 artículos de la lista de RW. De ellos, se pudieron obtener 18 artículos completos en las plataformas analizadas (15 retractaciones, 3 EdC). Catorce de los artículos identificados se referían a tratamientos dudosos, ineficaces o “alternativos”. Los países

de origen más comunes fueron India y Egipto. La mediana del número de citas en Google Scholar tras la retractación fue de 29,5 y la media de 42,9 (rango: 0 - 128).

Conclusiones: las plataformas de intercambio de investigación deberían implementar mecanismos para evitar que investigaciones no fiables estén disponibles en ellas.

Palabras clave: Retractaciones; Investigación compartida; Comunicación científica.

INTRODUCTION

The retraction of a scientific paper is a serious event, implying the withdrawal of a study from the valid scientific record.⁽¹⁾ Occasionally, retractions take place for more trivial reasons, such as the mistaken duplicate publication of a paper by a journal, but, more frequently, some sort of misconduct is involved.⁽²⁾ Regardless of its reasons, though, after a retraction a study is void and cannot be used by other authors anymore. Therefore, it cannot be cited.

On the other hand, it is known that retracted papers are still cited post retraction.^(3,4,5,6) For example, a paper that used falsified data to claim benefits for Omega 3 fatty acids was still cited eleven years after its retraction;⁽⁷⁾ and according to a Retraction Watch scoreboard, the record among such papers apparently belongs to a paper with more than 1200 post-retraction cites over a period of 15 years.⁽⁸⁾

Questions about unreliable studies, their retractions and citations became especially significant during the pandemic, since the urgency of the problems during this period led to an influx of low-quality, hastily peer reviewed or even faked papers.^(9,10,11,12) Some of these papers even went on to have traction on defining health policy and clinical guidelines.^(13,14)

However, the question remains of why are these papers cited. A possibility is that they still can be found over the Internet, unmarked, in databases or research-sharing platforms. In order to investigate this possibility, the availability of retracted, non-identified papers was assessed in two well-known such platforms: ResearchGate and Academia. These platforms allow researchers to share pre-prints, preliminary accepted and similar research papers directly among colleagues.⁽¹⁵⁾ The first has about 20 million members from 190 countries⁽¹⁶⁾ and is sometimes dubbed “the Facebook for scientists”;⁽¹⁷⁾ while the second claims to have more than 230 million registered users.⁽¹⁸⁾ These are the two most prominent research-sharing platforms currently used by researchers.

Therefore, the aim of this paper is to identify the availability of retracted papers on internet platforms, and also to characterize their impact in terms of (pre and post retraction) citations. Covid-19 retracted papers were chosen for the analysis, due to their obvious relevance for rapid, urgent policy decision-making and their clinical impact.

METHODS

From the “Retraction Watch” (RW) list of Covid-19 retracted papers⁽¹⁹⁾ all articles pertaining to Covid-19 treatment (e.g. vaccines, drugs and other therapies) were selected and checked for their availability in two research-sharing Internet platforms. “Expressions of concern” (EoC) were included, but commentaries, theoretical, demographic and “sociological” papers were excluded from the study. Meta-analysis and systematic reviews of treatment and interventions were also included. Search was ended on October 30, 2023. Only full papers were considered, and papers withdrawn from online repositories and subsequently not published were also included.

After their identification in the RW list, paper titles and authors were searched on the mentioned platforms Research Gate and Academia. In case a retracted or EoC paper was identified as available, the presence of a warning note was ascertained, either as an attached note, as a direct warning on the paper or as a warning on the site. If the warning note was not present (that is, the paper was “retracted/with EoC and available”) the following information was collected: retraction date; Journal of publication and Country originating the research (byline-listed country of first author). Availability in one of the two searched databases was sufficient for categorizing a paper as “retracted/with EoC and available”.

All search and data collection were carried out independently by two researchers, who afterwards compared and validated their findings. Discordances were resolved by consensus. Identified papers were also manually double-checked to guarantee that the available version was the same that was originally present in the RW list.

Finally, the citations that the retracted papers received were identified on the Google Scholar platform, and classified as prior - posterior to retraction/EoC date. When a paper first received an EoC and was later retracted, the EoC date was used to this end. Citations which could not have their exact date ascertained were separately counted.

RESULTS

Table 1 presents the identified papers, their journals of publication and associated pre/post retraction citations. At first, a total of 44 papers were selected from the RW list. Out of these, 18 could be obtained in the mentioned platforms (15 retractions, 3 EoCs). Except for papers N04 and N06 (Academia) all papers could be found on Research Gate. Two of the papers (N13 & N14) had been retracted (“withdrawn”) from pre-print repositories. One more paper was available as “abstract only” and was not included in the analysis.

Of note, 14 of the identified papers concerned dubious, ineffective or “alternative” treatments, such as Traditional Chinese Medicine, Ayurveda (two papers), Vitamin D (two) and Ivermectin (four papers).

 Retractions took place between October 2020 and November 2021, and their citations occurred during the whole timeframe of the study, i.e., until August 2023. The total number of identified post-retraction citations was 773, with median 29,5, mean 42,9, minimum of zero and maximum of 128 (a paper on the efficacy of Vitamin D for Covid treatment). In addition, 66 other citations could not have their precise (pre x post retraction) citation date ascertained; and 44 citations occurred within 30 days of retraction. The countries with the largest number of retracted papers were India and Egypt (four each).

Table 1. Retracted/EoC Covid-19 papers (RW list, treatment and interventions only) available on the Research Gate and Academia platforms without warning notes, search ended August 30 2023.
Retracted/EoC paper	Journal	Retraction/EoC date	Country	Citations pre	Citations post	N.D.
N01.Vitamin D sufficiency, a serum 25-hydroxyvitamin D at least 30 ng/mL reduced risk for adverse… [EoC]	PLOS One	Oct 14/22	Iran	277	47	14
N02.Ivermectin for Prevention and Treatment of COVID-19 Infection: A Systematic Review, Meta-Analysis… [EoC]	Am J Therapeutics	Feb 07/22	UK	128	128	18
N03.Use of ivermectin in the treatment of Covid-19: A pilot trial	Toxicology Reports	May 02/22	Brazil	40	14	1
N04.Utilizing of (Zinc Oxide Nano-Spray) for Disinfection against “SARS-CoV-2” and Testing…	Coatings	Mar 29/21	S Arabia	1	82	0
N05. Review of the Emerging Evidence Demonstrating the Efficacy of Ivermectin in the Prophylaxis… [EoC]	Am J Therapeutics	Feb 07/22	USA	105	86	10
N06. Remdesivir Efficacy in COVID-19 Treatment: A Randomized Controlled Trial	Am J Trop Med Hyg	Sep 02/22	Egypt	21	34	3
N07. Recovering with nature: A review of ecotherapy and implications for the COVID-19 pandemic	Frontiers Pub Health	Jan 04/23	India	65	25	0
N08. Proxalutamide Reduces the Rate of Hospitalization for COVID-19 Male Outpatients…	Frontiers Medicine	Jun 08/22	Brazil	34	20	2
N09. Potential of antiviral peptide-based SARS-CoV-2 inactivators to combat COVID-19	PLOS One	Aug 31/22	India	2	0	0
N10. Meta-analysis of Randomized Trials of Ivermectin to Treat SARS-CoV-2 Infection (accepted version)	OFID-PMC	Aug 09/21	UK	23	64	4
N11. Hydroxychloroquine in the Treatment of COVID-19: A Multicenter Randomized Controlled Study	Am J Trop Med Hyg	Oct 11/21	Egypt	93	61	8
N12. Efficacy of favipiravir in COVID-19 treatment: a multi-center randomized study	Arch Virol	Nov 22/21	Egypt	45	105	5
N13. Efficacy and Safety of Ivermectin for Treatment and prophylaxis… (withdrawn)	ResearchSquare	Jul 14/21	Egypt	35	87	1
N14. Efficacy and Safety of Ayurveda Intervention (AYUSH 64) as add-on therapy (withdrawn)	medRxiv	Sep 27/21	India	0	6	0
N15. Efficacy and safety of acupuncture therapy for asymptomatic infection of COVID-19: A protocol…	Medicine	Mar 12/21	China	1	3	0
N16. Can Traditional Chinese Medicine provide insights into controlling the COVID-19 pandemic…	Sci Total Environ	Nov 05/20	USA	0	3	0
N17. Ayurvedic management of moderate COVID-19 infection: A case report	AyuCaRe	Jan 18/21	India	0	2	0
N18. Antiviral treatment could not provide clinical benefit in management of mild COVID-19…	J Infec Public Health	Jul 07/22	Thailand	4	6	0
EoC: Expression of Concern; otherwise retracted. Country: country of first author. ND: pre x post date of citation could not be established. Full titles available at the RW list^(19)

DISCUSSION

Retractions indicate that a breach of reliability occurred in a research project, nullifying its results, and thus the citation of a retracted paper invalidates their very purpose. This study investigated the presence of retracted papers on research platforms and their post-retraction citation impact, finding a high citation rate. Since these papers have no warning notes, it is possible that they are read and inadvertently circulated among researchers, being then included as references; but another possibility is that such papers enter the “reference stock” of researchers  prior to retraction, thus evading detection⁽²⁰⁾. Also, some retracted papers may be inappropriately included as references without actually being read, and, still, some papers may receive citations precisely because they were retracted, although this last case is known to “…almost never happen”.⁽²¹⁾

In the present study, a large proportion (more than 40 %) of the investigated papers could be retrieved without any retraction identification. As noted, most identified papers concerned dubious, ineffective or “alternative” treatments. One of these was the interesting paper with the colorful title Can Traditional Chinese Medicine provide insights into controlling the COVID-19 pandemic: Serpentinization-induced lithospheric long-wavelength magnetic anomalies in Proterozoic bedrocks in a weakened geomagnetic field mediate the aberrant transformation of biogenic molecules in COVID-19 via magnetic catalysis, which claimed Jade amulets could prevent Covid infection.⁽²²⁾ This paper, fortunately, received zero post-retraction citations.

Post-retraction citations continue to contaminate the scientific literature with spurious results, and this effect may bias and affect meta-analysis or other reviews, with serious implications for clinical/policy definition practice.⁽²⁰⁾ For instance, a 2021 meta-analysis on the effects of Ivermectin for the treatment of Covid included a study (N13, Table 1) which “…was circulated widely by politicians and others promoting the use of the drug”.⁽²³⁾  This study was later found to have used fraudulent data, leading to their (the study and the meta-analysis) retraction. Study N13, however, still can be found over the Net, and similarly, the retracted meta-analysis (N10, Table 1) can also still be found in its “paper accepted” version.

The post-retraction citation rate found in the present study was very high. For example, an article from 1990 found this rate to be 8.9 citations/retracted article⁽²⁴⁾ and a more recent study, dealing with  post-retraction citations of systematic reviews, found that this average was 4.5 per retracted article over a span of six years.⁽²⁵⁾ In the present study, retracted papers had an average citation rate almost ten times higher - in less than two years. However, it should be noted that despite their continued presence over the Net, other factors may contribute to this high rate, such as the pandemic exceptional “sense of urgency”, the attention received by those articles and the high coverage of the search engine that was used. The most pre/post-cited study (324 total) concerned the efficacy of Vitamin D for Covid; and a study on Ivermectin and Covid was the most post-cited, with 128 citations.

As mentioned, Academia alone boasts more than 230 million registered users and close to 50 million uploaded papers. If these databases happen to harbor invalid research, the potential for its widespread dissemination is, therefore, large. However, regardless of the role of the platform it can also be argued that the author of a retracted article has a responsibility on impeding the paper from circulating trough “alternative” Internet channels.

CONCLUSION

Despite its limited scope, the present study draws attention to one of the possible reasons of why retracted material is still cited in the scientific literature, and sends an important warning to research-sharing platforms. Obviously, researchers should try to be more aware about the possibility of retractions on their reference material, but more “automatic” solutions seem also to be necessary. For instance, the Zotero reference management software automatically warns researchers if papers in their libraries have already been retracted,⁽²⁶⁾ Similar measures should be taken by research-sharing platforms, which have a responsibility on stopping the dissemination of invalid research. Thus, the availability of retracted material seems widespread, and it is suggested that both researchers and Internet platforms should be more proactive in taking measures in order to curb their use as scientific references.

REFERENCES

1. Barbour V, Kleinert S, Wager E, Yentis S. Guidelines for retracting articles. Committee on Publication Ethics; 2009. https://doi.org/10.24318/cope.2019.1.4.

2. Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications. Proc Natl Acad Sci U S A 2012;109:17028–33. https://doi.org/10.1073/pnas.1212247109.

3. Meyerowitz-Katz G, Sekhar P, Besançon L, Turner T, McDonald S. The Citation of Retracted COVID-19 Papers is Common and Rarely Critical. Epidemiology; 2022. https://doi.org/10.1101/2022.06.30.22277084.

4. Santos-d’Amorim K, Ribeiro de Melo R, Nonato Macedo dos Santos R. Retractions and post-retraction citations in the COVID-19 infodemic: is Academia spreading misinformation? Liinc Rev 2021;17:e5593. https://doi.org/10.18617/liinc.v17i1.5593.

5. Bolland MJ, Grey A, Avenell A. Citation of retracted publications: A challenging problem. Accountability in Research 2022;29:18–25. https://doi.org/10.1080/08989621.2021.1886933.

6. Piller C. Disgraced COVID-19 studies are still routinely cited. Science 2021;371:331–2. https://doi.org/10.1126/science.371.6527.331.

7. Schneider J, Ye D, Hill AM, Whitehorn AS. Continued post-retraction citation of a fraudulent clinical trial report, 11 years after it was retracted for falsifying data. Scientometrics 2020;125:2877–913. https://doi.org/10.1007/s11192-020-03631-1.

8. Retraction Watch. Top 10 most highly cited retracted papers. Retraction Watch 2015. https://retractionwatch.com/the-retraction-watch-leaderboard/top-10-most-highly-cited-retracted-papers/ (accessed August 28, 2023).

9. Lawson DO, Wang MK, Kim K, Eikelboom R, Rodrigues M, Trapsa D, et al. Lessons from the COVID-19 pandemic and recent developments on the communication of clinical trials, publishing practices, and research integrity: in conversation with Dr. David Moher. Trials 2022;23:671. https://doi.org/10.1186/s13063-022-06624-y.

10. Chirico F, Teixeira da Silva JA, Magnavita N. “Questionable” peer review in the publishing pandemic during the time of COVID-19: implications for policy makers and stakeholders. Croat Med J 2020;61:300–1. https://doi.org/10.3325/cmj.2020.61.300.

11. Jung RG, Di Santo P, Clifford C, Prosperi-Porta G, Skanes S, Hung A, et al. Methodological quality of COVID-19 clinical research. Nat Commun 2021;12:943. https://doi.org/10.1038/s41467-021-21220-5.

12. Maisonneuve H. COVID-19 as a source of poor publications. Joint Bone Spine 2022;89:105427. https://doi.org/10.1016/j.jbspin.2022.105427.

13. Teixeira da Silva JA. Silently withdrawn or retracted preprints related to Covid-19 are a scholarly threat and a potential public health risk: theoretical arguments and suggested recommendations. OIR 2021;45:751–7. https://doi.org/10.1108/OIR-08-2020-0371.

14. Almeida RMVR. How did the scientific publication system respond to the Covid-19 pandemic? Iberoamerican Journal of Science Measurement and Communication 2022;2. https://doi.org/10.47909/ijsmc.160.

15. Manca S. ResearchGate and Academia.edu as networked socio-technical systems for scholarly communication: a literature review. Research in Learning Technology 2018;26. https://doi.org/10.25304/rlt.v26.2008.

16. ResearchGate. About Us | ResearchGate. ResearchGate 2023. https://www.researchgate.net/about (accessed September 18, 2023).

17. Kintisch E. Is ResearchGate Facebook for science? 2021. https://doi.org/10.1126/science.caredit.a1400214.

18. Academia. Academia.edu | About. AcademiaEdu 2023. https://www.academia.edu/about

19. Retraction Watch. Retracted coronavirus (COVID-19) papers. Retraction Watch 2023. https://retractionwatch.com/retracted-coronavirus-covid-19-papers

20. Teixeira da Silva JA, Bornemann-Cimenti H. Why do some retracted papers continue to be cited? Scientometrics 2017;110:365–70. https://doi.org/10.1007/s11192-016-2178-9.

21. Unger K, Couzin J. Even Retracted Papers Endure. Science 2006;312:40–1. https://doi.org/10.1126/science.312.5770.40.

22. Oransky I. Amulets may prevent COVID-19, says a paper in Elsevier journal. (They don’t.) – Retraction Watch 2020. https://retractionwatch.com/2020/10/29/amulets-may-prevent-covid-19-says-a-paper-in-elsevier-journal-they-dont/

23. Oransky I. Ivermectin meta-analysis to be retracted, revised, say authors. Retraction Watch 2021. https://retractionwatch.com/2021/08/10/ivermectin-meta-analysis-to-be-retracted-revised-say-authors/

24. Pfeifer MP. The Continued Use of Retracted, Invalid Scientific Literature. JAMA 1990;263:1420. https://doi.org/10.1001/jama.1990.03440100140020.

25. Wang Z, Shi Q, Zhou Q, Zhao S, Hou R, Lu S, et al. Retracted systematic reviews continued to be frequently cited: a citation analysis. Journal of Clinical Epidemiology 2022;149:137–45. https://doi.org/10.1016/j.jclinepi.2022.05.013.

26. Stillman D. Zotero Blog » Blog Archive » Retracted item notifications with Retraction Watch integration 2019. https://www.zotero.org/blog/retracted-item-notifications.

FINANCING

This research was funded by the Brazilian Ministry of Education – CAPES Proex (code 001  and by the Brazilian National Council for Scientific Research (CNPq)

CONFLICT OF INTEREST

The authors declare no conflicts of interest

AUTHORSHIP CONTRIBUTION

Conceptualization: Renan MVR Almeida, Aldo José Fontes-Pereira.

Data curation: Renan MVR Almeida, Aldo José Fontes-Pereira.

Formal analysis: Renan MVR Almeida, Aldo José Fontes-Pereira.

Acquisition of funds: Renan MVR Almeida, Aldo José Fontes-Pereira.

Research: Renan MVR Almeida, Aldo José Fontes-Pereira.

Methodology: Renan MVR Almeida, Aldo José Fontes-Pereira.

Project management: Renan MVR Almeida, Aldo José Fontes-Pereira.

Resources: Renan MVR Almeida, Aldo José Fontes-Pereira.

Drafting - original draft: Renan MVR Almeida, Aldo José Fontes-Pereira.

Writing - proofreading and editing: Renan MVR Almeida, Aldo José Fontes-Pereira.