Scientific journals and research papers are evaluated by a metric known as their “impact factor,” which is based on the number of times a particular paper is cited by other papers. However, a new study from MIT and other agencies shows that this measure does not accurately capture the impact of medical papers on health outcomes for all patients, especially in low- or middle-income countries.
To better understand the impact of papers on health, metrics must take into account the demographics of the researchers who conducted the study and the patients who participated in it, said the research team. To that end, they’ve developed a metric they call the “factor of diversity.”
The new metric incorporates several factors, including the diversity of paper authors (in terms of gender and geographic location), the diversity of patients studied, and how interdisciplinary the research team is. In a new study, researchers evaluated more than 100,000 medical papers published in the last 20 years and found that most did not fare well on this metric.
“Medical knowledge systems are controlled by a group of academics who are not very inclusive, and not diverse at all,” said Leo Anthony Celi, senior research scientist at MIT’s Institute of Medical Engineering and Science, a physician at Beth Israel Deaconess Medical Center, associate professor at Harvard Medical school, and one of the authors of the paper.
The researchers hope their new study will generate more discussion about how to evaluate medical papers and ensure they contribute to positive health outcomes for diverse populations, not just the groups that have traditionally led and been the subject of medical studies.
Jack Gallifant, a physician at Imperial College London NHS Trust, led the new study, which appears today in PLOS Global Public Health. The authors also include researchers from institutions around the world, including the Mbarara University of Science and Technology in Uganda, the National Polytechnic Institute in Mexico, the University of the Philippines in Manila, the University of the Witwatersrand in South Africa, Handong Global University in South Korea, and the King Hussein Cancer Center in Jordan, and represents the fields of public health, pharmacy, medicine, computer science, engineering, and social sciences.
“Blind spots” in medical knowledge
Celi and his colleagues set out to develop the new index in hopes of finding a way to document and combat the lack of diversity among authors in leading medical publications. Most of these writers came from wealthy countries including the United States, and they were disproportionately white and male.
“The way medical knowledge systems are designed has led to the fact that 80 percent of publications come from 20 percent of countries, and then the guidelines that are disseminated to treat diabetes, treat hypertension, treat cancer, are informed through trials and observations. study in 20 percent of the country,” he said.
To measure the extent of this problem, the researchers created an index based on several factors. One factor is the diversity of study authors, including whether the authors are in high-income or low- and middle-income countries. The researchers also used an algorithm that categorizes authors as male or female. This is important not only for promoting inclusion, but also because a lack of diversity among medical study authors can lead to failure to fully consider how certain diseases affect different groups of people, said Celi.
“What happens when all the writers involved in a project are the same is that they will have the same weaknesses. They will all see the problem from the same angle,” he said. “What we need is cognitive diversity, which is based on life experiences.”
Another factor the researchers included was the diversity of the study authors’ departmental affiliations. Under this metric, papers are scored higher when they include authors from multiple disciplines. For example, a medical study that includes doctors, nurses, and engineers will be rated higher than a study that includes authors from either field.
The final metric the researchers propose is based on the diversity of traits of patients included in a particular study, including gender, race-ethnicity, language, age, and geography. However, because many of the papers analyzed in this study were not publicly accessible, the researchers were unable to obtain this information for many papers and so did not include it in their final analysis.
Celi stressed that the metrics chosen for this study should be considered only as a starting point for measuring the impact of health outcomes in a more equitable way.
“What we want is to start a public dialogue on this topic, and we also want the community to contribute and recommend what metrics should be,” he said.
Track diversity
Once the researchers identified the criteria they wanted to analyze, they used the OpenAlex database, which aggregates information from many other academic article databases, to extract author metadata from about 130,000 medical papers, from more than 7,500 journals, published between 2000 and August 2022.
They found, as they expected, that women writers were consistently underrepresented. Across all journals, male authors outnumber female authors, but the representation of female authors continues to increase. As of 2021, the ratio of female to male authors per publication is between 0.30 and 0.42 in major journals.
The researchers also found a lack of geographic diversity. For papers published in 2021, there are more than 5 million authors from high-income countries, 1.5 million from upper-middle-income countries, around 470,000 from lower-middle-income countries, and more than 27,000 from low-income countries.
“Not surprising, but we want to quantify it. And we want to be able to track this over time, so we can know if we’re making progress,” said Celi.
The researchers found that overall, open access journals scored better on the diversity index than non-open access journals. Among the top 25 open access and non-open access journals (ranked by impact factor), open access journals consistently have a higher proportion of female authors and writers from low- and middle-income countries.
Many entities, including academic institutions, journals and funding agencies can and should play a role in increasing the diversity of authors publishing in the medical field, said Celi. Several funding agencies, including the National Institutes of Health, have created initiatives, such as the Artificial Intelligence/Machine Learning Consortium to Advance Health Equity and Researcher Diversity (AIM-AHEAD), which requires research teams to be led by principal investigators from the institution. that are poorly funded, or that have historically served minority communities.
“Funders, universities, journals and media too, to some extent, are all responsible for the problems we have seen, and each of them must innovate in operationalizing their mission and vision,” Celi said.
#evaluate #impact #medical #research
