Peer review

Reader question:

What does it mean when they say someone’s work is “outstanding by peer review”? Peer review?

My comments:

Peer review literally means review by one’s peers.

It’s the opinion of one’s fellow practitioners, in other words.

Therefore, if your work is considered outstanding by peer review, it means people of your profession think your work is very good, outstanding actually meaning better then most.

Peers, you see, are people who share the same qualities as your own. A review, of course, is an examination and assessment on something, usually an academic piece of work, on whether it is any good.

Peer review is the expert opinion of the people who share your profession or interest. I say expert opinion because their opinion means something because as fellow professionals, they know what you’re doing and they know what they’re talking about – or supposedly so.

Opinions of people from a different trade, on the other hand, may not count as much. If you are a biochemist, for example, then other biochemists from your own organization and other organizations, such as universities or colleges are your peers. Linguists and mathematicians from your organization and others on the other hand are not considered your peers. Hence, their view may not mean much because they’re not supposed to know a lot about your field of work.

Anyways, the long and short of it is, if your work is considered outstanding by your peers, via peer review, if means a lot. It means your colleagues and fellow practitioners all approve of your effort.

All right?

All right, let’s read a few examples of “peer review” to hammer the point firmly home:

1. As pop culture would tell you, scientists are old white guys with crazy hair. While that perspective is heavily biased (my hair is crazy, but not white), it isn’t totally unfounded.

The people who make science share their knowledge through academic journals, which traditionally take their contents very seriously. The journals accept science by peer review, meaning that the most prestigious, whitest haired, top-of-the-line scientists make sure the contents of the journal are up to snuff. And you can only read the journal if you, as part of the scientific elite, choose to pay for access.

But this model is outdated…or so would say the open access journals, which sprung to popularity about a decade ago. Open access journals claim their goal is to remove legal, financial, and technical barriers between people and their science. The only thing keeping people from reading the contents should be access to the internet itself.

The problem is, open access journals don’t have quite as spiffy a reputation as traditional journals. And this was what inspired the recent efforts of John Bohannon.

John wrote a spoof paper and sent it to hundreds of open access publishers. 157 published it. And then Science published him.

“Any reviewer with more than a high-school knowledge of chemistry and the ability to understand a basic data plot should have spotted the paper’s short-comings immediately,” John writes. “Its experiments are so hopelessly flawed that the results are meaningless.”

John submitted a paper that proclaimed a new wonder drug. He set up the paper with a simple formula: “Molecule X from lichen species Y inhibits the growth of cancer cell Z.” He substituted each variable with molecules, lichens, and cancer cell lines to create hundreds of papers. Each was unique enough to not attract attention, but the structure was similar enough to be used as a constant in John’s investigation. He submitted the paper using false names and institutions that he generated randomly from databases of common African names, words in Swahili, and African capital cities.

He included the same flaws in each paper – data that showed the opposite of his conclusions, an obvious lapse in the methods, and a control group that didn’t receive one of the constant level of radiation as the others.

Over 150 open access journals accepted the fake paper. John writes that over 250 of his papers went through an editing process, but that 60% showed no sign of peer review.

The final verdict is that open access journals have a long way to go. There may in fact be some merits to the peer-review system…even if you have to pay to get in.

- “Who’s Afraid of Peer Review?” Shakes Up Scientific Community, by Poncie Rutsch , BUNewsService, October 8, 2013.

2. In the United States, some basic biomedical and clinical research is funded by private companies and investors. But much of it is funded by the NIH with government money. That raises the question of whether that money is being spent on research that is worthwhile for the population as a whole, whether it finds cures and uncovers the causes of disease. “Is the NIH still funding the path-breaking research that is likely to be influential?” asks Leila Agha, an economist at Boston University and a coauthor on the study. “In high-level applications, can [peer review] distinguish the best research?”

For all that science is a data-based enterprise, scientists don’t have the data on whether peer review really does end up funding the best research. “If you were a congressperson or a taxpayer, you might say ‘show me some data that peer review is good at picking things that turn out to be important,’” says Jeremy Berg, a biochemist at the University of Pittsburgh. “But until this study was done, the answer was ‘we believe it but we can’t prove it.’ As scientists, that’s kind of embarrassing.”

When a scientist wants to get NIH funding for a study, she writes up a grant proposal that reports results from preliminary studies, gives goals for the project, outlines the future experiments and estimates the time and resources they will require. The researcher submits her grant, and it’s assigned to a study section of 20 to 30 researchers who work in disciplines closely related to that of the grant proposal.

Within the study section, the grant will be assigned to three reviewers, two of whom provide detailed comments, and a reader, who provides additional comments. The reviewers will give the grant an overall score based on five criteria: Significance, scientific approach, potential innovation, the proposing scientist’s skills and whether the researcher’s university has the resources to support the work. About 40 to 50 percent of grants will be “triaged” at this stage. The rest go to the study section as a whole. After about 10 to 15 minutes of discussion, the grants receive final rankings by priority, with the lowest scores being the best and most likely to be funded.

In recent times, this means that most grants — even those that score well — will not get funding. NIH has a current annual budget of around $30 billion, but that number has not kept pace with the increasing number of scientists applying for research money. In 2017, only about 16 percent of new applications were funded. This makes applying for grants more competitive, and thus makes it even more important that peer review is selecting the research with the highest potential payoff.

Agha and Danielle Li, an economist at Harvard University, wanted to determine whether peer review could successfully predict the influence of the subsequent research. They examined the funding scores for a total of 137,215 peer-reviewed grants funded between 1980 and 2008. For each of the grants, they hunted down how many published scientific studies or patents the grant yielded within five years of the grant’s success. Li and Agha also looked at how many citations the scientific studies for each grant had accrued.

As they assessed the scores and the grants’ success rates, the researchers tried to factor out the scientists’ institutions, previous funding, previous work and field of study. The results showed that grants with higher scores did, in fact, tend to have more patents and more highly-cited publications associated with them. For each 10-point drop in score, a grant was 19 percent less likely to produce a high-impact publication and 14 percent less likely to produce a patent. The economists report their findings April 23 in Science.

“It’s good news. It’s suggesting that [grant reviewers] do on average have a clue,” says Lars Lefgren, an economist at Brigham Young University in Salt Lake City. “Some people complain that the NIH may be biased in terms of awarding grants to people with big names or established track records but who don’t have the most exciting or novel research. This study suggests those types of concerns may not happen on average.”

- A peer-reviewed study finds value in peer-reviewed research, ScienceNews.org, April 23, 2017.

3. Following the data that was revealed by Channel 2 News about the effects of pollution in the Haifa Bay area according to which 20% to 30% of the babies that were born in the area were born with a small head, the Israeli Ministry of Health responded today to these claims.

The Israeli Ministry of Health professionals examined the data that was gathered in health clinics across the country related to the size of babies heads over the last six months. The Israeli Ministry of Health clarified: “The data shows that there is no difference in the size of babies’ heads across the country. The Ministry of Health is continuing to process the data.”

“Related to the findings in the research that was done in the Haifa area, we stress that like all scientific research, the information is validated and its conclusions are implemented via peer review so that we can ensure the quality of the results,” the Israeli Ministry of Health added.

- Israeli Ministry of Health: “There is no difference in the size of babies’ heads”, JerusalemOnline.com, February 3, 2016.

About the author:

Zhang Xin is Trainer at chinadaily.com.cn. He has been with China Daily since 1988, when he graduated from Beijing Foreign Studies University. Write him at: zhangxin@chinadaily.com.cn, or raise a question for potential use in a future column.