What’s Peer Review and Why it Matters.

WELCOME MBA STUDENTS. IF YOU NEED A BREAK, IF YOU FEEL LIKE SCHOOL AND WORK ARE GETTING TO YOU, TAKE A WALK THROUGH MY BLOG. YOU WILL BE AMAZED,  PLEASED AND SHOCKED. WELCOME.

JD

Global warming update: In its 2007 report, the UN Intergovernmental Panel on Climate Change stated that 40% of the Amazonian rain forest could be gone in a short time because of climate change. The source cited is not peer-reviewed. Its authors are a public policy analyst, that is, an advocate, and a journalist at the Guardian of London. Neither is a scientist. The main thing is that they did not even try to get their piece published in a real, scholarly, and therefore peer–reviewed journal.

Reliance on sources that are not peer-reviewed is forbidden by the UN Panel’s own rules. The fact that the IGPCC violated its own rules does not imply an evil intent but carelessness or zealots’ quasi religious enthusiasm. ( I keep telling you that climate change is a religion) I ask myself: How long would I continue to patronize a car mechanic who told this level of untruths?

The story was in the Telegraph, a UK left-wing newspaper, on January 25^th. This came up after the 300+ mistake I talked about before about the time it will take for Himalayas glaciers to melt down. (It’s 300 years longer than announced by the Panel, according to the correction given by the Panel itself!)

You can find everything including linked to references in the

“ Watt’s Up With That” site, linked to this blog under “Climate change.”

In the English-speaking world, the one I know well, scholars make their careers by publishing in scholarly journals. Most scholars are also university professors but most university professors are not scholars.

In every discipline, such as Economics, or Physics, there are only two or three journals that matter a lot, and a multitude that matter a little or hardly at all. There is sometimes overlap between disciplines such as Sociology and Management ( my own disciplines). Scholars in one discipline frequently try to publish in journals of a different but related discipline.

All scholarly journals that matter are peer-reviewed. I think there is no exception. This is how peer review works. First the traffic directors of the process of peer review.

In my experience, journals’ chief editors are always well-known scholars who have themselves published uncommonly much in first-rate refereed journals. They may be controversial; their appointment may trigger unseemly rage but never because they are judged incompetent. Journal editors are normally not paid for their hard, time-consuming work but they often receive course relief and office space from their university. Editors get huge prestige and satisfaction from the job. There is much competition for the position of editor of any major scholarly journal. There is no formal way to achieve the position. (I got an offer once and I decided not to pursue it. Too much work for me.)

This is how editors of scholarly journals organize peer reviewing.

An individual or, more often, a team of scholars, works on a study and then, writes it up. In the disciplines I know, this process often takes three years. The final result is a 20 to 30-page paper.

Then, they choose a journal and they submit their paper for publication. Choosing a journal is difficult and delicate. The most prestigious journals may not be the best ones for the particular study depending on subject matter and originality of findings. Yet, publishing is a prestigious journal insures a readership and career advancement.

Upon receiving the submission, the editor give it a quick read. He rejects outright those submissions that are an obvious waste of time. That may be most submissions. For the remainder, he looks through his stable of reviewers for a few who might be competent to judge the particular submission. The reviewers are scholars who have themselves published in his journal. They know the standards and they are usually not eager to open the doors wide: A bad article in the journal in which I published mine six months ago makes mine look bad. Poor quality is contagious in the minds of readers. It’s like high-school: If I hang out with sluts, everyone will think I am a slut too.

The editor then sends the paper to two or, more often, three reviewers. The reviewers do not know the names of the authors of the paper they are reviewing. They are expected to excuse themselves if they recognize the authors or if there might be any potential conflict of interest. The reviewers don’t know each other either until after they have done their review. They are not told who their co-reviewers might be. This cuts down on the possibility of conspiracy for or against the paper they review.

The reviewers do not get paid for their work. They do it as a public service with the intent to act as gate-keepers. They are motivated to let what they think is important and believable get published. They are motivated to stop garbage and falsehoods, and even suspicious-looking findings.

The submitters, the paper’s authors, do not know who their reviewers are. They are never supposed to find out. This practice cuts down on any reviewer’s fear of retaliation. The reviewers are thus roughly “peers” of the submitters. The important point is that they are not superiors in some organizational hierarchy. They are all scholars with a doctorate playing the same game. It’s common for the submitters to be seniors to their reviewers.

This process, is known technically as “double-blind peer review.”

It’s “double” because the reviewers don’t know who the submitter is and he does not know who they are.

Reviewers are expected to critique everything about the paper, the quality of the study, the credibility of the findings, the technical worth of the methods used, the quality of the data. They are explicitly charged with paying attention to the rigorousness of the conclusions drawn from the results: “ We discovered that twenty cats out of a hundred are black, therefore, dogs are white,” would not pass muster. That’s climate science logic. I am sorry, that was a bitchy thing to say!

Reviewers even make comments on the quality of the writing. As a reviewer, I rejected several papers (submissions) outright because they were too difficult to read or because I suspected the authors were not clear in their own minds about what they were discussing.

In practice, it’s not rare for reviewers to specialize in one aspect of a paper. One will be a methods judge, another, a data person (that was me), another a logic expert. That’s one reason to have two or three reviewers rather than one. The other reason is an honest attempt to achieve fairness for the submitters.

There are more exotic specializations: For several years, several social science journals would send me papers to review that they suspected were written – in English – by French speakers. I did what needed to be done. No vacillation; no compassion!

After a delay that may be as long as three or even six months, the editor receives and studies the contributions of all the reviewers.

I surmise that often the several reviews agree substantially well. Personally, every time I have seen the critiques of co-reviewers, of people who had done what I had done in parallel, I was pleased. I never saw incompetent or silly critiques. When there is convergence, the editor has little trouble formulating a response to the authors. When there is no convergence between the reviewers, the editor calls on a fourth, and even on a fifth reviewer. It does so until a clear judgment emerges.

It happens almost never that the response is, “Done.” In my whole career, I sent back a submission I had reviewed to the editor only once saying, “This is very good. Publish it.” My co-reviewers did not agree.

In prestigious, first-rate journals, most submissions simply get rejected. The only good outcome that is likely is called, “Revise and resubmit.” The words speak for themselves. The editor is saying” If you can satisfy reviewer 1 on this point, and reviewer 2 on that point, and reviewer 3 on these other points, I will look at your re-submission with favor .” Again, that’s just about the best you can expect. This response implies no guarantee of future publication.

Remember that your career depends on the outcome. It’s common for five years to elapse between the beginning of a study and the publication of the paper. I begun a study in 1977 that was finally published in a good journal in 2000, after rejections from five good journals. I did stop working on it for about fifteen years though. Yes, that was an extreme case.

What I have just described (thank you for your patience), is a fiercely competitive gate-keeping system. It may be one of the most competitive systems this side of the Olympics. It’s hated by many academics for a variety of reasons, most bad, some good. One of the good reasons to hate it is that it keeps out many good contributions. Many innovative scholars become bitter because their good work is rejected for bad reasons. The system is also not immune to fads, academic fads that is, that you would not recognize as fads unless you were in the discipline. The system also throws a tarpaulin over forbidden areas of inquiry, especially in the social sciences. So, right now you cannot publish anything on the relationship between race and intelligence, no matter how good the research and the writing. That’s a fact, at least in the US. It might be slightly different in other English-speaking countries but my intuition says not.

Scholarly journals may sometimes be captured by activist or zealot minorities. It would take too long to go into this rare phenomenon. Ask me in a comment if you want to know more. Finally, practically all scholarly journals have a bias against what’s known as “negative findings.” It’s an interesting issue. Again, ask me in a comment if you want to know more.

The peer review system is mostly suppressive. In this respect, it works extraordinarily well. It rarely allows garbage to pass.  It’s absolutely harsh on opinions in general. It’s pitiless toward opinions not supported by strong findings. In this respect, peer-reviewed journals are unlike any other source of information on the planet.

Here is another contribution to truth of double-blind peer-reviewed publication: Having an article published in one the prestigious journals is like asking for contradiction. If you present strong evidence that dogs chase cats because cats look to them like food, within a short time, there will be someone submitting research showing that dogs don’t chase cats, that when dogs chase cats, they do it because the cats provoked them, or that cats chase dogs. Editors like to publish replications of published studies. They especially like contradictory replications. The best career ladder for a young upstart researcher is to demonstrate that some older, well-established scholar  (like me) is full of it. If you are dreadfully wrong in a double-blind peer-reviewed journal, chances are you are not going to be allowed to stay wrong for a long time.

Double-blind peer review is not perfect but I think it’s the best truth-seeking device we have for the moment. Everything else is inferior.

That’s why it matters whether any policy-making or policy-influencing body such as the UN Intergovernmental Panel on Climate Change relies on peer-reviewed studies or on someone’s agenda, religious belief, or pseudo-scientific nightmare.

Added Jan 2016:

Three of my recent books are available on Amazon Kindle and on some other reading devices. None is a scholarly book. Two are in English and one in French. The first is entitled: I Used to Be French: an Immature Autobiography (2014).This one also exists in paperback. You can obtain the paperback from me through this clever email address: iusedtobefrench@gmail.com. It’s listed under my name: Jacques Delacroix. My second book in English is only in electronic form. It’s entitled: Indecent Stories for Decent Women. (2015) It’s under the pen name: “Jean René Adolph.” Reflecting on its title will suggest why I am not using my real name in this one. http://www.amazon.com/dp/B018ZYR9DS
I have a third book, written in French, that is completely different from the others: Les Pumas de grande banlieue: histoires d’émigration. (2014) This one is also only in electronic form. It’s under my real name: Jacques Delacroix. I pay attention to feedback. Thank you. I am open to sponsoring and commercial publication of any of the three.