Unethical behaviour in science is a long and dishonourable story. And it is continually being updated. As recently as 2006, the international scientific community was deeply shaken when the South Korean stem-cell researcher, Hwang Woo Suk, was found to have fabricated his research evidence, although his articles had been rigorously reviewed by and published in Science magazine.

Closer to home, ETH Zurich reacted firmly to the bleak reality of duplicity in science when in 2002 Bell Labs physicist, Jan Hendrik Schön¹, was found to have engaged in scientific fraud. The university reacted by instituting the ‘Procedure to address allegations of research misconduct at ETH Zurich²’ to combat falsification in research that, by extension, would also affect publishing.

Three researchers at ETH Zurich’s Professorship for Social Psychology and Research on Higher Education (D-GESS) have now addressed review procedures for publishing scientific papers. Drs Lutz Bornmann, Irina Nast and Professor Hans-Dieter Daniel, in an article to be published in Scientometrics³, called their analysis ‘Do editors and referees look for signs of scientific misconduct when reviewing manuscripts?’

The ETH Zurich investigation explored the rules that define acceptance standards for the publication of scientific manuscripts. The project evaluated 46 research studies published between 1967 and 2006 that examined editors’ and referees’ criteria for the assessment of manuscripts and their grounds for accepting or rejecting manuscripts. . The project revealed that scientific papers were reviewed according to nine main criteria which reviewers felt were most important. Each of those nine areas had up to six different underlying dimensions.

The ETH Zurich researchers primarily focused on ethics, the last of the nine criteria that were listed in descending order. ‘Relevance of contribution’ was number one, seemingly influenced by a manuscript’s perceived “importance, newness and originality”. Interestingly, none of the 46 studies defined falsification of information or fabrication of data as a specific area on which to focus. That absence, in itself, may answer the question of how reviewers could miss obvious cases of fraud, such as that of Hwang or Schön.

Despite this omission, the ETH Zurich researchers found repeated instances of ‘quality of research’ as one of the underlying themes of each of the top three main criteria. In opposition to criticism levelled at the peer review process in the wake of the more recent sensational incidents of scientific fraud, the researchers conclude that the importance of ‘quality of research’ in the assessment of a manuscript does indeed play an important role in the review process.

"The Peer Review is an important aspect in quality control"

ETH Life: Papers submitted for publication are subjected to review and assessment by editors and peer reviewers, whose ‘referee reports’ ultimately determine whether a manuscript will be published. The process leaves readers with the impression that a journal’s content is scientifically and ethically sound. Nevertheless, is this rather a case of ‘reader beware’?
Lutz Bornmann: Science is based on trust: scientists trust that published research results are real and not contrived. Independent of this is the scepticism that is an essential part of scientific work and that calls for work to be critically reviewed. Every scientist must critically examine not only his own work but that of his colleagues as well so that research with reliable results can continue and progress. Certainly no lawyer in the matter of peer reviews would claim that all research results are honestly obtained. In general, though, articles that have been subjected to peer review by a respected journal can be trusted more than articles that have not been reviewed.
The review process appears to have remained essentially unchanged since 1967, the year of some the oldest studies of your analysis. Is the review process as it stands outdated?
The traditional form of peer review as it is carried out in [most] journals remains a proven process, despite periodic cases of fraud. In the past few years, for example with the Open Access Initiative and technical opportunities enabled by the Internet, certain journals have instituted new forms of peer review. To name one, Atmospheric Chemistry and Physics has a two-stage assessment process that combines interactive discussion and the traditional method of review.
First, manuscripts go to the editor who quickly vets them (the ‘access review’). Then the papers are immediately posted to the journal’s online discussion forum where remarks from invited peer reviewers and interested members of the scientific community are published. The second step of the assessment process is the traditional peer review.
For a while Nature also experimented with a two-stage review process until they realised that the pool of scientists ready to take part in the revamped process was too small. The publisher found as well that remarks made in the forum were of little use. Nature recently ended this experimental phase.
Peer reviewers certainly know how tempting it can be for researchers to manipulate the results of an experiment: the rewards for success are manifold. Are peer reviewers turning a blind eye to a known weakness in the system?
LB: No, I don’t believe so. Instead, the cases of fraud so far uncovered (ex: Hwang) have led to efforts to address and abolish weaknesses in the peer review system. I can imagine that such cases serve to strengthen reviewers’ awareness of fraud, and may lead to uncovering more instances of it.
The traditional peer review process finds ‘theory’,’ design/conception’, and ‘discussion of results’ most important for judging the basic quality of research of a manuscript’. Did this surprise you?
Not at all. Our study’s findings reinforced my opinion that the peer review is an important aspect in science of the quality control process. On the other hand, I would certainly have had my doubts had ‘writing/presentation’, for example, presented itself as the most important area.