Gilbert Ling: The Peer Review System Suppresses Innovation And Progress

[burtlancast]

Here are some hard facts showing the scientific world itself has publicly condemned the peer review system.
Peer review system suppresses innovation and progress

Peer review system suppresses innovation and progress
That peer review system suppresses innovation and progress has been confirmed by survey and survey conducted by among other institutions and agencies, the National Institute of Health (NIH) itself.

Nationwide Poll of Physicists, Chemists and Biologists
In response to a request from Congressman Bob Casey of Texas dated 9/27/1973, a seven-member committee was formed, including myself. The purpose was to poll scientists' opinions concerning the peer review procedures as used by various government agencies. A questionnaire was written and approved by representatives of both the National Science Foundation (NSF) and the National Institutes of Health (NIH). In cooperation with the Industrial Research magazine, the committee mailed out questionnaires to 500 physicists, 500 chemists and 500 biologists. 175 physicists, 189 chemists and 343 biologists responded. Only one in eleven scientists believes that the peer review system, as it operates today, is fair and reasonable.

Poll of 3823 Scholars in Humanities on Peer Review
The American Council of Learned Societies conducted an opinion poll of 3823 scholars in seven disciplines: Classics, History, Linguistics, Literature, Philosophy, Political Science, and Sociology. The results were reviewed in the New York Times (9/30/1986). Three out of four respondents believe that peer review procedures were biased in favor of researchers or scholars on a prestigious campus or who used currently fashionable ideas that do not upset the establishments's elders. The majority also say that peer review process often overlooked pioneering voices, in favor of conservative opinions, sanctioned by the academic establishment or trendy views already approved by intellectual "in-groups".

Congressional Investigation of the Peer Review System of the National Science Foundation in 1975
Polls like those cited above show that increasingly more and more people in science and in other fields of study feel that the widely used peer review system is undermining the normal progress. Undoubtedly these dissatisfactions eventually reached the responsible section of the Congress, namely the Committee on Science and Technology. A congressional hearing on the peer review system as practiced at the National Science Foundation in the funding of scientific research took place in Washington, D.C. between 7/22 and 7/31/ 1975.

Testifying before the Committee, I presented my difficulties of getting continued support from NSF as new findings made it increasingly clear that the scientific view of the establishment was wrong. Thus in many ways, my experience supported what Representative John B. Conlan had commented in general terms (see linked page lp11). In the end beside my verbal presentation I also deposited a total of 63 pages of written documents in support of my view that the peer review system being practiced at the time at NSF does not permit the survival of scientists pursuing new and innovative research. Unfortunately as far as I know, this Congressional hearing did not make any significant impact on the way NSF distributes taxpayers' money to promote research beyond the publication of an 1158-page book by the Subcommittee.

National Institutes of Health's Own Two-Year Investigation of Peer Review
As a result of a two-day long seminar held in 1974, top NIH officials recognized the need to look more deeply into the peer review system. A 14 member Grant Peer Review Study Team (GPRST) was appointed early in 1975 in order "to examine in critical detail, the entire process of peer review and make, where necessary, recommendation for modifications or change". Among the specific subject spelt out for investigation was "the capability of the peer review system to accommodate a really new and unusual scientific idea..." (Grant Peer Review Report to Director of NIH Vol. 1, pp.3-4).

A most extensive investigation followed. Public hearings were held in Chicago, San Francisco and Bethesda( Maryland). And written comments were solicited from all categories of people who had direct contact with the system. After most of two years, a final report of three volumes was submitted to the Director of NIH, Dr. Donald Fredrickson on 12/29/1976.

In Volume II, Supplement B, Part 3, are the following testimonials cited:

"...strong bias toward applications ..that fit into framework of more or less standard ongoing research without breaking new ground or developing innovative new directions or research" (Division of Research Grant Study, Section Member) (p.a.-3-10).

"Novel ideas which challenge dogma never get funding priority..." (Institute IRG member) (p. G-3-10)

"General Comment-Major problems with the current system...is that innovation by long- term projects are essentially unfundable by the system" (DRG Study Section member) (p.a.-3-10).

"For the review of the idea which runs contrary to prevailing thought, the system freezes..." (Advisory Council Board member.)

These are some of the testimonials. Their messages are loud and clear. The GPRST recommended the development of an experimental program involving limited support of speculative, high risk unorthodox or innovative research proposals. The GPRST then continued: "Such a study might be part of a larger much needed effort to examine the processes of decision-making in allotting research support."(Document #2). This statement clearly shows that top-ranking NIH officials were very aware that something was not right in the allocation of research support.

Unbelievably, the then departing NIH director Donald Fredrickson struck down the recommendation and gave the explanation for this action to the effect that "everyone thinks his or her research to be innovative." This is simply not true. But no one argued with him. His decision was final---after more than two years of hearings and a three-volume document written on the subject.

The quotations cited above---one wonders if Fredrickson has read them--- show quite clearly that the word, innovative, was not intended to represent something new. Instead, it represents something that is both new and against traditional view or practice---as most of the big authoritative dictionaries define the word, innovative. Every scientist definitely does not write proposal against traditional views or practices.

But apparently none of the fourteen signatories for the report saw fit or were otherwise unable or unwilling to explain to Dr. Donaldson that he had misundestood the word, innovative as they had intended.

This incidence shows just how a single person occupying the right place could take away the hope of untold millions with a stroke of his pen. And few even noticed it.

Can anyone access www.gilbertling.org ?
I've never been able to, and had to fall back on the wayback machine from archive.org.

 

[burtlancast]

The peer review system

The Peer Review system

The peer review systems adopted for government-sponsored research funding can be roughly sorted out into two types. One type was first adopted by the Office of Naval Research (ONR) and the National Science Foundation (NSF); the other type was that introduced by the National Institutes of Health (NIH).

1.The ONR-NSF system
This system places greater responsibility on the individual program managers. A researcher seeking support writes a proposal describing what he/she plans to do and submits it along with the endorsement from the Institution where he/she is employed. The specific program manager covering that field reads the proposal and sends it out to different experts in the field and seeks their opinions. Based on the opinions gathered, the program manager makes the final decision to fund the proposal or to reject it. The good side of this system is that it has flexibility. When the program manager is dedicated and has the needed knowledge, courage and integrity, the system can work well.

The down side of the ONR-NSF system is that if the program manager does not have enough knowledge and self confidence, and/or cares little about the true objective of scientific research, he/she can do a great deal of damage to Science.

Thus in the words of Representative John B. Conlan from Arizona: " ( the system can turn into) an 'Old Boy's System' where program managers rely on trusted friends in the academic community to review their proposals. These friends recommend their friends as reviewers....It is an incestuous buddy system that frequently stifles new ideas and scientific breakthroughs, while carving up the multimillion dollar Federal research and education pie in a monopoly game of grantsmanship..." (from Testimonial on July 22, 1975, as a member of the Subcommittee on Science, Research and Technology of the Committee on Science and Technology, U.S. House of Representatives, 94th Congress, First Session. Quoted from "National Science Foundation Peer Review, Special Oversight Hearings., Publication No. 32, p. 5; U.S. Government Printing Office, Washington D.C.).

2.The NIH system.
NIH is a much larger conglomerate of semi-independent Institutions (e.g., National Cancer Institute, National Heart Institute). Each has its own research laboratories and researchers, constituting what is called "intramural activities". It is the "extramural activity" that oversees funding of research conducted by researchers outside NIH, often in the Universities. This funding is handled by a single NIH institution called the Division of Research Grants. Although there is a second level of peer review by a national advisory council made up by scientists and laymen, the major decision on the funding of research proposal is made by scientists sitting on the Initial Review Groups (IRG) also called Study Sessions. As of 1993, there were a total of 84 study sessions belonging to 16 NIH Institutes.

Research proposals, written by research grant applicants are distributed by the Divisioon of Research Grants to one of the Study Sessions for review. Of those proposals recommended for approval, each of the Study Section members gives (anonymously) a priority score ranging from 100 (highest) to 500( lowest) and the average determines funding or non-funding. Sometimes a difference of a few points determines "life" or "death" of the proposal. Even though each Study Section member (which usually number between 10 and 20 but may be more) gives a numerical score, only two of the members read any one proposal. For this reason, there is no chance for a proposal to be funded, unless those two readers absolutely wish the proposal to be funded; they are the proposals's only advocates. None of the other member really have the detailed knowledge of the proposal to challenge the opinions of the readers, not to mention that they must all cooperate with one another so that proposals read by other members of the Study Section can be accepted or rejected as recommended by their readers.

The power wielded by the two "readers" can be seen from an NIH-distributed document giving advise to grant applicants: "the author of a project proposal must learn all he can about those who will read his proposal and keep these readers in mind constantly as he writes." Since single copies of this advice could be obtained from the Division of Research Grants, NIH (see linked page lp11a), it represents what is considered acceptable and perhaps even desirable by the NIH authority.

There are two ways of looking at this advice. For the great majority of grant applicants working in an area of research where the foundation concepts have been firmly established and acceptable to all, this advice tells you about a system with which one might find fault but perhaps also live.

It is an altogether different story when the applicant works in an area of science where the foundation is far from being unequivocally established, as it is in the case of cell physiology. It is in this situation that the absolute, unchallengeable power given to the "readers" of the Study Sections, with their long four-year tenure, their high visibility, their privileged tradition of recommending their own successors, as well as their own needs for the same NIH money for their own research now being disputed, that the working of the system takes on a significance of truly alarming nature. It is here that the NIH peer review could fall into the same pitfall of generating an "incestuous buddy system" that "stifles new ideas and scientific breakthroughs" as Representative John B. Conlan had described in regard to the NSF system.

 

[Blossom]

Can anyone access www.gilbertling.org ?

Yes.

 

[Hugh Johnson]

Works just fine for me.

 

[jaakkima]

How did you reach these pages on the website?

 

[burtlancast]

How did you reach these pages on the website?

Internet Archive: Wayback Machine

 

[jaakkima]

Oh yes, I meant from within the website... but I found the answer, that they are hyperlinks in the text essay on his homepage.