Reforming Science: Methodological and Cultural Reforms

One major concern is the imbalance among members of the research workforce and the funding currently available to sustain that workforce. In some respects the current scientific workforce resembles a pyramid scheme with a small number of principal investigators presiding over an army of research scientists, postdocs, students, and technicians who have little autonomy and increasingly uncertain career prospects. Despite efforts to foster the career development of young and female investigators, senior researchers remain disproportionately male and increasingly senior.

The well-known “publish or perish” mentality can drive some scientists to write papers whether or not they have anything new and important to say. The result is an unwieldy scientific literature in which the most highly cited papers are concentrated in a small number of journals (19) while a large proportion of papers are cited infrequently or not at all (2). Jinha has estimated that 50 million scholarly articles have been published since 1665, when the first modern journal appeared, with more than half of these published during just the last 25 years (21). Much of the recent scientific literature is repetitive, unimportant, poorly conceived or executed, and oversold; perhaps deservingly, much of it is ignored. However, the sheer number of papers generates an enormous burden on the peer review system (9). As the pool of qualified peer reviewers is inadequate to meet demand, the critical role of peer review in screening and correcting manuscripts prior to publication cannot properly function. These factors, as well as the difficulty publishing negative results (publication bias), serve to undermine the reliability of the scientific literature (13, 18).

A Darwinian struggle for existence can produce multiple behaviors, including competition, aggression, reciprocity, altruism, cooperation, and sometimes, unfortunately, cheating. Each of these behaviors may be readily observed among contemporary scientists. Competition can be beneficial when it provides a motivation for resourcefulness and innovation. However, competition in excess can be a compelling incentive for scientific misconduct ranging from egregious fraud to more subtle transgressions, such as selective reporting of results (12). The need to distinguish one's work from that of competitors can induce scientists to exaggerate the importance of their findings or downplay potential caveats (20). While this may seem relatively innocuous in the short run, the cumulative effect can be the erosion of public confidence in the scientific method (10). Another casualty of the competitive environment is timely communication. Scientists often keep valuable information to themselves for fear of being “scooped” by competitors, and attempts to place work in the most highly selective journals may delay publication for months or years. The entire shape of the research effort is distorted as researchers scramble to conform their work to targeted funding opportunities and steer away from risky lines of inquiry or projects requiring a lengthy time investment. Unfortunately, in science risk and reward often go hand in hand (24), so a focus on conservative projects means that opportunities to make revolutionary breakthroughs will be missed.

The current scientific enterprise is a highly competitive environment with a winner-takes-all system in which the greatest rewards are given to individuals who excel in scientific discovery, publication prestige and quantity, and grant funding. Since its inception, science has operated with the priority rule, which means that when different scientists work on the same problem, the credit goes to the one who provides the answer first (33a). Furthermore, success in science tends to beget more success, in a phenomenon that Merton called the “Matthew effect” after the Gospel of Matthew: “For whosoever hath, to him shall be given” (27a). The benefits of scientific success even appear to translate into better health, as recipients of Nobel Prizes live longer than those who are only nominated (30a). In any human endeavor, competition can be healthy when it promotes ingenuity and creativity and harnesses primal human energies. Thus, it has been argued that the priority rule in science is beneficial to society (33a). However, a winner-takes-all system has inherent dangers, including a greater likelihood of cheating by participants. In this regard, scientific fraud is a form of cheating that can lead to publications (as well as retractions). Fraudulent publications that affect public perception and policy, such as the 1998 Lancet article that linked autism to the measles vaccine (11), can be destructive to society. Despite occasional high-profile instances of fraud that adversely affect public opinion, society at this time continues to have an overwhelmingly positive view of science and scientists (22). Nevertheless, this trust is precious and must not be taken for granted, as illustrated by the rapid loss of trust in climate science and scientists that has followed the release of emails and computer files in the incident known as “climategate” (25).

The priority rule means that credit and its associated rewards go to the first individual or group of individuals who announce a discovery irrespective of the number of scientists who have contributed to the solution of the problem. How did the priority rule become the dominant economic system in science for assigning rewards? Why do scientists accept this economic system? There are no simple answers to these questions, but the priority rule was in place long before science became a major human enterprise. For example, Newton and Leibnitz quarreled in the 17th century about priority in the invention of calculus. A list of scientific priority disputes is maintained by Wikipedia (36). In the early days of the scientific revolution, most inquiry was carried out by individuals with sufficient means and leisure to devote to thinking, experimentation, and the publication of books. In a system that began without prizes, salaries, or grant applications, priority may have been the only source of prestige among a small number of colleagues. However, the priority rule probably contributes to many of the current maladies in science. The effort to get there first and grab the largest share of the credit undoubtedly contributes to such practices as citation bias, secrecy, and the appropriation of others' ideas and data. The competitive environment created by the priority rule is vividly apparent in Watson's unauthorized inspection of Rosalind Franklin's X-ray diffraction data as described in the The Double Helix (35). A more recent example is provided by the dispute over priority in the discovery of HIV (34). The race to be first encourages risk taking that can lead to scientific breakthroughs that benefit society while at the same time creating conditions that are detrimental to science. Hence, it may be worth reconsidering the priority rule and whether it is the best reward system for science.

Although the edifice of scientific understanding is sometimes envisaged as an accumulation of individual discoveries, in reality science is a community effort comprising innumerable interdependent contributions. Credit is disproportionately awarded to principal investigators for what is truly the product of teamwork, and nearly all scientific contributions are heavily dependent on knowledge obtained earlier. As Newton famously remarked, he was able to see further by standing on the shoulders of giants. In the spirit of an Amish barn-raising, a celebration of the collective achievement of science should subsume individual achievement.

Honest retractions may occur as a result of errors of methodology, conclusions, or approach. We call them “honest” because no malicious intent is involved. This category accounts for the majority of retractions (12). To reduce honest retractions would require implementing reforms that would reduce the number of such errors, and here we would make a few general suggestions under the category of methodological reforms.

Dishonest retractions occur when the deliberate manipulation or fabrication of results is discovered, prompting a retraction of the published work. Reducing dishonest retractions might be achieved either by increasing the penalties or reducing the incentives for misconduct. Severe penalties ranging from the loss of reputation to criminal charges of fraud are already in place, raising doubts about whether further penalties would be effective. The adoption of uniform standards for reporting retractions and perhaps the institution of a centralized database for documenting scientific misconduct might help to ensure that individuals guilty of misconduct are recognized. Scientists are conscious of reputation, and unflattering notoriety remains a potent disincentive for cheating. Nevertheless, there is a limited role for sanctions. Although it is important for misconduct to have consequences, unequivocal instances of misconduct are uncommon and represent only a portion of undesirable behavior. Therefore, we would focus efforts on reducing the incentives for dishonest actions by scientists.

As the Guinness Book of World Records shows, the desire to be first, whether in science or in the consumption of hot dogs at one sitting, is human nature and as such unlikely to change. Scientists will continue to race to be the first to achieve a goal, and journals will continue to vie for original reports. Perhaps the best place for a methodological reform to replace the priority rule is at the level of academic promotion and in the awarding of symbolic rewards, such as scientific prizes. In the present system, promotion decisions typically depend upon performance review by other faculty members who have a limited understanding of an investigator's work, which increases reliance on surrogate measures of quality, such as grant dollars, bibliometric analysis, and journal impact factor (8). A reform of the promotion process based on careful peer evaluations of scientific quality and the specific contributions of the authors might help to reduce the present emphasis on priority. Furthermore, increasing the value of collaborative publications when considering promotion could provide important incentives for greater cooperation between scientists.

Science traces its ancestry to natural philosophy, which in turn emerged from philosophy. Many early scientists were fully grounded in the philosophical fields of their time and contributed to both disciplines. Descartes separated philosophy from theology while making seminal contributions to mathematics and physics. Leibnitz discovered the calculus while at the same time proposing the metaphysical theory of monads. In this regard it is worth remembering that Ph.D. degrees granted in the natural sciences are actually Doctorates of Philosophy. However, scientific training today does not include significant instruction in philosophy despite the critical importance of the philosophical branches of logic, epistemology, and ethics to science. In fact, there are numerous instances in which philosophical thought has greatly influenced scientific discovery and vice versa. Einstein credited the philosopher Immanuel Kant with inspiration that led to the theory of relativity, and Einstein's scientific contributions have in turn influenced philosophy (29).

Philosophy is currently regarded as including four branches: epistemology, logic, ethics, and metaphysics. We believe that the knowledge found in each of these branches can contribute significantly to improving the scientific enterprise. Formal training in logic and epistemology could reduce the number of errors by scientists. One common error in science is the attempt to make positive inferences from negative data (e.g., ruling out a mechanism or cause and effect from negative experimental data). Errors in logical thought can lead to dogma and affect the direction of entire fields of study. For example, the conclusion that humoral immunity had no role in protection against many intracellular pathogens was based on an inability to demonstrate the efficacy of antibody with the methodology used (7). This conclusion evolved into dogma and greatly affected the direction of research, including the development of vaccines. However, this constituted a logical error because a possible protective role of antibody could not be excluded by experiments that showed no protection. The original conclusion was eventually shown to be faulty when hybridoma technology allowed the generation of protective monoclonal antibodies. A stronger foundation in epistemology, supplemented by an awareness of philosophical issues involving language, might also avoid some of the problems created in science from the misuse of such terms as “descriptive” (5) and “mechanistic” (6) in the categorization of projects and papers. Ethics, or “moral philosophy,” has already returned to scientific training as students are increasingly taught formal ethical principles in science, but much more could be done in this realm with the goal of reducing the number of dishonest retractions. Metaphysics, or the study of reality, is perhaps the most problematic branch of philosophy for scientists who regard it as a domain outside scientific pursuit. However, metaphysics is the antecedent of natural philosophy and is thus in the ancestral line of the scientific enterprise. Knowledge of metaphysical questions could help scientists to frame problems and perhaps approach the boundary without crossing it.

In science, certainty is often defined in probabilistic terms. Knowledge of probability and statistics is essential for the design, execution, and interpretation of many scientific experiments. Although most scientists have some knowledge of probability and statistics and can calculate “P values” using statistical software, the level of statistical expertise varies greatly among individuals. In fact, much deeper knowledge of the foundations of these disciplines is needed. For example, a recent paper in Science was retracted because the authors assumed that two correlated variables were independent while these variables were in fact additive and dependent (30). As the late statistician Stephen Lagakos observed, “Sure, you can lie with statistics … but it's a lot easier to lie without them” (R. Schooley, personal communication).

There is conclusive evidence that the use of checklists can reduce errors in human activities ranging from aviation to surgery (17). Science should be no exception. An increased use of checklists in the conduct of scientific experimentation and publication could conceivably reduce scientific errors and consequently improve productivity, reduce waste, and prevent retractions. Clearly, different types of checklists would be needed for different occasions. For example, a simple checklist can be constructed for a scientific result in which a stimulus appears to cause an effect (Table 1), which if followed would enhance conceptual rigor and reduce the likelihood of a false conclusion.

The reward system in science will be difficult to reform. However, any attempt to reduce the incentives for cheating will have to address structural issues in the payoffs for scientists. Winner-take-all strategies have been proposed to be male evolutionary strategies that disproportionally reward risk taking in the production of offspring (33). Hence, it might be argued on more than one level that science is too “masculine.” A reform to the payoff system may also have other benefits in addition to reducing the incentives for cheating. Given the increasing connectivity between fields and specialties of science, there is an increasing need for collaboration, yet a system of winner-takes-all is inherently unfair to collaborators. A different reward system could promote team science and thus promote the overall progress of the scientific enterprise. A less “masculine” scientific culture could be a fairer, more honest, more cooperative, and more successful enterprise. In this regard, we note that efforts are already ongoing to promote and recognize collaboration and team science that provide guidance for communication, sharing credit, and handling conflict (3).

Such a culture would place a greater emphasis on the derivative and collaborative nature of scientific advances, along with a reduced emphasis on rewarding hypercompetitive behavior and the cult of the “rock star” investigator. The evolutionary biologist David Sloan Wilson has recounted an attempt to improve the productivity of egg-laying hens by Purdue researcher William Muir (37). The approach of selecting the most productive individual hens from each group to breed the next generation was compared with selecting the most productive groups of hens. Unexpectedly, the latter approach was most successful because the individuals within successful groups had learned to function cooperatively, and the happier hens laid more eggs. Productivity plummeted when the star performers were grouped together, and all but three hens in this group were dead by the end of the experiment. After a lecture describing these results, a professor in the audience exclaimed, “That describes my department! I have names for those three chickens!” In fact, we all know who those chickens are.

Science functions best when scientists are motivated by the joy of discovery and a desire to improve society rather than by wealth, recognition, and professional standing. In spite of current pressures, it is perhaps remarkable that many scientists continue to engage in selfless activities such as teaching and reviewing, decline to publish work that doesn't meet stringent standards for quality and importance, freely share reagents and knowledge without worrying about who gets the credit, and take genuine pleasure in supporting the efforts of other investigators. Such individuals should be recognized and emulated.

The systems biologist Uri Alon has written a thought-provoking essay on how to build a motivated research group (1). He concludes that individuals need to be matched with projects appropriate for their talents and passions and that they require both autonomy and connectedness with other members of the group. The research group is but a microcosm of the entire research community. A healthy scientific environment is one in which the freedom to do what one wants is complemented by support and stimulation from a community. This will enhance productivity and innovation.