Rigorous Science: a How-To Guide

Rigor is a prized quality in scientific work. Although the term is widely used in both scientific and lay parlance, it has not been precisely defined (1). Rigor has gained new prominence amid concerns about a lack of reproducibility in important studies (2, 3), an epidemic of retractions due to misconduct (4), and the discovery that the published literature is riddled with problematic images (5). Insufficient rigor may be slowing the translation of basic discoveries into tangible benefits (6, 7). New initiatives aim to understand deficiencies in scientific rigor and to make research more rigorous (8–10). Here, we consider the meaning of rigorous science and how it can be achieved.

The word rigor is derived from an old French word, “rigueur,” meaning strength and hardness (11). In scientific vernacular, the underlying concept of strength resonates in the expressions “hard data” and “solid work” used to convey a sense of reliable and trustworthy information. In common usage, the word “rigor” has evolved to mean the quality of being exact, careful, or strict (12). Although the words “exact” and “careful” also apply to science, additional definition is needed since practicing rigorous science means more than mere exactness and care in experimental design. An experiment in which all components were exact in their proportions and the procedures carefully executed would still not be considered rigorous in the absence of appropriate controls. Hence, the definition of scientific rigor requires a deeper exploration than can be provided by simple perusal of the dictionary.

The scientific literature adds surprisingly little to our understanding of rigor, with the term almost always used without definition, as if its meaning is self-evident. The NIH has recently defined scientific rigor as “the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results” including “full transparency in reporting experimental details so that others may reproduce and extend the findings” (13). While we credit the NIH for providing a starting point for discussion, we find the NIH definition of rigor to be both excessively wordy and disconcertingly vague, as well as complicated by an insistence on transparency and reproducibility, which may be desirable but are arguably separate from rigor.

We suggest that rigorous science may be defined as theoretical or experimental approaches undertaken in a way that enhances confidence in the veracity of their findings, with veracity defined as truth or accuracy. Rigorous science could be entirely theoretical, as exemplified by a thought experiment used to illustrate a principle, such as Schrödinger’s cat or Maxwell’s demon in physics, or entirely experimental, as illustrated by Cavendish’s measurement of the gravitational constant at the end of the 18th century. However, in the biomedical sciences, most research has both theoretical and experimental aspects.

Different fields vary in the level of uncertainty that they are willing to accept with regard to conclusions. Certainty in science is often couched in terms of the probability that the null hypothesis may be rejected, which in turn depends on the methodologies employed. For example, the Higgs boson was announced when physicists were certain to “five sigma” or a P value of 3 × 10⁻⁷ (14). In contrast, many biological and medical studies accept a P value of 0.05, although more stringent criteria have been advocated (15). Does this make physics more rigorous than biology? Not necessarily—differences in the complexity of physical and biological phenomena as well as limitations in methodology determine the level of certainty that is practically achievable in these disciplines. Hence, a definition of rigorous science cannot rely on strict and arbitrary levels of certainty.

Traditional Chinese philosophy, Hinduism, Islam, and Judaism are each founded on five elements, pillars, or sacred texts. In Judaism, the first five books of the Hebrew bible are collectively referred to as the Pentateuch. Here, we humbly propose a Pentateuch for scientific rigor (Fig. 1).

Rigorous scientific practices enhance the likelihood that the results generated will be reproducible. However, reproducibility is not an absolute criterion of rigorous science. One might rigorously characterize the mass, composition, and trajectory of a comet that collides with the sun, but those measurements could never be reproduced since each comet is unique. Nevertheless, improvements in scientific rigor are likely to improve reproducibility.

The five principles outlined above provide a road map for increasing rigor in the biomedical sciences (Fig. 1). An obvious step is to strengthen didactic training in experimental design, statistics, error analysis, logic, and ethics during scientific training. We have previously called for improvements in graduate education, including areas of philosophy such as logic, as well as in probability and statistics (19). A recent survey of over 1,500 scientists found that there is considerable support for such reforms (20). However, with the exception of statistics, none of these disciplines are formally taught in the training of scientists, and even statistics is not always a curricular requirement (21). Reforming scientific education will be a major undertaking since most academic faculty lack the necessary background to teach this material. A multifaceted approach would include greater attention in the laboratory environment and improvements in peer review and constructive criticism, as well as formal didactics. This would need to be accompanied by a cultural change in the scientific enterprise to reward rigor and encourage scientists to recognize, practice, and promote it. To initiate the process, we make the following five actionable recommendations.