John P. A. Ioannidis

“ Improvements in study design standards could improve the reliability of results [43] . For example, for animal studies of interventions, this would include randomization and blinding of investigators [27] . There is increasing interest in proposing checklists for the conduct of studies to be approved [44] , [45] , making it vital to ensure both that checklist items are indeed essential and that claims of adherence to them are verifiable.
Reporting, review, publication, dissemination, and post-publication review of research shape its reliability. ”

“ Dynamics between different stakeholders are complex. Moreover, sometimes the same person may wear many stakeholder hats; e.g., an academic researcher may also be journal editor, spin-off company owner, professional society officer, government advisor, and/or beneficiary of the industry.
Research Currencies
Publications and grants are key “currencies” in science (Table 2) . They purchase academic “goods” such as promotion and other power. Academic titles and power add further to the “wealth” of their possessor. ”

“ Research efforts are embedded in wider societies, which have provided scientific developments that differ according to time period and location. What can be done to enhance the capacity of science to flourish and to assess and promote this capacity across cultures that may vary in attitudes toward skepticism, inquisitiveness, and contrarian reasoning? ”

“ Sharing of data, protocols, materials, and software has been promoted in several -omics fields, creating a substrate for reproducible data practices [28] – [29] . Promotion of data sharing in clinical trials may similarly improve the credibility of clinical research [30] . Some disadvantages have been debated, like the potential of multiple analysts performing contradicting analyses, difficulties with de-identification of participants, and the potential for parties to introduce uncertainty for results that hurt their interests, as in the case of diesel exhaust and cancer risk [31] . ”

“ The current system does not reward replication—it often even penalizes people who want to rigorously replicate previous work, and it pushes investigators to claim that their work is highly novel and significant [62] . Sharing (data, protocols, analysis codes, etc.) is not incentivized or requested, with some notable exceptions [63] – [64] . With lack of supportive resources and with competition (“competitors will steal my data, my ideas, and eventually my funding”) , sharing becomes even disincentivized. Other aspects of scientific citizenship, such as high-quality peer review, are not valued. ”

“ Given the abundance of data, research on research (i.e., meta-research) can derive empirical estimates of the prevalence of risk factors for high false-positive rates (underpowered studies; small effect sizes; low pre-study odds; flexibility in designs, definitions, outcomes, analyses; biases and conflicts of interest; bandwagon patterns; and lack of collaboration) [3] . Currently, an estimated 85% of research resources are wasted [5] .
Effective Interventions
We need effective interventions to improve the credibility and efficiency of scientific investigation. ”

“ The achievements of science are amazing, yet the majority of research effort is currently wasted. Interventions to make science less wasteful and more effective could be hugely beneficial to our health, our comfort, and our grasp of truth and could help scientific research more successfully pursue its noble goals.Provenance: Commissioned ”

“ Value is given to sound ideas and results that are replicated and reproducible [71] rather than publication per se. Further value is given to publications that lead to things that work, like effective treatments, diagnostic tests, or prognostic tools that demonstrably improve important outcomes in clinical trials. Additional value is obtained for sharing and for meaningful participation in peer review and educational activities of proven efficacy. A peer reviewer or an editor occasionally may contribute the same value as an author. ”

“ Different stakeholders have their own preferences about when reproducibility should be promoted or shunned. Pharmaceutical industry teams have championed reproducibility in pre-clinical research [53] , [54] because they depend on pre-clinical academic investigations accurately pinpointing useful drug targets. Conversely, the industry is defensive about data sharing from clinical trials [55] , which occurs at a point in the product development when re-analyses may correctly or incorrectly [56] invalidate evidence supporting drugs in which it has already invested heavily. ”

“ We must diminish biases, conflicts of interest, and fragmentation of efforts in favor of unbiased, transparent, collaborative research with greater standardization. However, we should also consider the possibility that interventions aimed at improving scientific efficiency may cause collateral damage or themselves wastefully consume resources. ”

“ Besides the changes adopted in the first example, obtaining grants, awards, or other powers are considered negatively unless one delivers more good-quality science in proportion. Resources and power are seen as opportunities, and researchers need to match their output to the opportunities that they have been offered—the more opportunities, the more the expected (replicated and, hopefully, even translated) output. Academic ranks have no value in this model and may even be eliminated: researchers simply have to maintain a non-negative balance of output versus opportunities. ”

“ Scientists are only one group in a larger network (Table 1) in which different stakeholders have different expectations. Stakeholders may cherish research for being publishable, fundable, translatable, or profitable. Their expectations are not necessarily aligned with one another. Scientists may continue publishing and getting grants without making real progress, if more publications and more grants are all that matters. ”

“ Registration of randomized trials [22] (and, more recently, registration of their results [23] ) has enhanced transparency in clinical trials research and has allowed probing of selective reporting biases [24] , [25] , even if not fully remedying them. It may show redundancy and allow better visualizing of the evolution of the total corpus of research in a given field. Registration is currently proposed for many other types of research, including both human observational studies [26] and nonhuman studies [27] . ”

“ Such currency values will vary across microenvironments and specific fields and situations. A putative currency value of 1 for a publication unit (e.g., a first- or senior-authored paper in a highly respectable journal in the field) , 5 for a sizeable investigator grant (e.g., an R01 in the US) , and 2 for a post-doctoral fellow means that a scientist would find equivalent value in publishing five such papers as first or senior author as in getting an R01 as a principal investigator, or in publishing two such papers as in getting a post-doctoral fellow to work for her. ”