Blacklists and Whitelists To Tackle Predatory Publishing: a Cross-Sectional Comparison and Thematic Analysis

There is increasing concern in the scientific community and society about “predatory” journals, also called fake, pseudo-, or fraudulent journals. These allegedly scholarly open-access (OA) publishing outlets employ a range of unethical publishing practices; despite claiming otherwise and charging for it, they do not provide editorial services and scientific quality control. Predatory journals thus exploit the OA model of publishing; they generate revenue by requiring authors or their institutions to pay article-processing charges (APCs). There is widespread agreement that fraudulent journals pose a threat to the integrity of scholarly publishing and the credibility of academic research (1–7).

There have been various attempts to derive criteria to characterize and identify predatory journals, in order to help researchers to avoid these outlets (8). These attempts include the compilation of lists of fraudulent journals (blacklists) or trustworthy journals (whitelists). The best-known list is the blacklist of “potential, possible, or probable predatory scholarly open-access journals” by Jeffrey Beall, a librarian who coined the term “predatory” journal in 2015 (herein referred to as Beall’s list) (9). Beall took his list down in 2017, probably due to lawsuits instigated by publishers included in the list (7). At present, the list is maintained and updated by an anonymous scholar at a different site (10). While blacklists aim to alert authors to presumed fraudulent journals, whitelists take the inverse approach of providing an index of vetted, presumably legitimate outlets. Whitelists can differ substantially in their governance structures and funding models. While some whitelists require subscription or membership fees from journals or publishers, others are independent of publishers. The selection of journals for inclusion in blacklists and whitelists is based on a set of criteria which a journal has to meet in order to be included. Predominantly, whitelist criteria refer to proficiency and adherence to best practices to confirm the legitimacy of a journal. In the case of blacklists, these criteria describe undesirable, unethical, and deceptive practices that are believed to characterize fraudulent journals (11). As such, the two types of lists present different perspectives on the same challenge, ensuring the quality and legitimacy of academic publishing practices. Approaches other than blacklists and whitelists include decision trees or checklists to help authors distinguish between fraudulent and legitimate journals, for example, Think.Check.Submit. (1, 12, 13).

Despite the ongoing discussions on fraudulent publishing and the growing body of research on its market characteristics and prevalence, the defining attributes of fraudulent, illegitimate journals remain controversial (14, 15). Given that the prevalence of predatory journals can be assessed only if their assessment is based on a clear definition of fraudulent publishing, systematic studies on the understanding of quality and legitimacy in academic publishing are needed. This study aims to contribute to a better understanding of prevalent notions of good and poor quality in academic publishing by analyzing the inclusion criteria and journals and publishers included in blacklists of fraudulent journals and whitelists of legitimate journals.

(This article was submitted to an online preprint archive [16].)

Two blacklists, the updated list of Beall (10) and Cabells Scholarly Analytics’ blacklist (17) (herein called Cabell’s blacklist), and two whitelists, the Directory of Open Access Journals (DOAJ; https://doaj.org/, accessed 2 December 2018) and Cabells Scholarly Analytics’ whitelist (17) (herein called Cabell’s whitelist), met our inclusion criteria. A subscription to the lists of Cabells Scholarly Analytics was purchased for this study, whereas access to the DOAJ and the updated Beall’s list was free of charge. While Beall’s list and the DOAJ are limited to OA journals and publishers, Cabell’s lists cover both OA and closed-access journals and publishers. Beall’s list included the fewest journals, but unlike the other three lists, Beall’s list contains two separate lists of journals and publishers, which are independent of one another. Journals included in Beall’s list of “standalone journals” are not linked to the publishers listed in Beall’s list of publishers. For this reason, we analyzed the lists’ contents separately for journals and publishers. Table 1 summarizes the features of the included lists.

This comprehensive study of blacklists of predatory journals and whitelists of legitimate journals triangulated quantitative and qualitative approaches. The qualitative analysis elucidated the multidimensional understanding of quality in academic publishing that underpins blacklists and whitelists. This multidimensionality is reflected at both the level of the specific topics addressed by criteria and the more abstract level of concepts. The thematic analysis of topics and concepts covered by the 198 inclusion criteria for the different lists resulted in seven topics and four broader concepts. It showed important differences between lists in the emphasis given to these topics: blacklists gave much emphasis to business practices, editorial services, and publishing practices. In contrast, whitelists covered policy most extensively, followed by business practices, editorial services, and peer review. Regarding the broader concepts, whitelists gave more emphasis to transparency and less emphasis to professional standards and ethics than blacklists. The two types of lists thus complement each other and contribute to a broader understanding of quality. Of note, the whitelist criteria were easier to verify than the criteria used by blacklists. In general, all lists appear to prioritize easily verifiable dimensions of a journal’s quality over the quality of scientific evaluation.

In the DOAJ, more criteria relate to transparency of business and publishing practices than to the quality of peer review. Studies have used transparency as a proxy of the quality of a journal (18). However, other studies have shown that statements made by a journal can be false and its transparency spurious (19). This indicates a risk of endorsing the legitimacy of a journal based on its transparent nature, while at the same time ignoring journals’ lack of best practices in peer review. Indeed, when John Bohannon, a science journalist at Harvard University, submitted a bogus scientific paper with major flaws to DOAJ-listed publishers, the weaknesses in their peer review were clearly exposed (20). Similarly, blacklist criteria predominantly relate to ethical issues and professional standards and not to the quality of the scientific evaluation of article submissions. Cabell’s whitelist appears more balanced in valuing different dimensions of journal quality, including peer review. The quality of peer review is difficult to evaluate. Standardized instruments have been used previously, for example, in the context of assessing the impact of open peer review (21, 22). Interestingly, publishers who were criticized for poor peer review and included in Beall’s list, such as MDPI or Frontiers, plan to make peer review reports openly accessible along with the article, so that readers can judge the thoroughness of its scientific evaluation. Evaluating the quality and thoroughness of scientific evaluation will require a sound definition of what constitutes “good” or “rigorous” peer review, but most standardized instruments appear to lack any theoretical foundation (23).

The quantitative analysis investigated overlaps between the contents of blacklists and whitelists. The considerable overlap between the two blacklists indicates that Cabell’s list may use Beall’s list as a source of predatory publishers. The relatively small overlap between the whitelists might be explained by the fact that the lists pursue different objectives regarding coverage. The overlaps that we found between blacklists and whitelists may be interpreted in several ways. First, these journals may be “false positives” on the blacklists, i.e., wrongly classified as fraudulent. Indeed, Beall’s list has been criticized for not distinguishing fraudulent from low-quality journals or from emerging journals, for example, journals from the global south. The latter may not be able to afford membership in associations or may not yet have been accepted as members and thus be misclassified by blacklists (24–26). Others have argued that even if describing undesirable practices, some of the criteria Beall used to characterize fraudulent journals are also applicable to established, presumably legitimate journals (27, 28). Second, these journals might be “false negatives” on the whitelists, i.e., wrongly classified as being legitimate, based on criteria that are easily verified and easily met but that miss other, fraudulent practices, for example, the lack of adequate peer review. Furthermore, the status of a journal may change over time, as publishers and editors abandon questionable practices or good practices. Lists therefore need to be kept up to date, and journals should be periodically reassessed. Third, some journals may operate in a gray zone for extended periods, meeting some blacklist and some whitelist criteria. Fourth, beside their common goal of identifying legitimate or illegitimate journals and publishers, the lists may follow other agendas, which might require a different weighing of inclusion criteria or might affect the inclusion or exclusion of certain journals and publishers. Although the overlap was small, the criteria in use for the different lists are unlikely to capture fully the quality and legitimacy in academic publishing. In other words, these lists can be useful, but they do not provide a completely accurate delimitation between legitimate and illegitimate journals. To gain a comprehensive understanding of the accuracy of lists, future studies could include additional lists in the analysis, such as the expanded version of Beall’s list (29, 30). We will examine the characteristics of journals that ended up on both blacklists and whitelists in detail in a follow-up study.

To our knowledge, this is the first systematic, comparative analysis of blacklists of predatory journals and whitelists of legitimate journals. A recent scoping review by Cobey and colleagues identified 109 characteristics of predatory journals, which were extracted from 38 empirical studies including a definition of predatory journals (31). In line with what we found for blacklist criteria, Cobey et al. report that most characteristics used to define predatory journals do not relate to the quality of the scientific evaluation of article submissions but rather to the journal’s business operations and revolve around the lack of transparency, integrity, and quality.

Our study has several limitations. First, we used Google to identify lists but may have missed some blacklists or whitelists. The results of Google searches differ across users in unpredictable ways and are not fully reproducible. Second, as fuzzy matching allows comparisons of strings on the basis of similarity rather than on a precise match, it is possible that we missed journals and publishers contained in both a blacklist and a whitelist. We downloaded the lists in December 2018. They therefore show a snapshot in time and might have changed since then. For instance, the overlap between the two Cabell lists reportedly was due to an internal system error by Cabells Scholarly Analytics and was rectified after we published the preprint (16). Third, qualitative analysis always entails a certain degree of subjectivity, as the assessor’s knowledge, background, and judgement influence data interpretation. To mitigate the subjective nature of data interpretation, two assessors analyzed the inclusion criteria. Fourth, in interpreting the criteria, we did not take into account potential list-specific weighting of criteria (the DOAJ has a hierarchy of criteria) but weighted all criteria equally for the sake of cross-list comparability. Finally, we restricted eligible blacklists and whitelists to interdisciplinary and internationally available lists. We thus did not consider country- or discipline-specific lists, which might differ in their understandings of quality, transparency, and legitimacy in academic publishing.

We used a mixed-methods approach, combining quantitative and qualitative methods. Using record linkage methods, we compared blacklists and whitelists in terms of overlap, i.e., with regard to the journals and publishers that they indexed. We then qualitatively examined and interpreted the inclusion criteria of blacklists and whitelists.

M.S., A.S., K.M., and M.E. conceived this project and drafted the research protocol. Subsequent development of the project was carried out by M.S. M.S. conducted the quantitative analysis. The qualitative analysis was performed by M.S. and A.S., in collaboration with K.M. M.S. and M.E. drafted the manuscript, and K.M. and A.S provided feedback.

M.E. is the President of Research Council of the Swiss National Science Foundation (SNSF). M.S., A.S., and K.M. work for the SNSF.

M.E. affirms that this article is an honest, accurate, and transparent account of the study being reported, that no important aspects of the study have been omitted, and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.