On the Nature of Quality in the Contexts of Academic Publication and Sustainability

Six experienced academic reviewers and editors explored the nature of quality in academic publication processes in the contexts of sustainability, education for sustainability and the Sustainable Development Goals (SDGs). This article documents their exploration as a collaborative autoethnography structured around the authors’ personal reflections on matters such as: how current quality indicators define the quality of academic publications; how effective current quality assurance processes may be; how congruent open access publication processes may be with the ideals of sustainability and of the SDGs; and about what new and different indicators of quality might look like. An inductive analysis of their reflections yielded three emergent and reoccurring themes: casting doubt on the fitness for purpose of current academic publication processes and means to assure their quality; seeking justice for all involved in academic publication; and creating opportunities for change. In writing this article, authors considered these themes and how academia might address them.

Introduction

Academic publication is changing rapidly, perhaps more rapidly than it has before. The number of journals is increasing, as are the number of articles and, for many, the number of articles per academic. The number of languages to which publications are being translated is also growing and, increasingly, research has multilingual representations. Access to publications is also expanding, in particular due to the adoption of open access business models, voluntarily for some academics but mandated by some states. Traditional publishing is flourishing, if measured by the financial returns to the shareholders of the five largest publishing houses (Mathews, 2018). Considering the number of articles published and the article processing fees involved, so is the new world of open access publishing, in the sense of financial returns to at least some publishers.

Academic publication is also inseparable from the lives of academics. Publications and the industries that support them interact with all aspects of academic research careers including job applications, job security, research funding, researcher reputation, salaries, and their influence as teachers and researchers. In support of these interactions, academic publication is being “measured” nowadays as never before and these measurements become elements of researchers’ personal and institutional identities. Journals, articles, authors, and their institutions are codified, compared and ranked by citation and reputation. Many academics will know their personal h-index, the impact factor of their favourite journal and the international rank of their institution. They may even know and be interested in their last article’s current number of online views, downloads, citations, and social-media indicators.

However, these trends are not necessarily all positive. Perhaps many would recognise a creeping inflation in academic performance indicators and in academics’ abilities to match these expectations with ever increasing research outputs. Indeed, Macfarlane (2019) and Garbett and Thomas (2020) have described some related trends. Academics may celebrate measures that indicate increases in the quality of their work and of its contributions and impacts; but they may also have an academic interest in how quality is measured and in the implications of using measures of quality that somehow have become accepted and acceptable without their involvement.

As suggested by Shephard et al. (2019) academics may wish to ensure that the dramatic increase in publication is matched by similar increases in the number of good ideas that enter intellectual discourses and by increases in opportunities for all academics to be involved in these discourses. Perhaps academics want publication and related opportunities to be fairly distributed, or based on the quality of academic endeavour, or on equality of opportunity for all academics? Or based on trust, as explored by Herwald (2018) in a recent critique of open access publication. Broad-based academic explorations of fairness (see as examples; Meriläinen et al., 2008; Marginson, 2008) identify disadvantages based on language, nationality, age, socio-economic class, access to funding, experience, race, and gender, and increasingly describe and analyse academia in the context of publication as hegemony.

Much of this analysis applies to all disciplines. The authors of this article, however, are all actively researching and publishing in the broad field of enquiry that relates education to sustainability, albeit from a diversity of disciplinary bases. We share a commitment to the achievement of the sustainable development goals (SDGs) and an understanding that higher education internationally has multiple roles to play in this quest. We are also all experienced academic reviewers and editors. We came together as current or past editors (in various capacities) of the MDPI open access journal “Sustainability” to explore the nature of quality in the contexts of academic publication, sustainability, education for sustainability, and the SDGs. This article documents this exploration in these contexts as a collaborative autoethnography (CAE) structured around the authors’ personal reflections on four questions of interest. How well do current quality indicators define the quality of publications? How effective are current quality assurance processes? How congruent are open access publication processes with the ideals of sustainability and of the SDGs? What new and different indicators of quality might we imagine?

Materials and Methods

For this study, the authors, as researchers, adopted a self-study research framework incorporating a CAE. CAE “engages two or more autoethnographers in a research team to pool their lived experiences on selected sociocultural phenomena and collaboratively analyze and interpret them for commonalities and differences” (Chang et al., 2013, p. 251).

This approach combines reflective self-review within a supportive collaborative framework. Our intention throughout our collaboration has been to emphasise the importance of “self” to both the practice being researched and to the practice of research in the context of collaboration (as exemplified by Hamilton et al., 2008; and Hains-Wesson and Young, 2017). The rigour of this research approach in this context is supported by the validity checks proposed by LaBoskey (2004) incorporating: self-initiation; being focused on improvement; being interactive; and based on trust. All checks support the validity of our undertakings and the interpretation of our approach as a CAE. Some of the authors formed part of a panel presentation at the 2019 European Conference on Educational Research (ECER) which explored interactions between academic publishing and sustainability education (Shephard et al., 2019), and then continued that discussion via emails. Other authors joined the discussion in late 2019 following an open invitation extended to those who had been editors and sub-editors to the journal Sustainability’s section “Sustainable Education and Approaches”. Discussions continued via email, with difficulty, throughout the 2020 Covid-19 pandemic.

Discussions initially established and agreed our questions of interest and subsequently encouraged each of the authors to each produce a personal written reflective commentary that addressed the questions articulated above. Reflective commentaries took various forms. Some were short essays and some were collections of emails. Authors chose not to agree precise definitions of complex constructs such as “open access” or “traditional,” rather allowing such meanings to come to light in our reflective commentaries or subsequent discussions. Authors also collaborated on the development of an annotated bibliography to share reading and insights from relevant academic sources. Reflective commentaries were anonymised by one author and then shared with all authors. As there were only six commentaries and as authors had corresponded beforehand, anonymity was accepted by all to be only symbolic within our group.

Three authors, acting as a research subgroup, read, and re-read these commentaries as the first stage of an inductive analysis as described by Thomas (2006). As Thomas suggests, “The primary purpose of the inductive approach is to allow research findings to emerge from the frequent, dominant, or significant themes inherent in raw data, without the restraints imposed by structured methodologies” (238). Thomas describes several rationales for general inductive analysis including: “… to develop a model or theory about the underlying structure of experiences or processes that are evident in the text data” (238), and as authors we confirmed that this described our intentions. Individual researchers within the subgroup developed their own perspectives on the dominant themes in the data and shared these within the research subgroup by email. The research subgroup subsequently video-conferenced on two occasions to negotiate and to agree the dominant recurring themes that together describe the underlying structure of the wider group’s experiences.

With reference to Thomas’s theme of assessing the trustworthiness of this analysis, and to Lincoln and Guba’s (1985) four general types of trustworthiness in qualitative research, it is hoped that the credibility, transferability, dependability, and confirmability of this analysis would be reasonable, given the voluntary nature of participation and the emphasis on “self” in the analysis. Nevertheless, and in line with Thomas’s advice, the credibility of the three dominant themes to arise from the analysis was tested using “peer-debriefing,” which involved ongoing discussions within our research group. Transferability and dependability of the analysis are tested, to a degree, by comparison with international literature within this article. Confirmability, in particular, has not been tested but may come later, as others work with, and within, similar groups to reflect on the nature of quality assurance in academic publication and its congruence with ideals of sustainability and the SDGs. Our study explored an arguably complex and subjective issue; hence, its scholarly value lies in its interpretive qualities and transferability potential (Marshall, 1996) on issues relating to academic publication including possible social injustices, equity, and the contribution of research publication for SDGs.

Results

Individual reflective commentaries provided a rich source of data. Although each commentary included its own unique stance on our questions of interest, we considered the collection to be sufficiently congruent to be analysed together. Several themes within these commentaries were evident to us but notably there was considerable variability in the degree to which each commentary emphasised each theme and how each theme was experienced and described by each of us. We identified and reflected on: different ideas about the nature of quality in the context of academic publication and the processes used to assure this quality; how open access publication in general may support progress towards sustainability but how current models of open access may not; what may need to change for academic publication to be compatible with the ideals of a sustainable future; our discomfort with current systems and our roles within them; the extent to which reputation dominates current notions of quality; how current systems have been designed to create certain outcomes; and how they may need to be redesigned to create different outcomes. We interactively and progressively re-interpreted and amalgamated these themes into three emergent and reoccurring themes that together represented novel reinterpretations of our questions of interest and all six reflective commentaries. Our three emergent and reoccurring themes were: fitness for purpose of current academic publication processes; seeking justice; creating opportunities for change. Our themes are described below and illustrated with quotations from our anonymised reflective commentaries. Some words or phrases have been replaced by #### to preserve anonymity.

Fitness for Purpose of Current Academic Publication Processes

Reflective commentaries included deep concern about the quality of our academic publication processes, and about the quality assurance processes that underpin them, to the extent that we worried about their fitness for purpose.

Although concerns frequently addressed particular facets of academic publication processes, some were certainly of a general and far-reaching nature;

If the publishing about sustainability issues in #### and in other journals is not in itself sustainable, where are we then steering?

I can’t help but feel that we have not yet optimised either the nature of our quality assurance processes or of the basis on which our publications could and should be sorted.

Reflective commentaries tended not to be overtly critical of particular publishers, particular journals or particular publication models. Critique relating to traditional and open access models, for example, tended to be balanced, as in;

… neither the traditional nor the new open access journals meet the suggested criteria for publishing in the field of ESD [Education for Sustainable Development…

Nevertheless, most commentaries expressed at least an implicit hope that open access as opposed to more traditional forms of publication showed most promise;

… if we contrast the strengths and weaknesses of this system [open access], my opinion is that it has more advantages than disadvantages.

Our reflective commentaries expressed concern about the fitness for purpose of the following attributes of academic publication.

• Impacts consequential to inadequate peer-review processes;

If for example the editors of a Special Issue involve unqualified reviewers, also papers may be published [which do] not completely meet all quality criteria.

• Limits to the power held by those charged with the responsibility of quality assurance to discharge their responsibilities to assure the quality of what academics publish;

I had hoped to develop a community of academics with particular and deep interests in sustainability education to explore these matters, but the complexity of the business model adopted by open access online publishers, editors’ limited abilities to influence and audit the system that they are part of, and the sheer numbers of invited editors, made the concept of a community of practice in this space quite impossible.

I see it as very important to accept only review articles on topics I do know well, and if I have enough time to read the text carefully. Yet, it seems like I cannot guarantee quality by rejecting, since I do not know what happens after that with the text.

I think that the editorial staff do not seriously consider reviewers’ comments against publications. For me, the most incredible thing that a paper I recommended for rejection was published a few months later but without any major address of the issues I raised in the review.

• Basic inadequacies in some editorial processes (such as choosing suitable peer reviewers) and, perhaps, unreasonable dependency on academics themselves to overcome these inadequacies;

I get too many articles to review that have nothing to do with my work, of course I don’t do the review.

The review process is generally well-organized, but totally depends on the editors of the Special Issue. Sometimes the #### [administrative] editors propose reviewers which do not have any expertise in the field of the paper, and sometimes the quality of the reviews is very low. Furthermore, only very little time is given for the review, the revision by the authors and also the decision of the editor. So high quality of the published papers is only guaranteed when editors of the Special Issue invest a lot of time to revise the papers themselves and only accept papers that are in line with all quality criteria. The Journal #### publishes (too) many papers, almost no paper gets rejected.

• Consequences of excessively long, and short, periods of time between writing and publication;

The longest academic processing time I have experienced was about 2.5 years in a well-reputed journal. After that there was an embargo of 1.5 years until it was on open access. After such a long time, you might not even agree with your own thoughts anymore.

My concerns with the quality of papers submitted to #### journal and #### journals in general relate to quality assurance process, including the limited time it takes from submission of article to acceptance, being pushy to editors and authors. While a lengthy process is not a guarantee for quality, and frustrates many authors for various reasons, review processes for quality would require more time.

In addressing fitness for purpose, our commentaries questioned the nature of quality in the context of academic publication. The ways in which reputation (of journals, editors, and of other authors who publish in the journals) has endured as a substitute for quality was particularly noted;

… some publications are prized, others bring disrepute by association.

… the sticking points remain the credibility of the figures of authority and what the proper noun of the publication and publishing house signify in terms of hierarchies of quality.

In most cases the perception that reputation is synonymous with quality was itself critiqued;

Since many of the #### researchers are involved in science, they are very afraid of publishing in journals that are not highly ranked and have a good reputation. They think a lot of their careers and possibilities to receive funding. Not so much on reaching a broad audience, as I interpret the situation.

I also know that some people use the fact that people of international reputation are publishing in #### journal, and would not be associated with a journal against which there were issues about academic process and peer review, as a measure of quality. In my view, this is not a universal indicator and should not be applied blindly.

Reflective commentaries also sought to clarify how quality assurance aspirations differ between traditional paper-based and open access online publications;

It is clear to me that the major quality control in traditional paper-based journal publishing is a product of limits imposed by the number of printed pages. Because there are only a certain number of pages to fill, only the best articles can be chosen so there is no need for an absolute definition of quality, only ways to enable the sorting or ranking of submitted articles. This is undertaken by the most highly qualified, experienced, and influential peer-reviewers and editors and is essentially a self-maintaining hegemonic system that excludes many and much. In some respects, quality is defined by what and who is excluded, rather than by what and who is included.

A typical open-access online-publication system has no page limits so all those articles submitted that meet prescribed minimum quality can be published. Now there is a need for an absolute definition of quality as this is far more than a sorting process. Minimum quality for the journal #### is defined by what two peer reviewers think, with the proviso that the peer reviewers must themselves have a minimum publication record.

It would be difficult to read these six reflective commentaries by academics deeply invested in academic publishing for the common good, without noting deep frustration. This extends far beyond resignation to accept an unfit-for purpose-system as inevitable, but rather extends to a shared commitment to achieve change.

Seeking Justice

The reflective commentaries identified unjust procedures in the academic publishing industry. The commentaries described biases related especially to regions, language, and wealth and noted that some have control over processes and products while others are excluded. Seeking justice for all involved in academic publishing was evident in all commentaries in forms such as:

• Critiquing traditional forms of publishing, as well as viewing all forms of publishing as part of a deeply flawed system;

[Academic publishing] is recognised and experienced in most quarters of the world as a vastly elitist, exclusionary and inaccessible industry.

I find it surprising that many academics choose - - - to hold to the myth that peer review is a fair, appropriate method of assessment – or even that the bias may be “unconscious” in its favour of Eurocentric sources, references and contexts worthy to be studied.

• Identifying systematic bias in the system and the power dynamics inherent to the system that has perpetuated this bias;

… most belong to publishing multinationals with interests in the education sector: Elsevier, Springer, Routledge, Taylor & … and use only the English language for publishing and are almost all based in Great Britain, the United States. With a very little presence of Ibero-American magazines, I wonder natural selection? Or big editorial interests?

• Recognising multiple ways in which many are excluded from a highly hierarchical publication system;

What goes unquestioned (but is inevitably manifested and experienced) are the power dynamics of whom gets to review; their academic agenda within their own discipline (and in inter/trans/multidisciplinary areas, the porousness of their bias/preference); and their understanding of academic practices and knowledges around the world.

• Reflecting on the publication process in relation to the SDGs;

In the context of our international mission of sustainability it would be difficult to find commonalities between the quality assurance processes involved here and the ethos of the sustainable development goals, with their focus on social justice and inclusion.

• Celebrating the advantages of open access in avoiding the exclusion of academics;

There are some great positives in this open-access online system. Many of those who were excluded from the traditional process are now included and valued.

A very high number of papers are published and it is also very easy to suggest and do a Special Issue. Therefore, also less dominant/hegemonic topics can get a space.

No doubt it works well for many academics who were previously marginalised, as long as they, their institution or their funders can afford to pay for publication, or by virtue of contributing as editors or peer reviewers or both have earned credits to pay for publication. This highly inclusive system has much in common with modern day notions of social justice.

• Noting that notions of exclusion, and matters of cost in particular, might divide researchers as those who are employed by universities that are able to pay from those who are not;

[A]uthors have to pay a lot for getting their paper published. For researchers of the big and well-funded research universities this is not a problem, but it may be a problem especially for emerging researchers and researchers from the Global South.

[W]hat is being done for those researchers who again are most at risk of exclusion from participating in this discussion? How are publishing processes equitious and just, beyond the rhetoric?

• Ensuring access to academic endeavours for all. Commentaries emphasised the necessity of making research results available for groups other than researchers employed at universities, such as school teachers in training who need to become capable of assessing the risks of current trends and working for a sustainable future from a holistic and reflective perspective that questions dominant but possibly unfair and unsustainable socioeconomic models;

This includes - - - non-affluent institutions, independent researchers, academics who are displaced or in exile, in addition to the many NGOs, politicians, members of the public across the world who may benefit from such knowledge.

Especially for ESD and our approach of transdisciplinarity it is important that academic papers can be read also by people from outside of academia and also from university members with limited access to non-open assess publications (smaller universities, universities from the Global South …).

• And discussing publishing as a matter of freedom;

I have seen violence to the extreme with academics in exile – in addition to being in danger from the state, authoritarian surveillance, and effectively no protections from what our sector holds up as academic freedom, they spend considerable time trying to find ways to publish research about their communities, conflict, displacement or their political activism, when they have no means to pay for its place.

The reflective commentaries also addressed power-related issues, what the role of higher education is in the ESD process, and how publishing relates to higher educational policy. In seeking justice in the future, commentaries suggested that the system is not fair as it is now, and that it seems to foster obedient scholars to ensure that this unfairness perpetuates;

Hegemony reproduces itself in this production process, which is one where we shape the rungs of the hierarchical ladder.

Publication is meant to fulfil the function of dissemination of contributions to knowledge to the (international) public which may use or learn from them. While all around the world there have been challenges to the hidden nature of this stultified and often misrepresentative mythology of the university – not least the calls to decolonise, degender or localise - the myth maintains and is maintained by the publication industry.

Undoubtedly, the reflective commentaries found biases that characterise research publication and that require attention and change; in general but perhaps particularly with respect to higher education’s contribution to and commitment to achievement of the SDGs.

Creating Opportunities for Change

Reflective commentaries showed a high level of agreement on the principals of open access publication, including its central role in creating opportunities for change in the practice and praxis of academic publishing. Commentaries also agreed on the necessity of change, in particular in seeking social justice, commensurate with the guiding principles of the SDGs. The reflections showed strong reservations about the quality assurance processes currently adopted by both traditional and open access publication models and an explicit emphasis that these processes have not been created with current SDGs in mind. Commentaries suggested that although open access processes are to be preferred, their appropriateness cannot be taken at face value, as exclusionary tendencies may be simply hidden in rhetoric about representation, inclusivity, and equity; highlighting the tension between social justice imperatives and current quality assurance goals. Changes will be necessary at both operational and philosophical levels.

To my mind, there are technical concerns and larger philosophical underpinning concerns that require surfacing for a discussion on the representation, archiving and dissemination of knowledge on sustainability through academic publications. The technical and philosophical are interwoven in some places, but in other spaces the threads are mismatched, threadbare, unravelling – and in other instances, not part of the same weft or weave. I use the allegory of handicrafts because there is a plasticity to journal publications – they are consciously shaped, edited, reduced and finally objectified with a DOI.

Our publication processes surely need to prioritise certain research questions over others and perhaps modify our expectations of quality to the challenges of the task at hand; a process requiring discourse within a community of practice.

Journals also can become much like aesthetic objects – there is an art to them, a jouissance in their writing and reading. … I thought it prudent to remind the reader that there are mechanics, levers, drivers and an industry behind these processes, in addition to aesthetics of quality which inherently relate to the worth of the product.

Central to the open access movement and consistent with the global drive for the SDGs, is a hope that focused collective energy around specific global challenges, that impact the most vulnerable in systemic ways, will lead to actual change in those target areas. In particular, the shift from traditional publishing to open access is often framed as the need to: centre the representation of knowledge domains of, from and about the most vulnerable; expand the diversity of methodologies and knowledge systems which may be brought to bear to construct such knowledge domains; and support a reckoning with the geopolitics inherent to notions of quality.

“…international” journals are rarely held to account for the geographic representativity of their boards and whom is chosen to sit on them. Many of these journals are kept afloat through the academic labour of reviewers who will never see their countries located on the editorial board, and never be considered for inclusion because their countries and institutions have little branding appeal.

Also more marginal perspectives on research topics as well as research from marginalized countries (Global South) should have a voice and therefore opportunities to be published in well-known journals of the field.

I have started refusing to review for international journals that have no African editorial board members, for instance, but fear such individual attempts are weak. “Calling out” about the issue is a professional risk. When power is wrought by such editors, who to all extents and purposes are not accountable to anyone, there is a culture of fear of exclusion for speaking out about such issues.

Reflective commentaries emphasised that opportunities for change need to be created at a distributed level, relating to how academics are employed, encouraged, and rewarded. National and political policies, for example, may interact with publication design to create drivers that may result in consequences that are not always intended or indeed understood.

The university in my country is partly paid from the state according to how much the researchers publish. I have experiences how my colleagues publish more and more, and all joint texts are not a result of deep discussions. We are all more and more hurrying around in the corridors like rats in a running wheel. It is not only the quality of the articles that are in danger, but also the quality of the teaching and of the entire academic discussion.

However, academic publishing is something many authors are invested in so deeply that they might not recognise what is going on. Reflective commentaries also show that, internationally, authors might be complicit in perpetuating the hegemonic practices and asymmetrical power plays even in open access models. Of particular interest is the “hidden” role of the benefits to individuals from participating in knowledge production processes that perpetuate exclusionary practices;

Where academic peers are unaware of the hidden curriculum of research production in their context, and are being shaped by its rewards and punishments, they too begin to promote such academic practice across the globe often without cognisance of the impact to their discipline.

Power is wrought through publications, and as such it may not be in the interests of those whom the system privileges to change it. There seems little appetite to upset this status quo in a systemic way.

Finally, the commentaries agree that academics and academic discourse need to be at the heart of the changes that surely must be achieved.

I suspect that our inability as academics to hold an active discourse on the nature of quality in academic publication will continue to lay open the world of academic publishing to academic publishers, rather than to academics, and perpetuate the role of academics as mere pawns in the process.

How can we make sustainability research and publishing sustainable and come out of the Neo-liberal frameworks? Why not actively encourage the discussion on this topic? We are many feeling uncomfortable with the current situation.

Discussion

Our analysis of six reflective commentaries of those who negotiate editorial and review work for the publication of academic articles in the context of sustainability education, as CAE, raised strong critiques of the current conditions, philosophies, and practices of academic publication, both traditional and open access. Overall, our analysis identified deep concern about the quality assurance processes that underpin academic publication, to the extent that we worried about fitness for their intended purposes. In this section of the article, we discuss the ways in which our particular professional experiences relate to larger, often disparate, arguments in the wider circumstances of academic publication, initially with respect to traditional academic publication processes but then with a focus on open access publications. On the way, we give attention to seeking social and environmental justice in the context of the internationally agreed SDGs, to creating opportunities for change and to the conditions that maintain the current circumstances in which change must be forged.

Geopolitics and Global Hegemony

Of all of our concerns, that with most congruence to a large and increasing base in the academic literature relates to geographical, geopolitical and language-based inequalities, biases, and prejudices. Inequalities certainly exist at the level of research design and participation. For example, Diestro Fernández et al. (2017) highlight in their study on education journals, the limited presence of Ibero-American journals in the two most prestigious rankings, and that “the reasons for the great gap between Ibero-American and Anglo-Saxon journals are economic resources, language or access to large international databases, aspects so basic that they almost cut off any possibility of equalization” (241). Gradim and Morais (2016) point out with respect to language inequalities that “… this change arises not only as a result of the evaluations to which they are subjected, but also because the media that make up the databases mostly use English as the language of publication” (129). Roberts et al.’s (2020) analysis of 26,000 empirical articles published between 1974 and 2018 in top-tier cognitive, developmental, and social psychology journals suggests that “…systemic inequality exists within psychological research and that systemic changes are needed to ensure that psychological research benefits from diversity in editing, writing, and participation” (1295). An analysis of research on higher education for sustainable development concluded that this discourse is western-dominated (Barth and Rieckmann, 2016). Notably, however, publication in elite journals that focus on higher education studies (such as Higher Education and Studies in Higher Education), do include authors from many nations (Kwiek, 2020). Biases, as well as inequalities, likely exist at the level of access to publication. In a study conducted by two editors, for example, each from different global north-south regions, journal editors of 24 journals indicated not only their awareness of problems in the publication process, but identified “…the non-publication of papers authored in the global south” as a contentious issue (Richards and Wasserman, 2013, p. 823). One approach that goes some way towards the genesis of an explanatory model for these inequalities, biases and prejudices is to interpret the status of higher education internationally as a global hegemony, as attempted by Marginson (2008) who concluded that “…achieving a greater plurality in language, knowledge and research are crucial to a more plural world and more egalitarian political economy.” (87).

Peer Review and Transparency

Our concerns about peer-review processes are similarly and widely represented in the literature. Indeed, our own concerns could be described as somewhat pedestrian in comparison with analyses by others. Smith (an experienced editor in the medical sciences), as just one example of many analyses that reached the same conclusion, described peer review as “…a flawed process, full of easily identified defects with little evidence that it works” (Smith, 2006, p. 182). There have, of course, been attempts to improve peer review. Some journals in a wide range of disciplines do address the need for peer reviewers to be supported as they develop peer-review skills, and journals and disciplines do differ in their liking for double-blind or single-blind reviewing. A great deal of effort has focused on making peer review more open and so perhaps more able to contribute to improvements in the quality of research publications and in the quality of research in the field of enquiry. Stackman, for example, promotes developmental peer-review as a community of practice (Stackman, 2018). Peer review is also being trialled in pre-print formats, where peers can offer support for the development of a particular article in an open public forum, prior to the article being submitted for publication. There are, however, many obstacles to openness, particularly in the context of making judgements about the quality of research. While our own research was underway, in Covid-19 lockdown conditions, German media was critiquing a pre-print, submitted to an open academic forum by a virologist working on this virus, in a “targeted tabloid campaign” (Mathews, 2020). Clearly making peer review more open has additional consequences that may limit its acceptance within academic communities.

Academic Agency and Discursive Closure

Our reflective commentaries also emphasised our collective concerns about academic agency in the context of such powerful hegemonic determinants of the business of academic publication and academics’ intimate involvement with it and within it. Meriläinen et al. (2008) use Laclau and Mouffe’s political theory of discourse to explore geographical bias, but similar arguments can no doubt be made with respect to other exclusionary biases in academic publication, relating to gender and young researchers as examples. These authors argue that: power operates through academic publishing practices in ways that create “discursive closure” …“Some discourses have more opportunities to present themselves; they monopolize communications and they make use of strategies of control to ensure their eminence”(586); those most at danger from losing attention are seduced to marginalise themselves; everyday mundane actions are as powerful as structural limitations; and that as academics engage in these practices they reify and reinforce the status quo. Similarly, Obeng-Odoom (2019) develops the concept of a knowledge hierarchy that “reinforces the privileged status of knowledge produced in the north, while seeking to undermine the potential transformative power of southern knowledge” (211). Such analyses suggest that challenges to cultural hegemony require fundamental changes in academic conditions and academic behaviour to be successful; although there are arguments for agency even in challenging circumstances. Connell et al.(2018, p. 41), for example, describe the development of an “under-commons,” driven by academics in “peripheral but not powerless positions.” We add to these broader concerns our worry that although those who are most invested in the system at present have the most to lose by changing it, they are the best placed to do so in the short term. Those excluded on the basis of age, experience, gender, language, race, resources, and geopolitics currently have the least power to effect change. Authors for this paper include those operating from the Global South and North and have made a conscious decision not to exclude references in languages other than English, to support and extend the validity of our arguments.

Doubts About Open Access

Turning to open access and hopes that open access publishing processes would increase access to authors, reviewers and editors from marginalised contexts, whilst ensuring access to research outcomes for the common good, a broad and expanding literature cast doubts on these hopes, congruent with our own. Suarez and McGlynn (2017) and Misra et al. (2020) draw attention to unreasonable costs of open access publication to those who can least afford it. Some analyses are focused on differentiating between different models of open access, such as those that have page charges and those that do not, and linking analyses with other means to distinguish between journals, such as impact factors. van Vlokhoven (2019), for example, develops an argument that moving to a system in which all journals are APC-based open access (APC article processing charge) will cause journals to accept lower quality articles and lead to an overall decline in journal quality and diffusion of knowledge. Teixeira da Silva et al. (2019) comment on open access mega-journals, identifying large peer-review and editorial boards, post-publication peer review (as well as pre-publication peer review) and article selection processes that focus on “technical or scientific soundness” rather than novelty, importance and interest “…and is thus perceived to be less rigorous by readers and authors.” (425) as characterising these journals. Their analysis in particular questions whether the peer review system is robust enough to accommodate for effective post-publication peer review. Similarly, Romesburg (2016) questions the quality of peer review in open access journals and wonders “…if open access publishing threatens to pollute science with false findings.” (1145). It is important to note that these analyses, while powerful, are necessarily subjective, based as they are on measures of quality that are themselves often idiosyncratic, discipline specific and essentially “academic.” Even so, Vrieze (2018) describes how some open access journal editors resigned after alleged pressure to publish mediocre papers. Clearly their concerns were also of an academic nature and so relevant to our academic enquiry. Implicit within these analyses are a wide range of suggestions to improve the quality of peer review, and editing; by shunning open access journals in general, by making all academic work free to publish and free to access, or perhaps by nationalising the academic publication industry as suggested by Mathews (2018) and thereby removing cost as either a structural limitation or as an incentive to quality reduction. In this sense, as Luchilo (2019) points out, “the movement towards expanding open access is very vigorous. Its strength lies in the variety and breadth of initiatives. On the government side, the European Union definition that by 2020 all articles published in Europe from projects with public funding must be in open access is a very strong signal, although difficult to comply with” (UNIÓN EUROPEA, 2018). Other initiatives in the same direction are being taken in other regions – for example, an agreement between MIT and the Royal Society of Chemistry (Fay, 2018).

Alternative Ways to Monitor, Judge and Assure the Quality of Publications

Our reflections interrogated the nature of quality in the context of academic publication and inevitably lead us to seek alternative ways to monitor, judge, and assure the quality of publications, particularly in the new world of online open access publication where the absence of limits on the number of pages that can be published creates a strong incentive to diversify measures of quality. Current disquiet about: the dominance of impact factors in journal ranking processes, as applied to individual papers (San Francisco Declaration on Research Assessment¹); academic decision-making around where to publish (Niles et al., 2020); and on overuse or misuse of the h-index (Kreiner, 2016); may provide additional support for an exploration of alternatives now. Some aspects of publication quality, such as verifying the information provided by publishers and addressing the menace of predatory journals, can be undertaken by certification, for example by the Directory of Open Access Journals (DOAJ) (Borchert and Boczar, 2016). Yet, such measures may not address issues associated with insufficiently knowledgeable, or critical, authors, editors, and peer reviewers; or with bias in our quality assurance processes that escape detection when reputation and citation dominate our understanding of quality. We argue that value lies not in shunning traditional or open access publishing but rather in developing quality indicators that are fit for purpose and that will ensure biased or otherwise substandard research does not end up in knowledge streams.

Quality in the Context of Higher Education’s Contribution to the SDGs

In some respects, a consideration of these diverse issues seen through the lenses provided by the SDGs may enable us to formulate an agreed approach to address general issues as well as those pertaining to academic publication that focus on sustainability and sustainability education. The SDGs anticipate that nations and institutions internationally will seek to; reduce inequalities (SDG10), build effective, accountable and inclusive institutions at all levels (SDG16), build partnerships to achieve the goals (SDG17) and achieve balance between socially- and environmentally oriented research compatible with attaining all 17 SDGs, rather than just some². At least at the level of aspiration these goals apply just as much to our traditional publication houses, new open access businesses and academic professional associations, as they do to governments and public institutions such as public higher education. Change may certainly occur at the level of individual journals (“What You Want Nature to Do Next,” Editorial, 2020, inviting readers to contribute to future publication policy; Ha, 2020, promoting affirmative action virtue theme collections) but perhaps the incentive of “quality measures” such as journal rankings may be necessary to initiate wider change. Imagine the situation if we ranked journals as we do now, based primarily on citation, but included other measures of quality such as; being free to read, being free to submit articles, free translation to many languages when submitted and availability in many languages when published, free availability of a panel of writing, ethical and statistical advisors to support all papers on initial submission, % of papers with at least one author (addressing perceived bias such as country, gender or age), % of papers peer-reviewed by at least one researcher (addressing perceived bias such as country, gender, or age) and considering all SDGs rather than just a few. In effect, such measures of quality address not only the outputs of research, but also the processes that contribute to research and they redefine publication quality as something beyond reputation. Quality indicators in parallel could also say more about the editors and peer reviewers used by journals, such as their personal h-indices, beyond the expectation that they have published in related areas. Something similar is happening at the institutional level. The Times Higher Education Impact Rankings³ run alongside more conventional rankings to address impact with respect to the SDGs and emphasise SDG17. One quality measure, for example, is the proportion of academic publications with at least one co-author from a developing country. We do not underestimate the difficulties in addressing these matters. The efficacy of institutional rankings for equity, for example, has been explored (in the context of Australian Higher Education Institutions) by Pitman et al. (2020). These researchers highlight the subjective nature of both higher education equity and higher education ranking systems and recommend that “… those constructing the methodology… move from a position of ‘valuing what one can measure’… to… ‘measuring what one values’” (261–262). Whether or not the SDGs will truly become an effective interpretative framework to drive substantive change remains to be seen. The scale of change needed to achieve the goals was recently highlighted, for example for SDG1, by Aston (2020), in describing the scale of poverty even in one highly developed country.

Limitations and Hegemonic Stability

It also remains to be seen how seriously institutions and academics take these alternative metrics and whether or not they would catalyse any substantial change, but there is a strong case for seeking broadly based academic change in the future, for example in the context of open science (Vicente-Saez et al., 2020) and particularly with respect to the fit between academic publication and the interests of the communities to which they relate (Gelmon et al., 2013; Mathies et al., 2020). Our own analysis must also be seen as what it is, based on the experiences of just six academics and so limited in many respects. There is, however, expanding interest in collaborative enquiry of individual experience as research. As described by Hernandez et al.(2017, p. 253) “CAE holds promise and potential as a critical method for fostering global collaboration that disrupts hegemonic theorizing. By listening to the voices of scholars from different parts of the world, published CAE projects could shed much-needed light on the necessity for contextualizing in theory-building.” As peer reviewers and editors from several different nations and contexts, and with different life experiences, we do wonder if current articulations of the quality of research and publications may be well suited to designing space rockets to take humans to another planet but less applicable towards achieving the SDGs to ensure sustainability on our own planet. As these same academics, but accepting our own differing personal emphases within the diverse issues addressed here, we must accept that many academics will be fundamentally opposed to any extension of “neoliberal” rankings (however well indicated as for the common good) for these purposes or to affirmative action to address inequalities of the past. Continuing inability by academics to reach consensus on such issues serves to maintain the hegemonic processes that create, profit from and celebrate the circumstances that academics exist in, and support.

Conclusion

A collaborative autoethnographic enquiry of the experiences of six academics, all practiced reviewers and editors, critiques the quality of aspects of academic publication including the quality assurance processes that underpin them, in general and with respect to higher education’s contribution to the SDGs. Our concerns address geopolitical inequalities of academic access as authors, peer reviewers, and editors, academic dependence on peer review and our collective inability to overcome its limitations, highly diverse opportunities for academic agency, and our hopes for open access and our disappointment in its current manifestations. We considered alternative ways to monitor, judge, and assure the quality of academic publications. We accepted that differences in our own emphases reflected the diversity of academic opinion and agency that itself serves to stabilise the hegemony that has created the circumstances in which only some academics thrive, and that will surely limit higher education’s contributions to the SDGs.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author Contributions

All authors worked collaboratively to explore our experiences. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Footnotes

1. ^ https://sfdora.org/read
2. ^ https://sdgs.un.org/goals
3. ^ https://www.timeshighereducation.com/rankings/impact/2020/

References

Aston, P. (2020). Statement on Visit to the United Kingdom, by Professor Philip Alston, United Nations Special Rapporteur on Extreme Poverty and Human Rights. Geneva: OHCHR.

Google Scholar

Barth, M., and Rieckmann, M. (2016). “State of the art in research on higher education for sustainable development,” in Routledge Handbook of Higher Education for Sustainable Development, eds M. Barth, G. Michelsen, M. Rieckmann, and I. Thomas (Abingdon: Routledge). 100–113.

Google Scholar

Borchert, C. A., and Boczar, J. (2016). Numeracy and evaluating quality in open access journals. Numeracy 9:1. doi: 10.5038/1936-4660.9.2.1

CrossRef Full Text | Google Scholar

Chang, H., Ngunjiri, F. W., and Hernandez, K. A. C. (2013). Collaborative Autoethnography. New York, NY: Routledge.

Google Scholar

Connell, R., Pearse, R., Collyer, F., Maia, J. M., and Morrell, R. (2018). Negotiating with the north: how southern-tier intellectual workers deal with the global economy of knowledge. Sociol. Rev. 66, 41–57. doi: 10.1177/0038026117705038

CrossRef Full Text | Google Scholar

Diestro Fernández, A., Ruiz Corbella, M., and Galán, A. (2017). Calidad editorial y científica en las revistas de educación. Tendencias y oportunidades en el contexto 2.0. Revi. de Investig. Educ. 35, 235–250. doi: 10.6018/rie.35.1.244761

CrossRef Full Text | Google Scholar

Editorial (2020). What you want nature to do next. Nature 577, 598–598. doi: 10.1038/d41586-020-00227-w

PubMed Abstract | CrossRef Full Text | Google Scholar

Fay, B. (2018). MIT and Royal Society of Chemistry Sign First North American “Read and Publish” Agreement for Scholarly Articles. Cambridge, MA: MIT Libraries.

Google Scholar

Garbett, D., and Thomas, L. (2020). Developing inter-collegial friendships to sustain professional wellbeing in the academy. Teach. Teach. 26, 295–306. doi: 10.1080/13540602.2020.1832062

CrossRef Full Text | Google Scholar

Gelmon, S. B., Jordan, C., and Seifer, S. D. (2013). Community-engaged scholarship in the academy: an action agenda. Change Mag. High. Learn. 45, 58–66. doi: 10.1080/00091383.2013.806202

CrossRef Full Text | Google Scholar

Gradim, A., and Morais, R. (2016). Tendências atuais na publicação científica: o português como língua de ciência. Observatorio (OBS) 10, 119–134.

Google Scholar

Ha, L. (2020). Online posting, single-blind review, open access, and the future of refereed journals. J. Mass Commun. Q. 97, 5–12. doi: 10.1177/1077699020904383

CrossRef Full Text | Google Scholar

Hains-Wesson, R., and Young, K. (2017). A collaborative autoethnography study to inform the teaching of reflective practice in STEM. High. Educ. Res. Dev. 36, 297–310. doi: 10.1080/07294360.2016.1196653

CrossRef Full Text | Google Scholar

Hamilton, M. L., Smith, L., and Worthington, K. (2008). Fitting the methodology with the research: an exploration of narrative, self-study and auto-ethnography. Stud. Teach. Educ. 4, 17–28. doi: 10.1080/17425960801976321

CrossRef Full Text | Google Scholar

Hernandez, K.-A. C., Chang, H., and Ngunjiri, F. W. (2017). Collaborative autoethnography as multivocal, relational, and democratic research: opportunities, challenges, and aspirations. Autobiogr. Stud. 32, 251–254. doi: 10.1080/08989575.2017.1288892

CrossRef Full Text | Google Scholar

Herwald, H. (2018). Popular Science Rather Than Predatory Journals Curie 27th November. Available online at: https://www.tidningencurie.se/en/debatt/popular-science-rather-than-preditory-journals/ (accessed November 12, 2020).

Google Scholar

Kreiner, G. (2016). The slavery of the h-index—measuring the unmeasurable. Front. Hum. Neurosci. 10:556. doi: 10.3389/fnhum.2016.00556

PubMed Abstract | CrossRef Full Text | Google Scholar

Kwiek, M. (2020). The prestige economy of higher education journals: a quantitative approach. High. Educ. 81, 493–519. doi: 10.1007/s10734-020-00553-y

CrossRef Full Text | Google Scholar

LaBoskey, V. K. (2004). “The methodology of self-study and its theorrectical underpinnings,” in International Handbook of Self-Study of Teaching and Teacher Education Practices, eds J. J. Loughran, M. L. Hamilton, V. L. LaBoskey, and T. Russell (Dordrecht: Kluwer), 817–869.

Google Scholar

Lincoln, Y., and Guba, E. (1985). Naturalistic Inquiry. Newbury Park, CA: Sage.

Google Scholar

Luchilo, L. J. (2019). Revistas científicas: oligopolio y acceso abierto. Rev. CTS 14, 41–79.

Google Scholar

Macfarlane, B. (2019). The neoliberal academic: illustrating shifting academic norms in an age of hyper-performativity. Educ. Philos. Theory 53, 459–468. doi: 10.1080/00131857.2019.1684262

CrossRef Full Text | Google Scholar

Marginson, S. (2008). Towards a global university hegemony. Crit. Int. 39, 87–107.

Google Scholar

Marshall, M. N. (1996). Sampling for qualitative research. Fam. Pract. 13, 522–525.

Google Scholar

Mathews, D. (2018). Is It Time to Nationalise Academic Publishers? London: Times Higher Education.

Google Scholar

Mathews, D. (2020). German ‘Star Virologist’ Targeted in Tabloid Campaign. London: Times Higher Education.

Google Scholar

Mathies, C., Kivistö, J., and Birnbaum, M. (2020). Following the money? Performance-based funding and the changing publication patterns of Finnish academics. High. Educ. 79, 21–37. doi: 10.1007/s10734-019-00394-4

CrossRef Full Text | Google Scholar

Meriläinen, S., Tienari, J., Thomas, R., and Davies, A. (2008). Hegemonic academic practices: experiences of publishing from the periphery. Organization 15, 584–597. doi: 10.1177/1350508408091008

CrossRef Full Text | Google Scholar

Misra, D. P., Ravindran, V., Sharma, A., Wakhlu, A., Ahmed, S., Negi, V. S., et al. (2020). Letter to the editor: an indian perspective on universal open access publishing: think of the fire before venturing out of the frying pan! J. Korean Med. Sci. 35:e85. doi: 10.3346/jkms.2020.35.e85

PubMed Abstract | CrossRef Full Text | Google Scholar

Niles, M. T., Schimanski, L. A., McKiernan, E. C., and Alperin, J. P. (2020). Why we publish where we do: faculty publishing values and their relationship to review, promotion and tenure expectations. PLoS One 15:e0228914. doi: 10.1371/journal.pone.0228914

PubMed Abstract | CrossRef Full Text | Google Scholar

Obeng-Odoom, F. (2019). The intellectual marginalisation of Africa. Afr. Identities 17, 211–224. doi: 10.1080/14725843.2019.1667223

CrossRef Full Text | Google Scholar

Pitman, T., Edwards, D., Zhang, L.-C., Koshy, P., and McMillan, J. (2020). Constructing a ranking of higher education institutions based on equity: is it possible or desirable? High. Educ. 80, 605–624. doi: 10.1007/s10734-019-00487-0

CrossRef Full Text | Google Scholar

Richards, I., and Wasserman, H. (2013). The heart of the matter: journal editors and journals. Journalism 14, 823–836. doi: 10.1177/1464884913493062

CrossRef Full Text | Google Scholar

Roberts, S. O., Bareket-Shavit, C., Dollins, F. A., Goldie, P. D., and Mortenson, E. (2020). Racial inequality in psychological research: trends of the past and recommendations for the future. Perspect. Psychol. Sci. 15, 1295–1309. doi: 10.1177/1745691620927709

PubMed Abstract | CrossRef Full Text | Google Scholar

Romesburg, H. (2016). How publishing in open access journals threatens science and what we can do about it. J. Wildl. Manag. 80, 1145–1151.

Google Scholar

Shephard, K., Wolff, L. A., Rieckmann, M., Christie, E., and Lysgaard, J. A. (2019). A conversation on academic publishing and academic research and their possible interactions with higher education’s contribution to education for sustainability,” in Discussion Paper at European Educational Research Association Conference 2-6 September 2019 (Berlin: University of Hamburg).

Google Scholar

Smith, R. (2006). Peer review: a flawed process at the heart of science and journals. J. R. Soc. Med. 99, 178–182. doi: 10.1258/jrsm.99.4.178

PubMed Abstract | CrossRef Full Text | Google Scholar

Stackman, R. W. (2018). Away from abilene and toward a community of practice ecosystem: developmental peer reviews realized. J. Manag. Inq. 27, 169–175. doi: 10.1177/1056492617726274

CrossRef Full Text | Google Scholar

Suarez, A. V., and McGlynn, T. (2017). The Fallacy of Open Access. Washington, DC: The Chronicle Of Higher Education.

Google Scholar

Teixeira da Silva, J. A., Tsigaris, P., and Al-Khatib, A. (2019). Open access mega-journals: quality, economics and post-publication peer review infrastructure. Publ. Res. Q. 35, 418–435. doi: 10.1007/s12109-019-09654-8

CrossRef Full Text | Google Scholar

Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. Am. J. Eval. 27, 237–246. doi: 10.1177/1098214005283748

CrossRef Full Text | Google Scholar

UNIÓN EUROPEA (2018). Recomendación (UE) 2018/790 de la Comisión de 25 de Abril de 2018 Relativa al Acceso a la Información Científica y a su Preservación. Brussels: UNIÓN EUROPEA.

Google Scholar

van Vlokhoven, H. (2019). The effect of open access on research quality. J. Informetr. 13, 751–756. doi: 10.1016/j.joi.2019.04.001

CrossRef Full Text | Google Scholar

Vicente-Saez, R., Gustafsson, R., and Van den Brande, L. (2020). The dawn of an open exploration era: emergent principles and practices of open science and innovation of university research teams in a digital world. Technol. Forecast. Soc. Change 156:120037. doi: 10.1016/j.techfore.2020.120037

CrossRef Full Text | Google Scholar

Vrieze, J. (2018). Open-access journal editors resign after alleged pressure to publish mediocre papers. Science doi: 10.1126/science.aav3129 [Epub ahead of print].

CrossRef Full Text | Google Scholar