Thematic analysis of stakeholder perceptions for co-creative healthcare XR resource design and development; traversing a minefield of opportunities

Introduction: The expansive curricular volume of healthcare education makes a necessity the incorporation of innovative methods and immersive media in it. The core challenge in such approaches is the timely development of relevant immersive content such as Virtual, Augmented or Mixed Reality (VR/AR/MR) resources for healthcare topics. There is currently significant interest in the use of co-creative methods for streamlining immersive content development.

Methods: A core research pursuit in this translational research field is the formulation of evidence-based, optimized workflows that streamline immersive content creation allowing for rapid expansion of innovative educational approaches in healthcare curricula. The purpose of this paper is to aggregate the perceptions of healthcare technologists and educators who participated in a series of co-creation sessions in order to elicit their best practice insights for design and development of XR educational resources using co-creative methods.

Results: According to our thematic analysis, findings of the qualitative study demonstrated that a rigorous organizational approach is required to maintain a constructive exchange of information and to keep the design process alive for both content and technical experts. In addition, rapid prototype and display of co-created features can empower their contributions and help them design more efficiently.

Discussion: Co-creative content production can benefit from adaption of existing frameworks and lightweight authoring environments that can facilitate generalized XR content development use cases.

1 Introduction

In the 20th century, the quantity of instructional material that healthcare practitioners must master has skyrocketed. Critical medical knowledge necessitates an adequate theoretical foundation in addition to implicit knowledge and experience (1). To this aim, the need for innovative educational approaches and practices is required. Immersive experiential technology can support medical education to flourish in this environment. Through their enhanced engagement, virtual patients, chatbots, and Virtual, Augmented, or Mixed Reality (VR/AR/MR) applications have a significant impact on the affective and educational condition of healthcare students (1–4).

Key obstacles in providing materials and innovative educational activities are time and cost overheads. A proposed solution for timely creation of relevant and valid immersive content is the implementation of participatory design methods. Co-creation as a healthcare educational digital content design methodology, is only lately gaining traction. With some conceptual works such as the ASPIRE framework (5) initial observations indicate that co-creation is a feasible method for streamlining XR content creation and reducing overheads (6). Participatory design methods have also been marked rather early on as plausible knowledge transfer methodologies (7–9). This work aims, through a thematic analysis of stakeholder interviews, to elicit optimal processes and workflows for co-creative design of XR resources. The end goal of this process is the reduction of development time for healthcare XR resources, the minimization of technical capacities required by a healthcare technology unit and the smooth transfer of educator requirements to technical specifications and implementations. To contextualize these aims, a brief discourse of the state of play in healthcare VR and participatory design methods follows.

1.1 Virtual reality in healthcare education

Extended Reality (XR), comprising AR, VR and MR, has enabled students to visualize abstract laws in physical ways (10–12). The sensory immediacy of these technologies results in an intuitive anchoring of essential information for the learner and facilitates paradigm building, which, in turn, enables learners to acquire robust and deep knowledge while minimizing the likelihood of establishing or maintaining conceptual errors (13).

The medical industry already possesses a body of immersive content. In 2019, the Royal College of Physicians acknowledged VR as a “…powerful educational tool for defined learning objectives…” (14) and listed multiple applications in medical education and surgery. In terms of resources, the global VR market was $3.10 billion in 2019 and is projected to reach $57.55 billion by 2027 (15, 16). This example demonstrates the possibilities for immersive content development in healthcare education. Nonetheless, this necessitates the comprehensive design and development of such resources in order to first test them and then implement them in practice. According to a previous study, the cost of planning and implementing VR training exercises for hospital-based healthcare practitioners was around $106,387.00 (17). Considering the reusability of VR resources, this was subsequently deemed a viable approach. However, the ongoing evolution of medical knowledge and training can quickly render VR resources unusable, hence diminishing their reusability. Therefore, it is essential to lower the implementation and design expenses of VR resources and to distribute the development burden of such resources. This is where participatory design methods and co-creation approaches can be utilized to address the issue of content generation and availability.

1.2 Co-design and co-creation process

Participatory Design (PD) is regarded as one of the most crucial prerequisites for an acceptable and effective design and development process of reusable e-resources, as it relies on the early and active engagement of all key stakeholders (18). PD techniques promote and strengthen collaboration among team members and permit developers and educators to evoke collective creativity throughout the design and production of educational resources (19). Particularly, PD approaches include regular meetings with end-users, clinical and technical specialists, in real time, or focus groups and surveys with the primary objective of eliciting a list of user and technical requirements. This can be accomplished through co-creation workshops that increase the early active participation of stakeholders as co-creators and the rapid prototyping capability of the technique (20). Specifically, stakeholders are asked to construct use case scenarios in the early stages of the project in order to elicit a well-defined set of needs. Consequently, a hybrid strategy incorporating design thinking approaches can be utilized. This method permits the investigation of the stakeholders’ true demands and the elicitation of design concepts that encourage valuable and immersive results (21). The working prototype of the produced resources is routinely displayed by the developers to the stakeholders during co-creation workshops in order to collect feedback, recalibrate, and proceed with the necessary modifications (22).

Co-creation provides dynamic support for the resource design and development process and is characterized by a stakeholder association that reflects mutual physical, mental, and business practices, as well as the ability to split the creative process (23, 24). This process fully engages stakeholders in the co-design process (25, 26) underlining the significance of mutuality, receptivity, and non-hierarchical relationships (27, 28) as a co-design characteristic. This level of insight into the co-creation process enables the identification of (7) information distribution as one of the most important co-creation effectiveness drivers. It has been suggested that working with diverse stakeholders in a co-design setting facilitates the production of more inventive concepts and ideas (8). During the co-design process, stakeholders, for whom the final resources are being built, are the experts and so have the opportunity to contribute to the development of information, ideas, and concepts (9). Throughout the whole design and development process, they are able to provide the required tools for the expression and ideation of their requirements, as well as make crucial judgments (9). Co-design is employed as a technique for collaborative design, with key stakeholders collaborating concurrently in the co-creation process. Co-creation is predicated on the concept that the participation of end users is crucial to the creative process, bringing together each other's perspectives, beliefs, wants, and preferences and jointly producing solutions (29).

The adoption of co-creative methods in the production of immersive content has received considerable attention (30). This methodology's central concept is the flexible scheduling of combined educator-technologist activity. This can be accomplished through Scrum pushes encouraged by the Agile development paradigm (31), utilizing semantic back-ends in pervasive game development platforms. However, the effective deployment of co-creative methodologies demands the precise specification of the objectives and goals of the immersive content, i.e., Agile requirements elicitation. In addition, the co-creation and co-design workshops for XR healthcare materials must be refocused in light of contemporary research in the relevant sector.

Having conducted a series of several participatory workshops between technologists and healthcare education experts for developing XR resources for the ENTICE project (https://entice.eu/), this qualitative study aims to outline what works and what doesn't in participatory methods during the development of XR healthcare resources. Specifically, in that context, this work has the overall aim of exploring the core needs for successful co-design of XR resources for medical teaching. To achieve this overall aim we have set out to explore the following study objectives: (1) What are the core concerns of XR resource co-design teams for medical teaching? (2) What do medical education and technology stakeholders consider as key enablers for supporting co-design of medical XR resources and episodes?

2 Methodology

2.1 Co-creation methodology and participants

This study included eight co-creation participants and active contributors from the ENTICE resource co-design team. Academic and technological backgrounds, as well as levels of co-creation experience, ranged from highly experienced co-creative technologists and researchers to academics and coders with little prior experience. The backgrounds of the participants in this study reflect the evolution of the XR immersive educational materials and their interdisciplinary nature to some extent. Two (2) learning technologists (software engineers), one (1) technical manager, two (2) senior medical educators, and three (3) teaching assistants participated in the interviews. All participants were core members of the XR team from the early stages of the project up until the final developments and were actively participating in the co-creation sessions.

It must be noted that co-creation sessions took place both during and after the COVID-19 pandemic. As such some were conducted online through teleconferencing software and others were face to face meetings. Six online sessions took place during the pandemic and over 10 took place face-to-face after the restrictions for the pandemic were lifted. Digital brainstorming boards, specifically, miro (miro.com) was used for allowing the depiction of participants’ ideas. Each co-creation session lasted between 90 and 180 min. In several of these sessions demo builds of the resources in desktop or VR form were presented and the team brainstormed on them. All sessions were video recorded and these recordings were the source of the XR resources’ design documents. Regarding the interviews, a member of the research team personally interviewed each participant. A problem centered interview approach was followed to touch on a variety of aspects for the co-creation process.

2.2 Design and procedure; problem-centered interviews with developers and educators

To get the necessary information, problem-centered interviews, a technique commonly utilized in qualitative research, were conducted. 1 interviewer and 1 analyst (Author #2 and Author #1 respectively) formed the thematic analysis team. The most useful features of the interviews were their ability to illuminate the participants’ subjective experiences, viewpoints, and impressions of the educational resource design and development process. Problem-centered interviews are characterized by a conversational process that allows interviewees to articulate their thoughts and subjective experiences. This interviewing approach reflected the key components of qualitative research, such as openness, adaptability, and orientation. In accordance with these ideas, the interview guide was developed as an organized discourse rather than a hard questionnaire, with the goal of eliciting spontaneous comment on all main aspects and viewpoints of the co-creation process. Participants were specifically requested to express views about their co-creation experience that were not necessarily addressed by the interviewer's inquiries. However, in order to establish and preserve a structured conversational flow, spontaneous follow-up inquiries and brief comments were also provided. The interview guide, as previously stated (9), was organized around five major subject categories. It ensured that the participants’ collected experiences were comparable and complete.

The interviews were recorded and transcribed. Each member of the thematic analysis team created their own initial thematic analysis and then a common discussion merged these results into a common thematic list. Upon preliminary analysis of the transcripts several different themes diverging from the interview's prescribed thematic areas emerged. In light of that fact we pivoted to an ab-initio thematic analysis to explore these new emerging themes. Thematic analysis is a staple method of extracting meaning and organizing feedback from qualitative data. In a rather seminal paper describing the method in 1985, for healthcare Benner (32) cites Lazarus (33), when describing the main challenge of thematic analysis as being “the task (..) to uncover the meanings in everyday practice in such a way that they are not destroyed, distorted, decontextualized, trivialized, or sentimentalized”. Aronson (34) describes the specific methodology we followed in detail at a later work. Summarily, thematic analysis steps are: (a) Collect and transcribe data, (b) Identify all data that relate to already identified patterns, (c) Combine and catalogue related patterns into sub-themes, (d) Follow through on sub-themes, (e) Build arguments for choosing themes and sub-themes according to the literature and finally, (f) Formulate a story line that stands with merit and facilitates deeper comprehension of the process and its underlying factors. Following this approach other authors have refined and presented a more recent perspective on thematic analysis, Braun, and Clarke (35) report that thematic analysis is a method for identifying, analysing, and reporting themes within qualitative data. Themes are identified by the researchers as important about the data in relation to the research question and represent some level of patterned response or meaning within the data set. In their work Braun and Clarke, (35), essentially present the same process as Arronson. However the newer by Braun and Clarke focuses more on the process itself and offers a more contemporary view and a more streamlined workflow. Specifically, thematic analyses is defined as six phases: (a) Familiarizing yourself with your data, (b) Generating initial codes, (c) Searching for themes, (d) Reviewing themes, (e) Defining and naming themes, (f) Producing the report. This workflow was also followed in the present work. In the present study, verbal data retrieved from the interviews were initially transcribed into written form to construct a thematic landscape. Thereupon, the process involved the production and collation of initial codes annotated in the transcriptions from the data to organise them into meaningful groups, resulting in a list of different codes identified across the data set. This process allows for the codes to be grouped within different themes, which will later on be subdivided into subthemes. After reviewing the collated themes, data within themes appeared to be cohere together meaningfully, while clear and identifiable distinctions between themes were eventually conducted. Finally, themes were defined and refined by the research team, resulting in the final list of themes and subthemes.

3 Results

We have chosen to present our thematic analysis in a series of tables. Each table describes the identification of each major theme and its subthemes from participator feedback. In each table the first row presents a series of characteristic quotes that the researcher identified as dominant from the steps (a) to (d) of the process defined in the thematic analysis literature. These are not the only, verbatim available, quotes about a topic. They express, however, a significant portion of the participants’ feedback during the interview. In the next column we provide the researcher's rationale for their assignment of these quotes as reflections on a specific theme and subtheme, step (e) in the thematic analysis process. It must be noted that these assignments were based on the specific quotes, their contextualization in the text, as well as their purview on the whole interview of the participant. Next to it we present the themes and subthemes that were identified from these processes. We elected to present our data in tabular form in order to provide a more coherent and concise description of the themes that we identified throughout our analysis. As such, while a brief analysis write-up is presented in tabular form, we found that the diversity of the themes of our case would create an encyclopedic, disjointed analysis if presented in narrative form. It must be noted that thematic definitions in this analysis are highly contextualized. For example some subthemes are synonyms but differing in the context of overall theme. “Tools and media” is present as a subtheme both in the “Organizational rigor” theme and in the “Authentic communication” theme. In the first theme “Tools and media” refers to planning and project management tools while in the second it refers to teleconferencing tools. Such contextualization is important both for demonstrating the breadth of every identified major theme, as well as demonstrating the role of “enabling” subthemes in many identified major themes. Also, some major themes like can pivot as subthemes in a different context. That is the case of the major theme “Organizational Rigor” which contain as a subtheme the “Facilitating authentic communication” subtheme. It is clear that this subtheme is, as a term, similar to the major theme “Authentic communication”, but the real meaning of the two items is different in context. In the first case “facilitating authentic communication” refers to the organizational provisions needed to support the subtheme, while in the second case “Authentic communication” explores user feedback on a totally different conceptual pivot, that is the facets and needs of communication in co-creation process. In the context of these notes we present the following tables of specific major themes as they were identified in our analysis.

Cross-disciplinary collaboration, its limitations and benefits was a major theme that was discussed by all interviewees. Its subthemes and relevant quotes from the interviews are summarized in Table 1.

Table 1

Table 1. Cross-disciplinary collaboration theme and subthemes as expressed by the participants.

Organizational rigor, the needs for appropriate collaborative storyboarding tools and well facilitate workflows, was also extensively mentioned in the interviews. Specific sub-themes with relevant quotes are summarized in Table 2.

Table 2

Table 2. Organization rigor theme and subthemes as expressed by the participants.

Authentic communication, the need for it, necessary pre-requisites and limitations stemming from inherent cross-disciplinary barriers were touched and expanded upon as summarized in Table 3.

Table 3

Table 3. Authentic communication theme and subthemes as expressed by the participants.

Authentic stakeholder engagement and its prerequisites were discussed and are presented in Table 4.

Table 4

Table 4. Authentic stakeholder engagement theme and subthemes as expressed by the participants.

Finally, educational rigor and the role of content co-creation as an educational method was brought up and is summarized in Table 5.

Table 5

Table 5. Educational rigor theme and subthemes as expressed by the participants.

4 Discussion

This work pivoted off a previous work (6) that explored existing themes and pitfalls in digital content co-creation. Expanding the interviewee base, this time we attempted a new, bespoke, thematic analysis in order to explore new emerging themes that were only hinted upon in the previous work but were expanded upon in this study with more participants. Bringing together these themes and topics we conceptually summarize them in the mind map of Figure 1.

Figure 1

Figure 1. Mind-map of themes for educational XR digital content co-creation as they emerged from participants’ interviews.

The key benefit from the co-creative approach in XR digital content design is the facilitation and expansion of cross-disciplinary collaboration. A key gap in digital content creation is that between the technical implementation team and the content experts. Alleviating this gap through timely and contextualized cross-pollination between these essential stakeholders is the main benefit identified by the participants of this study. For that, they identified the three (3) needs: (a) demonstrable clarity of purpose at all times of the co-creation process; (b) timely involvement of all stakeholders and (c) continuous iterative feedback. In essence, the participants emphasized the need for continuous, consistent involvement of the content experts with incremental, ever evolving iterations of the developing resources.

A similar outcome emerged from the themes that related to the needs for effective and efficient co-creation of XR digital content. Authentic stakeholder engagement emphasized also the need for a “clean slate” of technical capacities and the need for a “product” after every discussion to iterate upon. Also, following the previous needs, organizational rigor thematically emphasized on providing a very robust toolset and supporting media to enable the content experts in conveying their ideas into implementable features for the technical team.

In itself, the theme of authentic communication was expanded upon by several of the participants. To enable it, several participants emphasized on authentic presence, through face to face, instead of remote collaboration. Openness and appropriate, well defined facilitation was again brought up as key enablers for authenticity in communication. Compounding on these, participants mentioned the need for ample time of participation and balanced composition of the co-creation participants’ groups in order to not degenerate the session into either a developer sprint or to a lesson planning session.

An interesting note of one participant was the educational potential of the co-creation approach. As they observed, during the participatory design session, exchange of knowledge took place and some subject matter knowledge was transferred to members of the technical team. It is apparent that the educational rigor of co-creative methodologies is undocumented and such an anecdotal mention is of no real, evidence based, consequence. However the potential of co-creation methods for cross-pollinating knowledge transfer is documented (7–9).

From this thematic analysis two key requirements emerge for successful and efficient XR content co-creation in healthcare education.

1. Rigorous organizational process. Co-creation sessions present a warm and loose discussion based front. However, in order to be productive, facilitators need to have very structured discussion plan and collaboration tools (e.g., whiteboards, sticky notes and other enablers) to maintain a constructive exchange of information and keep the design process active for both the content and the technical experts.

2. Rapid prototyping and demonstration of co-created features. The key enabler for a successful co-creation session is a demo of the features that have been previously discussed. A common truism amongst all participants in our XR content co-creation sessions was «if you don't have a new prototype, let's not meet”. Through hands on experience the content experts can further enable their visions and understand, tacitly, technical limitations. In an ideal situation, a content authoring system that could implement in real-time rough ideas and storyboards might enable even shorter development cycles.

From these results it became clear that the themes of cross-discipline collaboration and organizational rigor have been emphasized. This is unsurprising, since these are the core hurdle in resource production and in bringing the educators’ needs exactly into the technical development pipeline. While rigorous topic presentation, meeting presence and basic communication strategies are always important, the core enabler for effective co-creation is authentic collaboration. In that context, on one hand, cross-discipline communication is the key enabler for this collaboration. On the other hand, organizational rigor provides the real world facilitation that allows these diverse groups to communicate effortlessly.

5 Conclusions

In earlier research (30), four crucial steps to the co-creative process in XR resources were discovered. Planning and preparation, actual co-creation, technical facilitation and training, and rapid prototyping comprise these steps. The “virtuous cycle” that enables quick deployment of XR resources and encourages simplicity of design and development across themes or even educational institutions uses this method to create an ever-improving fresh jumping point of feedback. Participatory design has therefore evolved as a means of democratizing digital citizenship, made even more important in light of the current pandemic (36). Democratization of XR content development in healthcare education may be possible by combining these procedures with Agile approaches. This qualitative methods study, aimed to cover two needs. First, it provided a set of practical considerations for organizing effective and efficient XR content co-creation sessions. Second, it points to the steps required for facilitating easier participatory creation for XR healthcare educational content.

To put this work in perspective, it is important, beyond its results to outline the core limitations that we identified and aim to alleviate in future research. The first and more important limitation is that of scale. This was a thematic analysis study that was conducted with only one team. That was a diverse multinational and experience team, however the fact remains that a thematic analysis on a cluster of such teams would hold further merit and is something that we are aiming to pursue in future work. Secondly, the educational scope of the team was medical teaching. All educators were medical doctors and anatomists working in medical schools. As such generalizing for the co-creative pipelines for other healthcare professions is difficult. Replication with more diverse groups of educators can help widening the scope of such results. Finally, in order to maintain research rigor we have used a standardized interview method and topic design. That provided a solid foundation on which to frame our results. However, it also limited the scope of the thematic exploration. A new bespoke framework for exploring specifics in XR healthcare participatory design would be able to go further in depth and in detail of the topic. However, even maintaining these limitations in mind, the results of this work do, in fact, remain relevant.

Experiential healthcare education is becoming an essential part of contemporary curricula. Basic curricular topics such as anatomy and advanced manual skills, like surgery, benefit from the immediacy and multisensory engagement of XR content (37). With the emerging mainstreaming of XR healthcare education, the problem of content availability is becoming increasingly exacerbated (38) Co-creative healthcare XR content design emerges as a viable solution for covering the curricular needs of the modern healthcare classroom (39). Moving from the cumbersome, months long, development cycle of commercial resources participatory design has demonstrated the capacity to produce high quality resources on par with any currently implemented technology resource in healthcare education (40). However, designing the resources is only part of the challenge. The implementation and curricular integration in a valid and pedagogically acceptable way are important steps that become technical or administrative barriers, respectively, for XR healthcare teaching. Experiences, as they emerge from systematic pursuit of such endeavors demonstrate that there are tools that can streamline the technical implementation, providing semantic annotation, discoverability and accessibility of basic assets for content creation (41). Additionally, rigorous processes for proving the efficacy of co-creative content creation have been documented, allowing for the administrative and curricular acceptance of XR resources developed more easily (42).

In that context, the results of this thematic analysis shown that in order to achieve this level of acceptance for co-created XR educational resources and episodes two are the crucial steps as they we have identified in this work. The first step is the development of a bespoke methodological framework for XR educational content creation. XR content is significantly more development intensive and less granular, in their interactivity context, than other resources like virtual scenarios and video-casts. Adapting existing frameworks, to cater bespokely to generalized use cases for XR content development, could further increase efficacy of co-creative content development.

Pairing with the previous methodological step, an authoring environment that can quickly prototype XR resource concepts appears a natural complement to the previous methodological enabler. Preliminary tools for such an environment have been proposed (43) but wider availability is still forthcoming.

Concluding, XR educational resource development through participatory methods emerges as a promising approach but only when certain pre-requisites are met. Further work, both in the methodological frameworks and in the technical enablers is needed in order to successfully traverse the minefield of opportunities, which is XR educational content co-creation in healthcare.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Research Ethics Committee of the Aristotle University of Thessaloniki. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

PA: Conceptualization, Project administration, Supervision, Writing – original draft. AV: Data curation, Formal Analysis, Investigation, Writing – original draft. JP: Methodology, Supervision, Validation, Writing – review & editing. CC: Funding acquisition, Methodology, Project administration, Resources, Writing – review & editing. VM: Writing – review & editing, Project administration. PB: Conceptualization, Funding acquisition, Supervision, Validation, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article.

This work has been partially funded by the “Evaluating Novel Tangible and Intangible Co-creative Experiential medical education” (ENTICE) Knowledge Alliances for higher education project, co-funded by the Erasmus+ Programme of the European Union (612444-EPP-1-2019-1-CY-EPPKA2-KA).

Conflict of interest

ViLabs (CY) LTD, Cyprus, is a commercial entity, that participated in an administrative and research capacity, in this work.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Antoniou PE, Arfaras G, Pandria N, Athanasiou A, Ntakakis G, Babatsikos E. Biosensor real-time affective analytics in virtual and mixed reality medical education serious games: cohort study. JMIR Serious Games. (2020) 8(3):e17823. doi: 10.2196/17823

PubMed Abstract | Crossref Full Text | Google Scholar

2. Antoniou P, Arfaras G, Pandria N, Ntakakis G, Bambatsikos E, Athanasiou A. Real-time affective measurements in medical education, using virtual and mixed reality. In: Frasson C, Bamidis P, Vlamos P, editors. International Conference on Brain Function Assessment in Learning. Cham: Springer (2020). p. 87–95.

Google Scholar

3. Dolianiti F, Tsoupouroglou I, Antoniou P, Konstantinidis S, Anastasiades S, Bamidis P. Chatbots in healthcare curricula: the case of a conversational virtual patient. In: Frasson C, Bamidis P, Vlamos P, editors. International Conference on Brain Function Assessment in Learning. Cham: Springer (2020). p. 137–47.

Google Scholar

4. Kyriakidou MR, Antoniou P, Arfaras G, Bamidis P. The role of medical error and the emotions it induces in learning–A study using virtual patients. In: Frasson C, Bamidis P, Vlamos P, editors. International Conference on Brain Function Assessment in Learning. Cham: Springer (2020). p. 3–12.

Google Scholar

5. Hassan N, Sellappans R, Chen PYV, Yap WH, Ayub E, Konstantinidis ST. Participatory approach in reusable learning object (RLO) development using ASPIRE framework: taylor’s university’s experience. In: Nair P, Keppell MJ, Lim CL, Mari T, Hassan N, editors. Transforming Curriculum through Teacher-Learner Partnerships. IGI Global (2021). p. 90–104.

Google Scholar

6. Varella A, Antoniou PE, Pickering JD, Chatzimallis C, Bamidis PD. Evaluating co-creative XR resource design and development; observations from the field. In: Dengel A, Bourguet M-L, Pedrosa D, Hutson J, Erenli K, Economou D, et al., editors. 2022 8th International Conference of the Immersive Learning Research Network (iLRN). Vienna: IEEE (2022). p. 1–5.

Google Scholar

7. Ranjan KR, Read S. Value co-creation: concept and measurement. J Acad Mark Sci. (2016) 44(3):290–315. doi: 10.1007/s11747-014-0397-2

Crossref Full Text | Google Scholar

8. Trischler J, Dietrich T, Rundle-Thiele S. Co-design: from expert-to user-driven ideas in public service design. Publ Manag Rev. (2019) 21(11):1595–619. doi: 10.1080/14719037.2019.1619810

Crossref Full Text | Google Scholar

9. Suhari M, Dressel M, Schuck-Zöller S. Challenges and best-practices of co-creation: a qualitative interview study in the field of climate services. Clim Serv. (2022) 25:100282. doi: 10.1016/j.cliser.2021.100282

Crossref Full Text | Google Scholar

10. Klopfer E, Squire K. Environmental detectives—the development of an augmented reality platform for environmental simulations. Educ Technol Res Dev. (2008) 56(2):203–28. doi: 10.1007/s11423-007-9037-6

Crossref Full Text | Google Scholar

11. Dunleavy M, Dede C, Mitchell R. Affordances and limitations of immersive participatory augmented reality simulations for teaching and learning. J Sci Educ Technol. (2009) 18(1):7–22. doi: 10.1007/s10956-008-9119-1

Crossref Full Text | Google Scholar

12. Wu HK, Lee SWY, Chang HY, Liang JC. Current status, opportunities and challenges of augmented reality in education. Comput Educ. (2013) 62:41–9. doi: 10.1016/j.compedu.2012.10.024

Crossref Full Text | Google Scholar

13. Olympiou G, Zacharia ZC. Blending physical and virtual manipulatives: an effort to improve students’ conceptual understanding through science laboratory experimentation. Sci Educ. (2012) 96(1):21–47. doi: 10.1002/sce.20463

Crossref Full Text | Google Scholar

14. Pottle J. Virtual reality and the transformation of medical education. Future Healthc J. (2019) 6(3):181. doi: 10.7861/fhj.2019-0036

PubMed Abstract | Crossref Full Text | Google Scholar

15. Fortune Business Insights. Virtual Reality Market Size (2021). Available online at: https://www.fortunebusinessinsights.com/industry-reports/virtual-reality-market-101378 (Accessed March 20, 2024).

16. Fortune Business Insights VR Health. Virtual Reality (VR) in Healthcare Market Size (2021). Available online at: https://www.fortunebusinessinsights.com/industry-reports/virtual-reality-vr-in-healthcare-market-101679 (Accessed March 20, 2024).

17. Farra SL, Gneuhs M, Hodgson E, Kawosa B, Miller ET, Simon A. Comparative cost of virtual reality training and live exercises for training hospital workers for evacuation. Computers, Informatics, Nursing: CIN. (2019) 37(9):446. doi: 10.1097/CIN.0000000000000540

PubMed Abstract | Crossref Full Text | Google Scholar

18. Bamidis PD, Dimitrova V, Treasure-Jones T, Poulton T, Roberts T. Augmented minds: technology's role in supporting 21st century doctors. In: Workshop on European TEL for Workplace Learning and Professional Development (TEL@ Work). Leeds: White Rose Research Online (2017). Available online at: https://eprints.whiterose.ac.uk/125750/ (Last Accessed April 04, 2024).

Google Scholar

19. Sanders EBN, Stappers PJ. Co-creation and the new landscapes of design. Co Des. (2008) 4(1):5–18. doi: 10.1080/15710880701875068

Crossref Full Text | Google Scholar

20. Lee G, Xia W. Toward agile: an integrated analysis of quantitative and qualitative field data on software development agility. MIS Q. (2010) 34(1):87–114. doi: 10.2307/20721416

Crossref Full Text | Google Scholar

21. Konstantinidis EI, Petsani D, Bamidis PD. Teaching university students co-creation and living lab methodologies through experiential learning activities and preparing them for RRI. Health Inform J. (2021) 27(1):1460458221991204. doi: 10.1177/1460458221991204

Crossref Full Text | Google Scholar

22. Bamidis PD, Konstantinidis EI, Billis AS, Siountas A. From e-homes to living labs: founding and organising the Greek active and healthy ageing living lab (thess-AHALL) and its networked services. Hell J Nucl Med. (2017) 20(2):112–25.

Google Scholar

23. Ertimur B, Venkatesh A. Opportunism in co-production: implications for value co-creation. Australas Mark J. (2010) 18(4):256–63. doi: 10.1016/j.ausmj.2010.07.004

Crossref Full Text | Google Scholar

24. Vargo Stephen L, Lusch Robert F. Evolving to a new dominant logic for marketing. J Mark. (2004) 68(1):1–17. doi: 10.1509/jmkg.68.1.1.24036

Crossref Full Text | Google Scholar

25. Krishna A, Morrin M. Does touch affect taste? The perceptual transfer of product container haptic cues. J Consum Res. (2008) 34(6):807–18. doi: 10.1086/523286

Crossref Full Text | Google Scholar

26. Troye SV, Supphellen M. Consumer participation in coproduction:“I made it myself” effects on consumers’ sensory perceptions and evaluations of outcome and input product. J Mark. (2012) 76(2):33–46. doi: 10.1509/jm.10.0205

Crossref Full Text | Google Scholar

27. Ordanini A, Pasini P. Service co-production and value co-creation: the case for a service-oriented architecture (SOA). Eur Manag J. (2008) 26(5):289–97. doi: 10.1016/j.emj.2008.04.005

Crossref Full Text | Google Scholar

28. Arvidsson A. Ethics and value in customer co-production. Mark Theory. (2011) 11(3):261–78. doi: 10.1177/1470593111408176

Crossref Full Text | Google Scholar

29. Dorta T, Safin S, Boudhraâ S, Marchand EB. Co-designing in Social VR. Process Awareness and Suitable Representations to Empower user Participation. arXiv preprint arXiv:1906.11004 (2019).

30. Antoniou PE, Bamidis PD. Devising a co-creative digital content development pipeline for experiential healthcare education. In: CC-TEL/TACKLE@ EC-TEL. RWTH Aachen University: CEUR Workshop Proceedings (ceur-ws.org) (2018).

Google Scholar

31. Schwaber K. Agile Project Management with Scrum. Microsoft press (2004).

Google Scholar

32. Benner P. Quality of life: a phenomenological perspective on explanation, prediction, and understanding in nursing science. ANS Adv Nurs Sci. (1985) 8(1):1–14. doi: 10.1097/00012272-198510000-00004

PubMed Abstract | Crossref Full Text | Google Scholar

33. Lazarus RS. The trivialization of distress. In: Hammonds BL, Scheirer CJ, editors. Psychology and health. (This chapter is reprinted from “Preventing Health Risk Behaviors and Promoting Coping With Illness” in the “Vermont Conference on the Primary Prevention of Psychotherapy,” Vol. 8, Rosen JC, Solomon LJ, editors. Hanover, NH: University Press of England) American Psychological Association (1984). p. 125–44. doi: 10.1037/10082-003

Google Scholar

34. Aronson J. A pragmatic view of thematic analysis. Qual Rep. (1995) 2(1):1–3. doi: 10.46743/2160-3715/1995.2069

Crossref Full Text | Google Scholar

35. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. (2006) 3(2):77–101.

Google Scholar

36. Godinho MA, Borda A, Kariotis T, Molnar A, Kostkova P, Liaw ST. Knowledge co-creation in participatory policy and practice: building community through data-driven direct democracy. Big Data Soc. (2021) 8(1):20539517211019430. doi: 10.1177/20539517211019430

Crossref Full Text | Google Scholar

37. Ntakakis G, Plomariti C, Frantzidis C, Antoniou PE, Bamidis PD, Tsoulfas G. Exploring the use of virtual reality in surgical education. World J Transplant. (2023) 13(2):36. doi: 10.5500/wjt.v13.i2.36

PubMed Abstract | Crossref Full Text | Google Scholar

38. Antoniou PE, Bamidis PD. 3D Printing and virtual and augmented reality in medicine and surgery: tackling the content development barrier through co-creative approaches. In: Suri J, Tsioukas V, Papadopoulos VN, editors. 3D Printing: Applications in Medicine and Surgery. Elsevier (2022) 2. p. 77–99.

Google Scholar

39. Antoniou PE, Economou D, Bamidis PD. Co-creative virtual reality content development in healthcare; evaluation methods and curricular integration. Imm Learn Res Pract. (2022):62–6. doi: 10.56198/A6PFYK690

Crossref Full Text | Google Scholar

40. Pears M, Henderson J, Antoniou PE, Ntakakis G, Nikolaidou M, Bamidis PD. Feasibility and acceptance of virtual reality reusable e-resources embedded in healthcare curricula. In: Liarokapis F, Shehade M, editors. 2022 International Conference on Interactive Media, Smart Systems and Emerging Technologies (IMET). Limassol: IEEE (2022). p. 01–8.

Google Scholar

41. Ioannidis L, Antoniou P, Bamidis P. Data modelling for visual entities to streamline virtual patient Re-purposing in virtual reality. In: Auer ME, Hortsch H, Michler O, Köhler T, editors. International Conference on Interactive Collaborative Learning. Cham: Springer International Publishing (2021). p. 1085–95.

Google Scholar

42. Antoniou PE, Pears M, Schiza EC, Frangoudes F, Pattichis CS, Wharrad H. Eliciting co-creation best practices of virtual reality reusable e-resources. Virtual Worlds. (2023) 2(1):75–89. doi: 10.3390/virtualworlds2010005

Google Scholar

43. Antoniou PE, Chondrokostas E, Bratsas C, Filippidis PM, Bamidis PD. A medical ontology informed user experience taxonomy to support co-creative workflows for authoring mixed reality medical education spaces. In: Economou D, Peña-Rios A, Dengel A, Dodds H, Mentzelopoulos M, Klippel A, et al., editors. 2021 7th International Conference of the Immersive Learning Research Network (iLRN). IEEE (2021). p. 1–9.

Google Scholar