Hamza Ouhaichi
Vogel Bahtijar1
Daniel Spikol- 1Department of Computer Science and Media Technology, Internet of Things and People Reseach Center, Malmö, Sweden
- 2Department of Science Education, University of Copenhagen, Copenhagen, Denmark
Multimodal Learning Analytics (MMLA) systems integrate diverse data to provide real-time insights into student learning, yet their design faces the challenge of limited established guidelines. This study investigates essential design considerations for MMLA systems during the research and development phase, aiming to enhance their effectiveness in educational settings. A qualitative approach employing semi-structured interviews was conducted with a diverse group of researchers in the MMLA field. Deductive and thematic analysis were used to identify key design considerations, including technology integration, constraints and learning scenarios. The analysis further revealed intersections between various design considerations, both confirming existing themes and highlighting new emergent ones. Based on the findings, the MMLA Design Framework (MDF) was developed to provide a structured approach to guide the design and development of MMLA systems. This framework, along with the identified design considerations, addresses the lack of conventional practices in MMLA design and offers practical insights for practitioners and researchers. The results of this study have the potential to significantly impact both research and educational applications of MMLA systems, paving the way for more effective and informed designs.
1 Introduction
In recent years, the field of education has witnessed significant advancements in technology and data analytics (Celik et al., 2022). One such advancement is the emergence of Multimodal Learning Analytics (MMLA) systems, which integrate multiple modalities such as text, audio, video, and sensor data to gain insights into students’ learning processes (Worsley, 2018). These MMLA systems have a promising potential in enhancing education by providing real-time feedback to both learners and educators (Junokas et al., 2017). With traditional learning analytics focusing primarily on log data analysis, MMLA offers a more holistic approach by considering various modes of information capture (Worsley, 2018). By analyzing multimodal data, these systems can help to comprehensively understand learner behavior and engagement levels (Prieto et al., 2018). However, designing effective MMLA systems presents several challenges and limitations (Ouhaichi et al., 2023a). Existing approaches often overlook crucial considerations for creating user-friendly and ethically sound educational tools (Ouhaichi et al., 2023b). The identification and understanding of key design considerations are paramount in the development of effective MMLA systems. Incorporating design considerations into the development of MMLA systems is paramount for ensuring their effectiveness and usability. The field of Human-Centered Learning Analytics (HCLA) emphasizes the importance of designing learning analytics tools that prioritize the needs and experiences of learners and educators (Shum et al., 2019). This approach draws inspiration from broader design theories, such as those in Human-Computer Interaction (HCI), which advocate for user-centered design principles. However, HCLA diverges from traditional HCI by prioritizing pedagogical and educational aspects over mere user preferences, ensuring that the design choices align with learning goals and outcomes. This shift in focus mirrors trends observed in health-tech solutions (Melles et al., 2021), where clinical efficacy and patient well-being often take precedence over user preferences alone. By grounding our research in HCLA principles, design considerations serve as guiding principles, shaping the architecture and functionality of the systems. In the field of MMLA, where diverse modalities such as text, audio, video, and sensor data converge, the complexity of these systems necessitates a thoughtful approach to design. Design considerations encompass a broad spectrum, ranging from data management and ethical implications to user experience and interface design. Each consideration plays a vital role in ensuring the integration of multiple modalities and the usage of the gathered data. By addressing these considerations comprehensively, MMLA systems can provide meaningful insights into learners’ behaviors, preferences, and engagement patterns. Moreover, a well-designed MMLA system facilitates educators’ ability to deliver personalized, data-driven feedback and interventions, fostering a conducive learning environment. Consequently, exploring and validating these design considerations are essential steps toward enhancing the effectiveness and usability of MMLA systems. Based on previous studies involving a systematic mapping study (Ouhaichi et al., 2023b), design science research (Ouhaichi et al., 2021) and action research (Ouhaichi et al., 2019), we identified key design considerations that are presented in the background section and summarized in Table 1. Therefore, this research aims to explore and investigate design considerations for MMLA systems through a semi-structured interview study with researchers in the field.
Table 1. Previously identified MMLA design considerations (Ouhaichi et al., 2023b).
While previous studies have identified potential design considerations, the investigation and refinement of these considerations necessitate the direct involvement of experts in the field. The emergence of MMLA systems has outpaced the development of conventional design practices. Methodologically, expert elicitation through semi-structured interviews is a well-established approach to gather in-depth qualitative data and insights that may not be captured by other research methods. This is particularly relevant in a rapidly evolving field like MMLA, where expert knowledge and experience can provide nuanced perspectives on the practical challenges and emerging best practices in system design. As researchers are at the forefront of MMLA system development (Ouhaichi et al., 2021), their perspectives offer essential insights into the practical challenges and opportunities associated with MMLA design. By engaging experts in semi-structured interviews, this study aims to triangulate findings from our previous research, ensuring the comprehensiveness and relevance of the proposed design considerations. This investigation is crucial for bridging the gap between theoretical frameworks and the real-world complexities of MMLA implementation, ultimately contributing to the establishment of robust and effective design guidelines for future MMLA systems. We address two research questions to drive this study:
1. How do researchers perceive and prioritize various design considerations in developing MMLA systems?
2. What implications do these considerations have for designing and developing effective MMLA tools?
To answer these questions, we conducted semi-structured interviews (Kallio et al., 2016) with a diverse group of researchers in the field of MMLA. The purpose of this study is to gain insights into the perspectives of researchers regarding the design considerations of MMLA systems. Understanding how experts perceive these considerations is critical for improving the design and usability of MMLA tools. By exploring these perspectives, we aim to contribute to developing more effective and user-friendly MMLA systems, ultimately benefiting educators and learners.
The remaining of the paper is organized as follows. Section 2 delves into the background on MMLA and underscores the importance of design considerations. This section sets the stage for understanding the connection of various elements in MMLA system design. Section 3 details the methodology employed, outlining the use of a semi-structured interviews and data collection. Section 4 presents the core of the study, showcasing the interview findings and engaging in a discussion on how researchers perceive and prioritize diverse design considerations. Moving forward, Section 5 conducts a thorough analysis of the implications arising from these considerations on the design and development of MMLA systems. In Section 6, we present a conceptual framework for designing MMLA systems based on the results and the analysis present earlier. Finally, Section 7 serves as the conclusion, summarizing key findings, accentuating their significance, and proposing avenues for future research.
2 Background
MMLA has evolved as an interdisciplinary field at the intersection of information technology, internet-of-things (IoT), and education. The integration of these domains allows for rich data collection from diverse sources such as wearable sensors, online platforms, learning management systems, and social media interactions (Blikstein et al., 2013). Despite its promising potential, MMLA systems encounter significant challenges that impede their practical effectiveness (Ochoa et al., 2016).
One of the primary challenges lies in data management and storage capacity. The sheer volume of multimodal datasets, often generated by IoT devices, poses complexities in terms of organization and storage. These large datasets necessitate robust infrastructure and innovative solutions to handle them effectively (Worsley et al., 2021). Moreover, the human element must be noticed. Privacy concerns and ethical implications are critical issues when developing MMLA systems even in controlled settings such a research lab. Lab environments, though controlled, still require stringent privacy protections for sensitive data like audio and video recordings to prevent unauthorized access or misuse (Smith et al., 2011). Ensuring informed consent and the right to withdraw are essential to uphold participants’ rights and dignity (Nissenbaum, 2009). Addressing these concerns is important to fostering user trust and ensuring the responsible use of MMLA technologies (Alwahaby et al., 2021). Considering these challenges, the design of MMLA systems must be underpinned by a comprehensive set of considerations. These considerations are vital guidelines that steer the development process, ensuring that MMLA systems are technologically robust, ethically sound, and user-friendly (Oviatt et al., 2018).
Design considerations are the various factors and constraints that must be accounted for during the planning and development of a product, system, or project. These include functionality, usability, aesthetics, environmental impact, cost, materials, and regulatory compliance. Unlike design requirements, which are specific, measurable criteria that a design must meet, design considerations are broader aspects that influence the design process and help shape the requirements. For example, according to Ulrich and Eppinger (2016), design considerations encompass elements like ergonomics and sustainability, guiding the decision-making process to ensure the outcome aligns with the overall goals and constraints of the project. These considerations include aspects such as data management, user experience, privacy, and the integration of diverse modalities and sensors. Table 1 presents a comprehensive list of the identified design considerations, which were derived from our systematic mapping study (Ouhaichi et al., 2023b), design science research (Ouhaichi et al., 2021), and action research (Ouhaichi et al., 2019). The table outlines key considerations such as data accuracy, system usability, ethical implications, and feedback mechanisms, which serve as the foundational elements for the MMLA system design. While prior research has identified initial design considerations through literature reviews and empirical work, engaging directly with MMLA researchers offers an essential evaluation step. This study bridges the gap between theory and practice by eliciting expert perspectives to confirm the relevance of existing considerations, uncover new ones not captured in prior research, and refine the overall framework. This direct engagement with practitioners ensures that the design considerations are subject to a level of scrutiny, ultimately leading to more effective MMLA system development and implementation. These considerations provide a framework for designers to address key challenges and issues associated with MMLA systems. However, further investigation is needed to understand the perspectives of researchers regarding these design considerations and identify any additional considerations that may arise in practice.
In alignment with the literature on learning analytics and educational technology, our approach emphasizes the importance of integrating these considerations comprehensively (Wise and Vytasek, 2017; Nguyen et al., 2020). For example, Wise and Vytasek (2017) highlight the need for systematic design in learning analytics to ensure the practical application of insights, while Nguyen et al. (2020) discuss design principles that enhance the effectiveness of learning analytics systems in higher education. Additionally, Shankar et al. (2020) provide a conceptual tool to support the development of multimodal learning analytics solutions, emphasizing the value of incorporating diverse data sources and analytics techniques. These studies underscore the importance of our identified design considerations in creating robust, user-centric MMLA systems that can adapt to various educational contexts. This research aims to explore the perspectives of researchers in MMLA to identify key design considerations for developing effective MMLA systems.
3 Methodology
Given the relatively new nature of MMLA systems and the lack of established design conventions, a multifaceted methodological approach is essential to comprehensively define design considerations. While prior research has identified potential considerations through literature reviews and empirical work (Ouhaichi et al., 2023a), the perspectives of researchers actively engaged in MMLA system development are crucial for exploring and refining these findings. Researchers, as domain experts, possess firsthand experience with the intricacies and challenges of MMLA design, making their insights invaluable for bridging the gap between theoretical frameworks and practical implementation. Therefore, incorporating semi-structured interviews with MMLA researchers into this study’s methodology is a deliberate choice aimed at consolidating existing knowledge, uncovering new considerations, and ultimately contributing to a robust and comprehensive set of MMLA design principles. The research approach employed for this study is qualitative research with a focus on semi-structured interviews (Kallio et al., 2016), following the guidelines for conducting and reporting case study research in software engineering (Runeson et al., 2012). The participants’ selection criteria ensured that participants possessed extensive knowledge in the field of MMLA and could provide valuable insights into design considerations. Data collection involved conducting individual semi-structured interviews with each participant. These interviews allowed for an in-depth exploration of their perspectives on various aspects of MMLA system design. Ethical considerations were considered throughout the process, including obtaining informed consent from participants and ensuring data confidentiality. Deductive and thematic analysis were the primary methods for analyzing interview data (Kallio et al., 2016). This involved identifying common themes or patterns within the responses and organizing them according to the relevant design considerations. An illustration of the research protocol and the different phases of this study is provided in Figure 1.
Figure 1. Research protocol and the different phases of this study.
To investigate design considerations for MMLA systems, this study employed a qualitative approach through semi-structured interviews. Initially, a purposive sampling strategy was used to identify and recruit experts in the MMLA field, ensuring a diverse range of experiences and perspectives. Participants were selected based on their demonstrated expertise in MMLA research, development, or implementation, evidenced by publications, presentations, or involvement in relevant projects. Following recruitment, individual semi-structured interviews were conducted, utilizing an interview guide designed to elicit in-depth responses regarding key design considerations. These interviews were audio-recorded and transcribed to ensure accuracy. Subsequently, the transcripts underwent a rigorous deductive-thematic analysis, employing a coding framework derived from prior MMLA research. Two researchers independently coded the data, with regular discussions to establish inter-coder reliability and resolve discrepancies. This iterative process involved identifying recurring themes and patterns related to design considerations, refining the coding framework, and organizing the data accordingly. Frequency analysis of codes was also conducted to assess the prevalence of different themes.
This study was conceived and led by the first author, who collaborated closely with the co-authors throughout the entire research process. The team met several times per week, utilizing a shared environment that facilitated comprehensive access to all research materials, promoting transparency and seamless collaboration. The collaborative effort spanned the entire research lifecycle, encompassing the development and refinement of the interview protocol, the formulation of interview questions, participant recruitment, data collection, coding, analysis, and interpretation of findings. The research process began with a thorough review of existing literature on MMLA systems and design considerations. This foundation informed the development of the initial interview protocol, which was then iteratively refined through discussions among the co-authors. To ensure the validity and relevance of the research approach, the team also sought feedback from external experts in the field of MMLA. Their input was instrumental in shaping and refining the study’s goals, methodology, and data analysis strategies. Once the interview protocol was finalized, the participant recruitment process commenced, employing a purposive sampling strategy to identify and engage experts in the MMLA domain. Following the completion of the interviews, the data was meticulously transcribed and analyzed using a combined deductive-thematic approach. The coding process, facilitated by Atlas.ti software, involved iterative discussions among the co-authors to ensure consistency and accuracy in the interpretation of findings. This collaborative approach allowed for the emergence of new insights and the refinement of existing design considerations, contributing to a more comprehensive and nuanced understanding of the challenges and opportunities associated with MMLA system design. We provide anonymized transcriptions of the interviews and a comprehensive statistics sheet as Supplementary material. These resources offer detailed insights into the participants’ perspectives and the frequency of various themes discussed, enhancing the transparency and robustness of our findings. This Supplementary material allows for a deeper understanding and independent verification of our analysis and conclusions.
3.1 Defining research objectives
In this study, our research objectives were carefully defined to explore the perspectives of researchers engaged in MMLA systems. The objectives of this research were synthesized from the findings of previous research (Ouhaichi et al., 2023a). The primary aim is to investigate and explore the perspectives of researchers in MMLA to identify key design considerations for developing effective MMLA systems. Specifically, we sought to understand how these experts perceive critical aspects such as data management, human factors, sensors, modalities, and ethical concerns in the context of MMLA. By defining these objectives clearly, we aimed to uncover nuanced perspectives and uncover potential gaps or challenges in the existing design approaches. Furthermore, our objectives encompassed the identification of implications associated with these considerations, shedding light on the broader impact of design choices in MMLA systems. Through a systematic exploration of these objectives, we intended to contribute valuable knowledge to the field, informing future developments and strategies in MMLA system design.
3.2 Developing interview protocol
Developing the interview protocol involved a systematic approach to ensure a comprehensive exploration of researchers’ perspectives on MMLA systems. The formulation of the interview questions (see Figure 2) was guided by the identified design considerations. Each question was crafted to delve deeply into these areas, encouraging participants to provide detailed insights. The protocol was structured to facilitate an open and exploratory dialogue, allowing participants to express their viewpoints freely. Additionally, the questions were designed to be flexible, enabling the interviewers to adapt the conversation based on the participants’ responses, ensuring a dynamic and responsive interaction. This iterative development of the interview protocol involved rigorous testing and refinement, including pilot interviews, to validate the effectiveness and clarity of the questions. Through this process, the interview protocol was tailored to elicit meaningful responses, aligning with the study’s research objectives, and ensuring the comprehensive exploration of the identified design considerations.
Figure 2. Interview questions and how they connect to each consideration.
3.3 Participant selection
We employed a systematic approach for participant selection, involving a total of ten interviewees, including two for pilot interviews. The selection process targeted individuals with experience in MMLA. To ensure a diverse and comprehensive perspective, invitations were extended to researchers with notable publications related to the design and utilization of MMLA systems. Participants were chosen based on their demonstrated expertise in MMLA, evidenced by publications, presentations, or direct involvement in MMLA related projects. To ensure a diversity of perspectives, researchers from different institutions, geographical locations, and areas of MMLA specialization were actively sought. The final selection was also influenced by the participants’ willingness to engage in interviews and the need to balance the sample across various dimensions of MMLA experience, such as research focus, methodological approaches, and application domains.
The initial contact phase was initiated through personalized email invitations sent to a varied group of researchers actively engaged in MMLA related projects. Potential participants were identified through a review of relevant literature, focusing on authors who had published or presented on topics related to MMLA system design, implementation, or evaluation. Personalized email invitations were sent to these individuals, outlining the study’s objectives, interview structure, and expected time commitment. The emails also emphasized the importance of their expertise and experience in contributing to the research. Out of the expressions of interest received, ten participants were selected for the interviews. Additionally, two pilot interviews were conducted with individuals possessing diverse backgrounds, including two PhD students: one is specializing in digital health using sensors and the other is focusing on privacy in IoT and Smart homes. These pilot interviews were instrumental in refining and adjusting the interview questions to better suit the study’s objectives. The feedback obtained from these initial interviews allowed us to identify ambiguities and improve the clarity and relevance of the questions. This process of refinement and adjustment was ongoing, as insights gained from subsequent interviews continued to inform and enhance the interview framework, ensuring a comprehensive and effective data collection process throughout the study. This systematic approach ensured a diverse and knowledgeable participant pool, enhancing the depth and breadth of insights gathered during the interviews. We provide the list of interviewees in Table 2, associated their position, years of experience, country, and connection to MMLA. All the details in the table were extracted from the prefilled from preparing for the interviews.
Table 2. List of interviewees.
3.4 Interviews and data collection process
We designed a systematic online data collection process to gather valuable insights. Participants completed a demographics and consent form before the interview, providing essential information and granting consent for recording and data processing. This ensured transparency and compliance with ethical considerations throughout the research process. The interviews were recorded to capture the nuances of the discussions. Before the main interviews, we conducted a thorough pilot phase, consisting of two initial interviews. These pilot sessions played a crucial role in refining the interview questions for clarity and relevance. Each of the interview questions were formulated to address the design consideration as illustrated in Figure 2. Note that the first two boxes in Figure 2 include four elements from Table 1 (Data Collection & Management and Sensors & Modalities). Following this iterative refinement, we conducted eight comprehensive interviews, each focusing on specific aspects of MMLA system design. We used automated tools to transcribe the interviews. The transcriptions underwent proofreading to rectify any discrepancies, ensuring improved accuracy and fidelity of the transcribed data. This process formed the foundation for subsequent analysis.
3.5 Data analysis methods
We adopted a dual approach incorporating deductive and thematic analysis methods (Kallio et al., 2016). The deductive analysis commenced with the predefined set of considerations, derived from prior studies, providing a structured framework for the initial coding (Ouhaichi et al., 2023b). Simultaneously, the thematic analysis allowed for the emergence of new considerations and perspectives, ensuring a comprehensive exploration of the data. The coding process, a crucial aspect of the analysis, was facilitated using the qualitative data analysis software Atlas.ti.1 This systematic approach enabled us to identify recurring themes, patterns, and connections within the data, ensuring a nuanced and multifaceted understanding of the participants’ insights.
The coding process was conducted by the first author using the predefined set of considerations as an initial framework. However, to ensure rigor and minimize potential bias, a collaborative approach was adopted. Regular discussions with co-authors were held throughout the coding process, starting from the analysis of the first interview transcript. These discussions facilitated a continuous refinement of the coding, allowing for the emergence of new codes and themes as patterns became evident across the data. Disagreements or discrepancies in coding were resolved through consensus-building discussions, ensuring a shared understanding of the data’s meaning and interpretation. While a formal inter-coder reliability analysis was not conducted, the iterative and collaborative nature of the coding process, combined with the use of a shared working environment to enhance consistency, contributed to the trustworthiness and validity of the findings.
4 Results
The findings from the interviews revealed several important design considerations for MMLA systems. In the results section, we break down the insights gathered from interviews with experienced MMLA researchers. The findings are split into two parts. First, we explore how the initial design considerations, pinpointed in prior studies, reinforced through the interview data. Then, we highlight the new considerations that emerged during the interviews, giving us a more detailed picture of MMLA system design. This approach aims to show both the affirmation or reinforcement of existing knowledge and the changes in perspectives design considerations. Table 3 presents an overview of the emergent themes and reinforced considerations derived from the semi-structured interviews with participants. Each consideration is accompanied by specific themes identified during the analysis. The table includes information on the participating researchers associated with each theme, the occurrences of themes, and it was emerged or reinforced. The participants’ codes (e.g., P3, P4) indicate their involvement in discussions related to each specific consideration and theme. This comprehensive presentation provides a clear snapshot of the key findings.
Table 3. Overview of the considerations and themes emerging from the interviews.
4.1 Reinforced initial design considerations
In this section, we present the insights obtained from the semi-structured interviews, reinforcing and expanding upon the initial considerations identified in previous research. The results reinforced the importance of these considerations and revealed emergent themes within each one. We provide details from our interviews and analysis for each consideration as well as establishing the interconnection of the results with literature. The themes and design considerations presented in this study are inherently interrelated, reflecting the complex and multifaceted nature of MMLA system design. During the interviews, it became evident that discussions about one theme often naturally extended into related areas. For instance, when exploring ethics and privacy, participants frequently touched upon the learning context, highlighting how these considerations influence and inform each other. This overlap is expected and essential, as it mirrors the real-world interconnectedness of these factors in MMLA systems. By providing overlapping overviews for each theme, we capture the holistic and integrative nature of the discussions, ensuring that our analysis accurately represents the nuanced perspectives of the experts. This approach underscores the necessity of considering multiple, intersecting factors to develop robust and effective MMLA systems.
4.1.1 Data collection
During the interviews, the importance of Data Collection was consistently emphasized. The discussions reinforced the importance of capturing multimodal data from various sources. Participants (P3, P5, P6, P7, P9, P10) stressed the need for real-time data collection techniques that are context-specific, enabling the acquisition of relevant and timely information, e.g., the social signal interpretation framework (Di Mitri et al., 2018). Existing data collection methods, including sensors, online platforms, and learning management systems, were seen as essential components of MMLA systems. Participants highlighted the need for continuous advancements in these technologies to enhance data granularity and precision. Additionally, the discussions underscored the importance of considering the ethical implications of data collection, ensuring informed consent and privacy protection for the individuals being monitored (Alwahaby et al., 2021).
4.1.2 Data management
The complexities associated with organizing and storing of multimodal datasets were emphasized (Ochoa et al., 2016). Participants (P3, P5, P6, P7, P9) highlighted the need for robust infrastructure and innovative storage solutions to handle the diverse and vast datasets generated by IoT devices and various sensors as mentioned by Ochoa et al. in MMLA data challenges. Moreover, participants stressed the importance of data security and access control in data management processes. The discussions emphasized the continuous evolution of data management techniques to address the challenges posed by the increasing volume and variety of multimodal data in educational settings.
P6: “We’re going to use a narrow system with fewer options to collect data. But then what? It does not matter what kind of learning you want to pursue. We need to consider two dimensions: the system’s constraints and the diversity of learning approaches. Not everyone follows a specific theoretical approach. Practitioners and learners may not have a defined pedagogical method for their self-driven learning paths. Solutions need to be coherent and flexible, adapting to different purposes and contexts, even if learning theories are not always applied.” – 7:49 ¶ 40 in Speakers 2.docx.
4.1.3 Human factors
The theme of Collaborative Design emerged and was mentioned by different participants (P3, P4, P5, P8, P10). Collaborative design processes, involving the active participation of all stakeholders, were seen as essential for creating MMLA systems that align with the diverse needs and expectations of users (Chejara, 2020). The discussions also delved into the challenges of facilitating collaborative design in a cross-disciplinary milieu. Types of Users emerged as a sub-consideration by participants (P3, P4, P6, P8, P9, P10). Beyond traditional user categories, participants discussed the need to differentiate users based on their roles, expertise levels, and learning objectives (Noel et al., 2018). To clarify, “human factors” encompass all individuals involved with the MMLA system, including stakeholders, developers, and decision-makers, while “types of users” specifically refer to the diversity of potential end-users, such as students, educators, and researchers. Tailoring MMLA interfaces and feedback mechanisms to accommodate the diverse requirements of users became a central point of discussion. Insights highlighted the significance of adaptive interfaces, ensuring personalized experiences that cater to the specific needs and preferences of different user types. Participants emphasized the impact of social interactions on the learning process, indicating the need for MMLA systems to consider what we named Interpersonal Dynamics (Oviatt et al., 2018). There was a consensus in the discussion around the challenges of fostering interpersonal relationships in online learning environments. Participants also discussed the role of MMLA in enhancing social engagement among learners (Riquelme et al., 2019).
4.1.4 Interpretation and feedback
Interpretation was a central theme, reinforced by all participants. They highlighted the challenges associated with interpreting diverse multimodal data accurately (Ochoa et al., 2016). Beyond data accuracy, the discussions emphasized the importance of contextual interpretation. Understanding the context in which data is generated, including the learning environment and user interactions, was deemed crucial for meaningful interpretation. Participants stressed the potentials and risks for advanced machine learning algorithms capable of contextual analysis, ensuring accurate and relevant interpretations of multi-modal data. Participants emphasized the need for timely and personalized feedback (P4, P6, P8, P10). The discussions revealed the challenges of providing real-time feedback to learners, particularly in dynamic learning environments (Huertas Celdrán et al., 2020). Personalization of feedback, tailored to individual learning styles and preferences, was highlighted as a key factor in enhancing the effectiveness of MMLA systems.
4.1.5 Learning scenarios
The Learning Scenarios consideration was reinforced and led to the emergence of several themes. Impact on Learning (P3, P4, P6, P8, P9) was a central theme, emphasizing the influence of MMLA systems on educational outcomes. Participants discussed the importance of understanding how MMLA interventions impact the learning process and overall academic achievements (Alwahaby et al., 2021). The discussions highlighted the need for continuous evaluation of the effectiveness of MMLA implementations, ensuring that interventions positively influence learning behaviors and outcomes. Learning Cases surfaced as a significant emergent theme. All participants emphasized the need to consider diverse learning cases, accounting for various subjects, age groups, and educational contexts (Eradze et al., 2020). Designing MMLA systems that can adapt to different learning scenarios and disciplines became a focal point of discussion. Participants stressed the importance of capturing the intricacies of different educational domains, tailoring MMLA tools to address specific challenges and requirements unique to each learning case. Learning Context (P5, P7, P9) was another emergent theme within Learning Scenarios. Participants emphasized the significance of contextual factors, such as cultural differences, socioeconomic backgrounds, and learning environments (Worsley, 2012). Understanding the unique context in which learning occurs was deemed vital for designing tailored MMLA solutions. The discussions revolved around the challenges of adapting MMLA interventions to diverse educational contexts, highlighting the need for flexible, context-aware designs that can cater to a wide range of learning environments.
4.1.6 Privacy and ethics
Privacy and Ethics were discussed extensively, with participants emphasizing their paramount importance in MMLA systems. Ethical considerations were reinforced by (P3, P4, P5, P6, P8) and centered around the responsible use of data. Participants stressed the need for transparent data usage policies, ensuring informed consent, and protecting the privacy rights of learners. Ethical considerations also extended to the development and deployment of algorithms, ensuring fairness, transparency, and unbiased analysis of multimodal data (Alwahaby et al., 2021). Privacy concerns were reinforced by (P5, P6, P7, P8, P9) highlighting the challenges of balancing data collection for educational insights with safeguarding user privacy. The need for stringent data anonymization techniques and secure storage mechanisms was emphasized. Participants also stressed the importance of educating users about the data collection processes, empowering them to make informed decisions about their privacy preferences within MMLA systems.
P5: “So, reflecting on what kind of data can be collected is crucial. Researchers and stakeholders alike need to form opinions on this. Often, when we are asked about cookies while browsing, we just accept without thinking because we need the resource. Some people are strict about data collection, while others are more lenient. It’s important to train people to have informed positions. Knowing why you are making a decision is much better than making a random or opportunistic choice without understanding the consequences.” – 7:52 ¶ 44 in Speakers 1.docx.
4.1.7 Sensors and modalities
Sensors and Modalities were explored in-depth, reinforcing their roles in MMLA systems. All participants had a consensus on the diverse range of potential sensors and modalities used, including text, audio, video, and sensor data. Participants emphasized the need for integrating multiple modalities to capture a comprehensive view of the learning environment. The challenges of synchronizing data from various modalities and ensuring seamless integration were discussed, emphasizing the need for interoperability and standardized data formats (Mu et al., 2020). Participants highlighted the importance of selecting appropriate sensors based on the learning scenario and the data required. Participants also emphasized the potential of emerging sensor technologies, such as affective computing sensors, in enriching the multimodal data captured for learning analytics (Huertas Celdrán et al., 2020).
4.2 Emergent design considerations
In addition to reinforcing existing considerations, the interviews uncovered novel insights and new emergent design considerations that adds to our understanding of MMLA system. These new considerations provide valuable perspectives for researchers, developers, and educators. The new considerations are Constraints, Development and Implementation, Research Orientation and Technology integration. Each of these emergent considerations contains several themes and their exploration enhances our knowledge about MMLA design, guiding future developments and strategies in the field.
4.2.1 Constraints in MMLA systems
There are various challenges in MMLA as was pointed out by Cukurova et al. (2020) and by Oviatt (2018). The theme of Cost emerged as a significant constraint in the development and implementation of MMLA systems, a finding mentioned by participants (P3, P4, P8, P10). Discussions revolved around the financial challenges associated with creating and sustaining advanced MMLA technologies. Participants highlighted the need for cost-effective solutions that balance technological sophistication with affordability. Cost considerations encompassed not only the initial development expenses but also ongoing maintenance, updates, and scalability (Cornide-Reyes et al., 2019). Technical Challenges were seen as a central constraint faced by researchers and developers in the MMLA domain. Participants emphasized the complexities arising from the integration of diverse modalities and the management of large datasets generated by Internet of Things devices and sensors. Addressing these challenges required innovative solutions, such as advanced data processing techniques and machine learning algorithms (Worsley, 2018). Participants stressed the need for continuous advancements in technology to overcome these hurdles and ensure the seamless operation of MMLA systems. The theme of Design Uncertainty emerged as a pervasive constraint in MMLA systems, a finding echoed by participants (P3, P4, P5, P6, P7, P9). Discussions centered around uncertainties related to data accuracy, algorithmic performance, and user behavior analysis. Participants emphasized the challenges of dealing with noisy data and the inherent unpredictability of human learning patterns. Moreover, the evolving nature of educational technology adds another layer of complexity, as new tools and methods continuously emerge, making it difficult to establish stable, long-term design practices. This uncertainty itself was highlighted as a significant obstacle, as it complicates the design and implementation of MMLA systems. Participants expressed that this unpredictability affects their confidence in making design decisions, optimizing system performance, and ensuring reliable outcomes.
P7: “We're switching from a platform we've been developing since 2014. We maintained it for almost 10 years, but the costs were enormous, amounting to thousands of euros per year. Despite efforts to secure funding to keep it running, we had to shut it down in June. Maintaining the platform required significant resources, including developers to handle new constraints and technologies. We've released a new version that is compatible with modern technologies, but closing the old platform will result in lost content. This will significantly impact users who preferred the previous version, but as researchers, we can no longer sustain the costs and technological incompatibilities.” – 7:19 ¶ 80 in Speakers 3.docx
4.2.2 Development and implementation
Development and implementation emerged as a category that encompasses the following themes. The interviews underscored the necessity of well-informed decision-making processes, involving considerations such as ethical implications, scalability, and user requirements. Participants emphasized the challenges of balancing technological sophistication with affordability and ensuring alignment with educational goals. Design Tools played a pivotal role in shaping MMLA systems, according to participants. The discussions emphasized the significance of various software tools, from data processing and analytics software to user interface design and visualization tools. Participants stressed the need for tools that are not only advanced but also user-friendly, enabling seamless integration of multiple modalities. The right design tools were deemed essential for efficient system development. Implementation considerations delved into the practical challenges of translating design concepts into functioning MMLA systems. Participants highlighted the complexities involved in integrating hardware and software components, emphasizing the need for interdisciplinary collaboration (Chejara, 2020). Maintenance emerged as part of the implementation and the interviews emphasized the importance of addressing system flaws and adapting to evolving educational contexts. Maintenance considerations extended to data management and privacy, ensuring the long-term effectiveness of MMLA solutions. Planning considerations focused on strategic alignment with educational objectives. Participants stressed the need for comprehensive project planning, including objective setting, defining scope, and establishing timelines. Aligning MMLA technologies with specific educational needs and challenges was highlighted. Strategic planning was seen as vital for successful system deployment, ensuring that the developed solutions effectively addressed the educational context.
4.2.3 Research oriented
The category of Research Oriented considerations encompasses vital aspects tailored to researchers engaging with MMLA systems. This set of considerations specifically addresses the unique needs and goals associated with scholarly inquiry, academic research, and educational exploration (Worsley, 2018). Design-based Research emerged as a theme within Research Oriented considerations. Participants emphasized the significance of integrating research methodologies directly into the design process of MMLA systems. This approach ensures that the tools developed serve not only practical educational purposes but also contribute valuable insights to the academic community. Design-based research methodologies were highlighted for their ability to bridge the gap between theory and practice, fostering a symbiotic relationship between academic exploration and practical application. A Research Application approach was identified as a guiding principle for research-oriented MMLA systems. The interviews revealed a consensus among participants about the importance of creating tools that facilitate rigorous research endeavors. MMLA systems underpinned by a research-centric approach enable academics to explore new pedagogical paradigms, conduct analyses, and generate knowledge that advances the field of education. These systems serve as platforms for experimentation, allowing researchers to validate theories, test hypotheses, and contribute to the broader academic discourse.
P7: “In the UK, especially in higher education, there's a cultural emphasis on student-centered learning. Everything focuses on the student’s experience, well-being, and happiness during learning. They avoid pressuring students or forcing them to do things they don’t want to do, instead encouraging them gently. However, this raises a question: do students or even some teachers actually know what is best for the students and their learning?” – 14:8 ¶ 17 in Speakers 5.docx
4.2.4 Technology integration in MMLA systems
The integration of advanced technological components forms a cornerstone in the development of effective MMLA systems (Ouhaichi et al., 2021). This section delves into the considerations associated with the integration and utilization of diverse technological elements, enhancing the functionality and capabilities of MMLA technologies. Technological Integration encapsulates the seamless incorporation of cutting-edge technologies within MMLA systems. Participants emphasized the need for integrating innovative hardware and software solutions to capture, process, and analyze multimodal data accurately (Tamura et al., 2019). This integration extends beyond traditional educational technologies, incorporating emerging tools from the fields of artificial intelligence, machine learning, and data analytics. The discussions highlighted the importance of leveraging state-of-the-art technology to enhance the precision and depth of insights derived from MMLA systems. The concept of EdTech Synergy emphasizes the harmonious integration of educational technology (EdTech) tools within MMLA systems. Participants recognized the potential of integrating existing educational platforms, applications, and digital resources to enrich the learning experience. This synergy enables MMLA systems to leverage the strengths of various EdTech tools, creating a comprehensive ecosystem that fosters interactive and engaging educational environments (Di Mitri et al., 2018). Participants highlighted the need for interoperability and compatibility between MMLA technologies and existing EdTech solutions to ensure a seamless user experience.
The design considerations for MMLA systems share similarities with those in other fields of educational technology, reflecting common challenges such as data management, privacy, and user experience. However, the unique specificity of MMLA systems lies in their focus on multimodality and analytics, which are crucial for capturing and interpreting diverse data sources such as text, audio, video, and sensor inputs. These aspects are directly reflected in the key considerations of data collection, sensors and modalities, and interpretation and feedback. For instance, data collection in MMLA involves integrating multiple data streams in real-time, which is a more complex process than in traditional learning analytics (Worsley, 2018). Similarly, the use of advanced sensors and modalities to gather nuanced data about learner interactions and behaviors is specific to MMLA and requires specialized design strategies (Di Mitri et al., 2018). The iterative nature of the MMLA framework further accentuates these overlaps, as continuous refinement based on multimodal data insights necessitates revisiting and integrating different design considerations at various stages. This interdependency highlights the need for a holistic approach in MMLA system design, ensuring that each consideration is addressed in a manner that supports the overall functionality and effectiveness of the system. By systematically mapping MMLA features to these design considerations, we underscore the unique requirements of MMLA systems and provide a structured methodology to navigate their complexities.
5 Discussion
In this study, our objective was to present and investigate perspectives of researchers regarding MMLA system design considerations. Through a qualitative approach involving interviews and thematic analysis, we aimed to understand the complexities and challenges experienced by researchers in MMLA. By exploring their viewpoints, our analysis seeks to shed light on the crucial aspects influencing the design and development of effective MMLA systems. The process of data synthesis involved transcription, coding, and thematic analysis. We identified several prominent patterns and recurring themes across different considerations. These patterns highlighted the diversity of perspectives, indicating the multifaceted nature of MMLA design. Commonalities emerged, revealing shared challenges and opportunities experienced by the participants, adding depth to our analysis. Our analysis revealed intersections between different considerations. Ethical considerations were found to influence data collection methods, ensuring responsible and respectful practices (Alwahaby et al., 2021). Additionally, the influence of research-oriented methodologies on feedback mechanisms became evident, emphasizing the symbiotic relationship between pedagogy and technology. Simultaneously, we highlighted disparities, emphasizing the complex balancing act required in designing MMLA systems that meet ethical, pedagogical, and technological standards (Tamura et al., 2019).
The resulting themes are overlapping across the design considerations, reflecting the complex and interconnected nature of MMLA system design. This overlap can be attributed to several factors, including the inherent interrelations and interdependencies among the various aspects of designing and implementing these systems. For instance, data management and data collection are closely linked, as efficient data collection methods are crucial for effective data management (Worsley, 2018). Similarly, the themes of collaborative design and user experience are interdependent, as involving stakeholders in the design process directly influences the usability and effectiveness of the system (Shum et al., 2019). The iterative nature of the proposed framework further accentuates these overlaps. Iterative design processes are characterized by continuous refinement and feedback loops, which naturally lead to revisiting and interlinking different design considerations at various stages (Worsley et al., 2021). For example, initial considerations for data privacy may evolve as new data management strategies are implemented and assessed, requiring ongoing adjustments to ensure compliance and user trust (Alwahaby et al., 2021). This iterative approach ensures that the design remains flexible and responsive to emerging challenges and insights, but it also means that themes cannot be entirely isolated from one another. The overlapping themes also highlight the multifaceted challenges faced by MMLA system designers. For example, the interplay between privacy concerns and the need for comprehensive data collection underscores the tension between ethical considerations and technical requirements (Oviatt et al., 2018). Addressing such overlaps requires a holistic understanding of how different design considerations impact one another and the overall system. From a structural perspective, the framework we propose reflects these overlaps through its phased and iterative design. Each phase is designed to address specific considerations while remaining flexible enough to incorporate feedback and adjustments from other phases. This approach not only ensures thorough coverage of all relevant aspects but also facilitates the identification and resolution of potential redundancies and conflicts.
Our findings reinforced previously identified themes from the literature, such as ethical guidelines and data management strategies (Alwahaby et al., 2021). These themes were substantiated through participants’ experiences and practices, reinforcing their importance in MMLA design. Furthermore, emergent themes, including the role of affective computing and the integration of real-world contexts, surfaced from the participants’ responses. These emergent themes underscored evolving aspects within MMLA design, reflecting the dynamic nature of the field. Researchers exhibited varied prioritization of considerations, often shaped by their distinct roles and experiences. Ethical considerations were universally regarded as paramount, emphasizing the ethical responsibility associated with handling multimodal data (Garaizar and Guenaga, 2014). Research-oriented methodologies were highly prioritized by scholars aiming to contribute to the educational research landscape, showcasing the symbiosis between academic objectives and system design (Ouhaichi et al., 2023b). The implications derived from our analysis are significant for practitioners, policymakers, and educators. With the resulting set of design considerations, practitioners can enhance and optimize existing MMLA tools to better achieve pedagogical objectives. By incorporating these well-defined considerations, developers and educators can ensure that MMLA systems are effective in capturing and analyzing learning data and support educational goals. This alignment helps create tools that are both responsibly designed and educationally impactful, addressing the complex needs of modern learning environments. Policymakers can formulate guidelines that support ethical practices and encourage innovation. For MMLA system designers, who are mainly researchers (Ouhaichi et al., 2023a), the implications include improved strategies for data integration, system usability, and alignment with educational objectives. For instance, a designer might implement more robust data collection methods to ensure comprehensive learner analytics. Educators, on the other hand, can benefit from these improved systems by receiving more accurate and actionable insights into student performance, which can inform their teaching strategies. Learners, as potential end users, stand to gain from enhanced feedback mechanisms that support personalized learning experiences. By addressing the specific needs and roles of each stakeholder group, we demonstrate how these design considerations lead to practical improvements in the development and use of MMLA systems.
Reflecting upon the outcomes of the interviews and the identified themes, the development and proposal of a design framework (presented in the following section) represents a concrete step forward in guiding the design and development process of MMLA systems.
6 A framework for MMLA system design
The need for a comprehensive design framework emerged as a significant finding from our interviews with researchers, who expressed various uncertainties and challenges in the development and implementation of MMLA systems. All participants highlighted the unpredictable nature of data accuracy, algorithmic performance, and user behavior, which complicates the design process. These uncertainties underscore the necessity for a structured approach to manage and mitigate potential risks. From our thematic analysis, it became evident that design considerations are not isolated but interrelated, with dependencies and priorities that necessitate a logical and temporal sequencing. For example, robust data management practices are foundational and must precede advanced data interpretation techniques to ensure accuracy and reliability. Similarly, the iterative design process is crucial for continuously refining system functionalities based on feedback and evolving requirements. The need for a framework also stems from the observed overlap between themes such as collaborative design and user experience, which must be integrated coherently within the design process. The proposed framework addresses these interdependencies by providing a phased and iterative approach, ensuring that each design consideration is systematically addressed and aligned with the overall objectives of MMLA system development. This structured methodology not only helps in managing uncertainties but also enhances the practical applicability and effectiveness of MMLA systems in real-world educational settings.
The MMLA design framework (MDF) illustrated in Figure 3, emerges as a guide derived from the analysis of our research findings involving the interviews. This framework encompasses six distinct phases, each strategically designed to navigate and address the identified and emerged considerations. Each phase unfolds a specific set of outcomes, collectively contributing to the overarching goal of enhancing the precision, ethicality, and effectiveness of MMLA systems in diverse educational settings. In this section, we present the six phases of the framework in details, showing that activities and considerations to be undertaken. The framework provides different implications on design practices as well as ability to analyze and assess the design of existing MMLA systems.
Figure 3. Framework for MMLA design and analysis.
6.1 MMLA design framework
The proposed framework operates on a phased and iterative approach, systematically guiding the design activities. In each phase, specific objectives are addressed, ensuring a structured development process. This phased and iterative approach allows for continuous improvement and adaptation, refining the framework based on feedback and evolving requirements.
1. Preparation and Needs Analysis: In the initial phase, the focus is on establishing clear educational objectives in alignment with curriculum requirements and learning goals. This involves comprehensive needs analysis to understand the specific requirements of stakeholders, including learners, educators, and institutions.
2. Data Collection and Management: This phase begins with the selection of appropriate sensors and online platforms for capturing multimodal data. Ethical data practices are implemented to ensure user privacy, data security, and informed consent. Robust data management strategies are developed to organize and store multimodal datasets effectively.
3. Privacy and Ethics: This phase involves investigating and identifying privacy and ethical issues to design the MMLA system. It is characterized by the integration of ethical guidelines and transparent data practices into the design process. Key actions include defining and implementing ethical guidelines, ensuring transparent data practices. This approach ensures that the MMLA system is not only effective but also aligns with ethical standards.
4. Development: Advanced data interpretation is utilized, involving the implementation of algorithms and machine learning techniques for accurate interpretation of multimodal data. Real-time feedback loops are established to provide continuous responses to learners and educators based on interpreted data. Cutting-edge technologies are seamlessly integrated to enhance the precision and depth of insights derived from MMLA systems. Harmonious integration with existing EdTech tools ensures interactive and engaging educational environments.
5. Interpretation and Feedback: This phase focuses on creating diverse learning scenarios that cater to different educational contexts and learner needs, ensuring adaptability and inclusivity. The impact of learning scenarios is evaluated, assessing effects on student engagement, performance, and overall learning outcomes. This iterative process refines scenarios for enhanced effectiveness.
6. Iterative Refinement and Validation: This phase focuses on evaluating the MMLA system for usability and effectiveness in learning and analytics. Expert consultation provides valuable feedback, validating the conceptual system and ensuring its alignment with real-world applications and scholarly discourse. The system undergoes iterative refinement, incorporating expert input and addressing potential limitations, to enhance its practical applicability and robustness.
6.2 Implications for design and analysis
The proposed framework holds significant implications for the design of MMLA systems. By emphasizing the identification of clear educational objectives and conducting thorough needs analysis, the framework ensures that the design process is rooted in the specific requirements of stakeholders, including learners, educators, and institutions. Moreover, the incorporation of advanced data interpretation, real-time feedback loops, and cutting-edge technologies in the development phase highlights the framework’s emphasis on leveraging state-of-the-art tools to enhance the precision and depth of insights derived from MMLA systems. The synergistic integration with existing EdTech tools further promotes interactive and engaging educational environments.
The systematic approach of this framework serves as a valuable framework for evaluating and refining existing systems. The framework’s emphasis on clearly defined educational objectives and comprehensive needs analysis provides a structured way to align established systems with evolving stakeholder requirements. In the context of data collection and management, the framework advocates for ethical practices and robust data management strategies. This implies that existing systems should undergo scrutiny to ensure transparent guidelines, user privacy, and secure multimodal datasets. The development phase encourages continuous improvement by leveraging advanced data interpretation, real-time feedback, and cutting-edge technologies. For existing systems, this translates to an ongoing commitment to innovation and compatibility with contemporary educational tools. The iterative refinement and validation phase is particularly pertinent for assessing existing systems. Through expert consultation and continuous improvement, the framework guides the evaluation and enhancement of established MMLA systems. Considering diverse learning scenarios and their impact contributes to ongoing assessments of system effectiveness and inclusivity. By incorporating ethical guidelines and transparent data practices, the framework promotes a responsible and ethical approach to assessing and advancing current MMLA systems.
The presented model shares similarities with existing frameworks such as the Multimodal Learning Analytics Model (MLeAM) (Di Mitri et al., 2018), the Multimodal Data-Value Chain (M-DVC) (Shankar et al., 2020), and DELICATE (Drachsler and Greller, 2016). However, MDF extends these models by providing a more comprehensive approach that integrates iterative refinement and validation phases, focusing on continuous feedback and real-world application alignment. Unlike MLeAM and M-DVC, which primarily emphasize the technical and analytical aspects of multimodal data, MDF also incorporates a strong emphasis on ethical considerations and privacy throughout the design process, similar to DELICATE but with a more detailed procedural framework. There is a need for a comprehensive review to identify which specific design considerations are addressed by each framework and how MDF uniquely contributes to advancing the field of MMLA through its holistic and iterative approach. This review would help clarify the distinct contributions of MDF and provide a clearer understanding of its practical implications and benefits.
7 Limitations
While our study provides valuable insights into the design considerations for MMLA systems, it is essential to acknowledge certain limitations that influence the interpretation and generalizability of our findings. The extent to which our interview questions capture the entire spectrum of MMLA design considerations may be subject to scrutiny. Although we designed questions to comprehensively cover relevant themes, there could be nuances or dimensions that participants did not explicitly address. Ensuring that our interview questions effectively measured the intended constructs related to MMLA design considerations is crucial. The interpretative nature of qualitative data introduces an inherent subjectivity that might impact the precision of construct measurement.
Our participant pool, while diverse in roles and experiences, primarily reflects a convenience sample. This approach was chosen for its practicality and ease of access to participants who are directly involved in the field of MMLA. Convenience sampling allows for quicker data collection and the inclusion of participants who have immediate relevance to the study’s objectives. However, it also has limitations, such as potential bias and lack of generalizability to the broader population of stakeholders, including policymakers and technology implementers. In future studies, employing alternative sampling methods, such as stratified or purposive sampling, could provide a more representative and comprehensive understanding of the diverse perspectives within the MMLA ecosystem. These methods would ensure that insights from a wider array of professionals are incorporated, thereby enhancing the robustness and applicability of the findings.
While we employed triangulation by integrating interviews and demographic data with existing literature, there could be inherent limitations in relying solely on participant perspectives and documented sources. Future research might explore additional data sources, such as observational studies or user interactions with existing MMLA systems, to enhance methodological triangulation. In acknowledging these limitations, we emphasize the need for ongoing research efforts to refine and expand our understanding of MMLA design considerations. Addressing these limitations will contribute to the robustness and applicability of future frameworks and models in the dynamic landscape of multimodal learning analytics.
8 Conclusion
This paper has provided an overview of design considerations for MMLA systems based on a semi-structured interview study with researchers in the field. The evolution of MMLA offers significant potential benefits for education; however, several challenges must be addressed to ensure their effectiveness in practice. By incorporating diverse perspectives from experts, this research reinforced previously identified key design considerations and identifying new emerging considerations that should be taken into account when developing MMLA systems. Most of the considerations contains several aspects which we presented in detail and provided its alignment with existing literature. The findings highlight both commonalities among researchers’ priorities as well as differences that require careful consideration during system development. Moreover, we developed MDF (the MMLA Framework for Design and Analysis) based on our results as a guide for designing and potentially analyzing existing MMLA systems. The insights gained from this study have broader implications for practitioners, researchers, and policymakers involved in the design and implementation of MMLA systems. In future research, we will focus on further exploring these design considerations and addressing potential limitations through a systematic literature review and using MDF to evaluate an authentic hybrid smart learning environment. Overall, this study contributes to the understanding and improvement of MMLA system design, ultimately enhancing educational outcomes.
Data availability statement
The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.
Ethics statement
Ethical approval was not required for the study involving human participants in accordance with the local legislation and institutional requirements. Written informed consent for participation in the study was obtained from the participants.
Author contributions
HO: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing. VB: Supervision, Validation, Writing – review & editing. DS: Supervision, Validation, Writing – review & editing.
Funding
The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2024.1356537/full#supplementary-material
Footnotes
References
Alwahaby, H., Cukurova, M., Papamitsiou, P., and Giannakos, M. (2021). The evidence of impact and ethical considerations of Multimodal Learning Analytics: A Systematic Literature Review. Available at: https://doi.org/10.35542/OSF.IO/SD23Y
Blikstein, P., Ochoa, X., and Blikstein, P. (2013) ‘Multimodal learning analytics’, ACM International Conference Proceeding Series, pp. 102–106. Available at: https://doi.org/10.1145/2460296.2460316
Celik, I., Gedrimiene, E., Silvola, A., and Muukkonen, H. (2022). Response of learning analytics to the online education challenges during pandemic: opportunities and key examples in higher education. Policy Futures Educ. 21, 387–404. doi: 10.1177/14782103221078401
Chejara, P. (2020) ‘Multimodal learning analytics for understanding collocated collaboration in authentic classroom settings’, Proceedings – IEEE 20th International Conference on Advanced Learning Technologies, ICALT 2020, pp. 390–392. Available at: https://doi.org/10.1109/ICALT49669.2020.00123
Cornide-Reyes, H., Noël, R., Riquelme, F., Gajardo, M., Cechinel, C., Mac Lean, R., et al. (2019). Introducing low-cost sensors into the classroom settings: improving the assessment in agile practices with multimodal learning analytics. Sensors (Switzerland) 19. doi: 10.3390/s19153291
Cukurova, M., Giannakos, M., and Martinez-Maldonado, R. (2020). The promise and challenges of multimodal learning analytics. Br. J. Educ. Technol. 51, 1441–1449. doi: 10.1111/BJET.13015
di Mitri, D., Schneider, J., Specht, M., and Drachsler, H. (2018). From signals to knowledge: a conceptual model for multimodal learning analytics. J. Comput. Assist. Learn. 34, 338–349. doi: 10.1111/jcal.12288
Drachsler, H., and Greller, W. (2016) ‘Privacy and analytics – it’s a DELICATE issue a checklist for trusted learning analytics’, in ACM International Conference Proceeding Series. Available at: https://doi.org/10.1145/2883851.2883893
Eradze, M., Rodríguez Triana, M. J., and Laanpere, M. (2020). ‘Context-aware multimodal learning analytics taxonomy’, CEUR Workshop Proceedings, 2610, pp. 1–6. Available at: http://ceur-ws.org/Vol-2610/paper1.pdf.
Garaizar, P., and Guenaga, M. (2014) ‘A Multimodal learning analytics view of HTML5 APIs: technical benefits and privacy risks’, in ACM International Conference Proceeding Series, pp. 275–281. Available at: https://doi.org/10.1145/2669711.2669911
Huertas Celdrán, A., Ruipérez-Valiente, J. A., García Clemente, F. J., Rodríguez-Triana, M. J., Shankar, S. K., and Martínez Pérez, G. (2020). A scalable architecture for the dynamic deployment of multimodal learning analytics applications in smart classrooms. Sensor 20. doi: 10.3390/s20102923
Junokas, M. J., et al. (2017) ‘Using one-shot machine learning to implement real-time multimodal learning analytics’, CEUR Workshop Proceedings, 1828, pp. 89–93. Available at: http://ceur-ws.org/Vol-1828/paper-13.pdf.
Kallio, H., Pietilä, A. M., Johnson, M., and Kangasniemi, M. (2016). Systematic methodological review: developing a framework for a qualitative semi-structured interview guide. J. Adv. Nurs. 72, 2954–2965. doi: 10.1111/JAN.13031
Melles, M., Albayrak, A., and Goossens, R. (2021). Innovating health care: key characteristics of human-centered design. Int. J. Qual. Health Care 33, 37–44. doi: 10.1093/INTQHC/MZAA127
Mu, S., Cui, M., and Huang, X. (2020). Multimodal data fusion in learning analytics: a systematic review. Sensors (Switzerland) 20, 1–27. doi: 10.3390/s20236856
Nguyen, A., Tuunanen, T., Gardner, L., and Sheridan, D. (2020). Design principles for learning analytics information systems in higher education. Taylor & Francis 30, 541–568. doi: 10.1080/0960085X.2020.1816144
Nissenbaum, H. (2009). Privacy in context: Technology, policy, and the integrity of social life. Privacy in Context. Stanford University Press.
Noel, R., Riquelme, F., Lean, R. M., Merino, E., Cechinel, C., Barcelos, T. S., et al. (2018). Exploring collaborative writing of user stories with multimodal learning analytics: a case study on a software engineering course. IEEE Access 6, 67783–67798. doi: 10.1109/ACCESS.2018.2876801
Ochoa, X., Worsley, M., Weibel, N., and Oviatt, S. (2016). Multimodal learning analytics data challenges. in proceedings of the sixth international conference on learning analytics & knowledge (LAK ‘16). New York, NY, USA: Association for Computing Machinery. 498–499. doi: 10.1145/2883851.2883913
Ouhaichi, H., Olsson, H. H., and Bosch, J. (2019). “Dynamic data management for machine learning in embedded systems: a case study” in Software Business. ICSOB 2019. notes in business information processing. eds. S. Hyrynsalmi, M. Suoranta, A. Nguyen-Duc, P. Tyrväinen, and P. Abrahamsson (Cham: Springer), 145–154.
Ouhaichi, H., Spikol, D., and Vogel, B. (2021) ‘MBOX: designing a flexible IoT multimodal learning analytics system’, in Proceedings – IEEE 21st International Conference on Advanced Learning Technologies, ICALT 2021, pp. 122–126. Available at: https://doi.org/10.1109/ICALT52272.2021.00044
Ouhaichi, H., Spikol, D., and Vogel, B. (2023a). Research trends in multimodal learning analytics: a systematic mapping study. Comput. Educ. Artif. Intell. 4:100136. doi: 10.1016/J.CAEAI.2023.100136
Ouhaichi, H., Spikol, D., and Vogel, B. (2023b) ‘Rethinking MMLA: design considerations for multimodal learning analytics systems’, in L@S 2023 – Proceedings of the 10th ACM Conference on Learning @ Scale. Available at: https://doi.org/10.1145/3573051.3596186
Oviatt, S. (2018) ‘ICMI – ten opportunities and challenges for advancing student-centered multimodal learning analytics’, in Proceedings of the 20th ACM International Conference on Multimodal Interaction. ACM, pp. 87–94. Available at: https://doi.org/10.1145/3242969.3243010
Oviatt, S., Grafsgaard, J., Chen, L., and Ochoa, X. (2018) ‘Multimodal learning analytics: assessing learners’ mental state during the process of learning’, in The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations – Volume 2, pp. 331–374. Available at: https://doi.org/10.1145/3107990.3108003
Prieto, L. P., Sharma, K., Kidzinski, Ł., Rodríguez-Triana, M. J., and Dillenbourg, P. (2018). Multimodal teaching analytics: automated extraction of orchestration graphs from wearable sensor data. J. Comput. Assist. Learn. 34, 193–203. doi: 10.1111/JCAL.12232
Riquelme, F., Munoz, R., Mac Lean, R., Villarroel, R., Barcelos, T. S., and de Albuquerque, V. H. C. (2019). Using multimodal learning analytics to study collaboration on discussion groups: a social network approach. Univ. Access Inf. Soc. 18, 633–643. doi: 10.1007/s10209-019-00683-w
Runeson, P., et al. (2012) Case study research in software engineering: guidelines and examples, Available at: https://doi.org/10.1002/9781118181034
Shankar, S. K., Ruiz-Calleja, A., Prieto, L. P., Chejara, P., Martinez-Mones, A., and Martinez-Mones, A. (2020). Multimodal data value chain (M-DVC): a conceptual tool to support the development of multimodal learning analytics solutions. IEE Rev. Iberoam. Tecnol. Aprendiz. 15, 113–122. doi: 10.1109/RITA.2020.2987887
Shum, S. B., Ferguson, R., and Martinez-Maldonado, R. (2019). Human-centred learning analytics. J. Learn. Anal. 6, 1–9. doi: 10.18608/jla.2019.62.1
Smith, H. J., Dinev, T., and Xu, H. (2011). Information privacy research: an interdisciplinary review. MIS quarterly. 989–1015.
Tamura, K., et al. (2019) ‘Integrating multimodal learning analytics and inclusive learning support Systems for People of all ages, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11577 LNCS, pp. 469–481. Available at: https://doi.org/10.1007/978-3-030-22580-3_35
Ulrich, K. T., and Eppinger, S. D. (2016). Product design and development. Available at: https://thuvienso.hoasen.edu.vn/handle/123456789/9147 (Accessed May 27, 2024).
Wise, A. F., and Vytasek, J. (2017). “Learning analytics implementation design” in Handbook of Learning Analytics, 151–160. Available at: https://doi.org/10.18608/HLA17.013
Worsley, M. (2012) ‘Multimodal learning analytics – enabling the future of learning through multimodal data analysis and interfaces’, in ICMI’12 – Proceedings of the ACM International Conference on Multimodal Interaction, pp. 353–356. Available at: https://doi.org/10.1145/2388676.2388755.
Worsley, M. (2018) ‘Multimodal learning analytics’ past, present, and, potential futures’, in CEUR Workshop Proceedings.
Keywords: multimodal learning analytics, design considerations, data mining in education, MMLA design, educational technology
Citation: Ouhaichi H, Bahtijar V and Spikol D (2024) Exploring design considerations for multimodal learning analytics systems: an interview study. Front. Educ. 9:1356537. doi: 10.3389/feduc.2024.1356537
Edited by:
Malgorzata Korolkiewicz, University of South Australia, AustraliaReviewed by:
Yannis Dimitriadis, University of Valladolid, SpainPankaj Chejara, Tallinn University, Estonia
Copyright © 2024 Ouhaichi, Bahtijar and Spikol. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Hamza Ouhaichi, aGFtemEub3VoYWljaGlAbWF1LnNl