Jiejin Luo- Graduate School of Human Sciences, Osaka University, Suita, Osaka, Japan
This research focuses on validating the impact of gamified technology-enhanced learning environments (TELEs) on motivation and academic performance. The study aims to categorize and design digital badges based on game mechanics and to evaluate the effects of two distinct types of digital badges on learning outcomes. The research involved 95 university students learning classical Japanese grammar. The findings reveal that digital badges significantly enhance learners’ intrinsic motivation, positively affecting all five dimensions of intrinsic motivation, while their impact on extrinsic motivation was found to be minimal. Additionally, when comparing the two categories of digital badges, no significant differences in effectiveness were observed. The primary interest of this paper lies in exploring the design and efficacy of digital badges within technology-enhanced learning environments, contributing valuable insights into how these elements can enhance student engagement and learning outcomes.
1 Introduction
Under the relentless impetus of evolving technological innovations, the contemporary educational landscape is undergoing a profound transformation. Technology Enhanced Learning Environments (TELEs) have emerged as a significant force, characterized by their strategic integration of technology-based educational systems designed to facilitate knowledge acquisition and competency development. TELEs consist of components such as pedagogical content (learning objectives, tasks, and instructional materials), interactive roles (mentors, educators, and peers), and a technological infrastructure (Dettori, 2009). In recent years, the rapid advancement of new technologies has coincided with increased interest in TELEs, perceived as ushering in a new era of educational transformation. However, while technology opens new learning possibilities, it is essential for educators to avoid becoming overly captivated by the allure of innovation and to critically examine its impacts on the psychology of learning and instructional outcomes (Bower, 2017). Achieving meaningful integration requires understanding the pedagogical roles of digital technologies and validating their impact with practical examples (Marín et al., 2024).
At the intersection of TELEs and game-based learning lies Gamified Technology-Enhanced Learning Environments (gamified TELEs), which integrate game design elements into educational activities. These elements, such as narratives, challenges, and rewards, aim to create engaging and motivating learning experiences by transforming conventional learning environments into immersive gaming spaces (Deterding et al., 2011). Gamified TELEs foster both intrinsic and extrinsic motivation, aiming to enhance the learning effect by stimulating critical thinking and reinforcing achievements. However, despite the recognized potential advantages of game-based education, there is a notable research gap concerning empirical quantitative studies on its impact on educational outcomes (Mayer, 2014). Further, the existing literature lacks sufficient analysis on what specific game design elements might better promote learning effects in educational settings.
Digital badges have become prominent in gaming, with their inclusion in game design dating back to milestones like Xbox Live Gamerscore (Microsoft, 2005), PlayStation Network Trophies (Sony, 2008), and Steam Achievements (Valve, 2008) (Dickey, 2005). As a gamification method, digital badges serve as rewards that signify milestones or achievements, motivating players to pursue more badges and reach higher goals (Hakulinen et al., 2013). In education, digital badges, also known as digital certificates or micro-credentials, represent skills or achievements, allowing learners to showcase their accomplishments on digital platforms. These badges include details like the issuer, criteria for earning the badge, date of issuance, and evidence of achievement (Muilenburg and Berge, 2016). While digital badges serve as motivational tools by recognizing and rewarding learners’ efforts (Gibson et al., 2015), questions remain about their effectiveness compared to traditional rewards, such as money or tangible goods, and the impact of different badge designs in educational contexts (Lu et al., 2023).
This paper aims to investigate the impact of gamified TELEs on learning outcomes, focusing on learner motivation and academic performance. The study will compare in-game digital badges with traditional rewards, examining their similarities and differences in educational models. Additionally, it will analyze the impact of various badge designs to determine differences in their effects on learning outcomes. By reviewing existing literature, conducting empirical research, and focusing on badge design within gamified TELEs, this paper seeks to address gaps in current research and advance understanding of how digital badges can optimize TELEs, ultimately creating more engaging and effective blended learning environments.
2 Literature review
2.1 Gamified technology-enhanced learning environments
Technology-Enhanced Learning (TEL) is a comprehensive term encompassing all methods that utilize technology to support the learning and teaching process. This includes e-learning, online learning, and digital game-based learning, all of which fall under the umbrella of TEL research (Dror, 2008). In the current landscape of TEL in higher education, while there is extensive research on virtual learning environments (VLEs), few studies connect VLEs to student agency. Various technological tools in TEL environments foster active learning and create opportunities for students to exercise their agency. Personal Learning Environments (PLEs) shift control from educators to learners, allowing them to select their own tools and cultivate autonomy. The digital teaching framework advocates for active participation and self-regulated learning, emphasizing the impact of design on educational outcomes (Marín et al., 2024). Consequently, Digital Game-Based Learning (DGBL) emerges as an important tool within TEL, making it crucial to study how design within TEL environments maximizes educational outcomes (Homer et al., 2020).
Digital Game-Based Learning (DGBL) refers to the use of games in educational contexts, and its evolution reflects how games can influence learning and their potential as educational tools. According to Plass et al. (2015), the focus of DGBL has shifted over the years, highlighting its educational possibilities. The concept of Digital Game-Based Learning was introduced by Prensky (2003) as an instructional approach that merges serious learning with interactive entertainment, leading to changes in players’ knowledge, skills, or cognitive outcomes through gameplay. DGBL has evolved over the years, showcasing how games can influence learning and their potential as educational tools. In the early 1970s to 1980s, video games became subjects of psychological research, with studies focusing on their effects on cognitive abilities, as seen in games like Pong and Atari 2600 (Kent, 2010). This period marked the initial recognition of games not just as entertainment but also as potential educational instruments. In the 1980s, theoretical frameworks emerged regarding the motivational impact of games. Malone (1981) identified challenge, fantasy, and curiosity as key elements that promote intrinsic motivation in learners. These concepts significantly shaped the development of educational games by fostering student engagement. The late 1980s and early 1990s saw the rise of personal computers, leading to the creation and validation of educational games such as Oregon Trail and Carmen Sandiego for their educational effectiveness (Shuler, 2012). These games demonstrated that learning could be enjoyable and effective. By the mid-1990s, the educational potential of commercial games began to be acknowledged. Titles like SimCity helped learners grasp complex concepts through simulation (Kim and Shin, 2016; Minnery and Searle, 2014). This period explored how commercial games could enhance understanding in educational settings. Since the 2000s, the widespread availability of broadband internet facilitated the rapid development of online multiplayer games. Notably, World of Warcraft was utilized in educational environments to cultivate communication and problem-solving skills among learners (Delwiche, 2006; Nardi and Harris, 2006). These games offered new educational possibilities by promoting social skills and teamwork through collaborative learning experiences.
In summary, as digital games have evolved and gained societal traction, they have become increasingly integrated into education, resulting in the development of educational games aimed at enhancing skills, recognition of non-educational games in improving cognitive abilities, and fostering intrinsic motivation for learners to engage in gameplay that enhances learning.
In this paper, Gamified Technology-Enhanced Learning Environments (G-TELEs) are defined as digital learning spaces that incorporate game design elements and mechanics into non-game contexts. Research on G-TELEs can be broadly categorized into two main areas. The first area examines the effectiveness of game-based instructional environments, primarily focusing on learning motivation and engagement, comparisons with traditional media, and cognitive domains of learning comprehension (Erhel and Jamet, 2013). Previous to recent studies have confirmed that Digital Game-Based Learning (DGBL) increases learners’ motivation and engagement in learning (Greenblat, 1981; Hays, 2005; Greenblat, 1981; Nadeem et al., 2023; Zheng et al., 2024). Comparative studies demonstrate the advantages of DGBL over traditional classroom learning (Hays, 2005; Vogel et al., 2006), and Clark et al. (2016) systematically reviewed digital games and learning research targeting K-16 students, revealing that digital games significantly improve student learning outcomes compared to non-game conditions. Thus, the application of G-TELEs primarily involves various aspects of learning success (Schweighofer and Ebner, 2015). The second category of research focuses on the technological aspects of game-based instructional environments, emphasizing the development and design of G-TELEs and integrating technology into learning and teaching contexts. For example, Ketelhut and Schifter (2011) created a game-based multi-user virtual environment (MUVE) science curriculum project called the “River City” project for middle school students. This study employs a cross-case analysis approach and a professional development model to assess the ongoing evolution of professional development related to the River City project, illustrating the strategic integration of educational technology and pedagogy in immersive virtual environments while addressing the critical need for effective professional development to bridge technological gaps between teachers and students.
Despite extensive research on the effectiveness of game-based learning in Technology-Enhanced Learning Environments (TELEs) and design research, there is insufficient evidence regarding which game design elements most effectively promote learning outcomes. Rewards, particularly digital badges as mechanisms for acknowledging and rewarding learners’ achievements (Gibson et al., 2015), have gained widespread recognition, yet a lack of quantitative research exists demonstrating their precise impact on learning outcomes within G-TELEs. Thus, this study focuses on validating digital badges as a game design element. In game research, the value-added approach is commonly used to explore the impact of various design elements on game-based learning outcomes, comparing the learning outcomes of students who learn by playing a game to those assigned to play the same game with an additional instructional feature (Mayer, 2014). This approach allows for understanding which factors within a game influence learning outcomes, enabling better game design tailored to learning content and learner characteristics to enhance learning effectiveness. Therefore, this study adopts the value-added approach, introducing different types of digital badges into an existing foundational game environment and quantitatively analyzing their impact on learning outcomes to determine whether digital badges can indeed enhance gamified technology-enhanced learning environments.
2.2 Digital badges as motivational tools
Digital badges (DB), also referred to as open digital badges (ODBs) or open badges, have emerged as a new online educational assistive technology. Various definitions exist, such as Alexandra (2013) description of digital badges as “validated indicators of accomplishment, skill, quality, or interest that can be earned in diverse learning environments,” while Gibson et al. (2015) characterizes them as representations of accomplishments, interests, or affiliations that are visual, available online, and contain metadata explaining the context, meaning, process, and result of an activity. Finkelstein et al. (2013) defines digital badges as ways to capture and communicate an individual’s skills, representing different levels of engagement and achievements. Synthesizing these definitions, several common points emerge: digital badges must be online, visual, and represent competencies linked to specific activities. These shared features also illuminate the associated benefits of digital badges.
The online features of digital badges enable self-directed learning, supporting anytime, anywhere access and providing economically disadvantaged students in higher education with new certification pathways (Grant, 2016). Their visibility enhances progress tracking and serves as a motivational tool by incorporating visual elements that stimulate learners’ visual senses, thereby improving learning efficiency (Wang et al., 2022). The use of digital badges to showcase achievements recognizes a variety of learning experiences in both formal and informal environments. Research by McDaniel et al. (2012) shows that digital badges are particularly effective in encouraging timely exam participation and providing constructive feedback on assignments, which reinforces students’ motivation to engage actively. Furthermore, badges as feedback tools provide immediate, recognizable rewards that bridge the gap between effort and acknowledgment, thereby fostering motivation and engagement (Goulding et al., 2024).
Digital badges help establish strong connections between learning objectives, student engagement, and progress tracking, allowing for a personalized learning experience. This relationship enables badges to serve as markers of achievement that not only recognize progress but also motivate continued efforts toward mastery. Research by O’Donovan et al. (2013) demonstrates that when badges are combined with visual elements such as progress bars and narrative storylines, they can enhance outcomes like attendance, participation, content understanding, problem-solving, and overall engagement. By rewarding learners for reaching milestones, badges help reinforce desired behaviors such as class attendance or active involvement in discussions, which encourages deeper engagement with learning. Schoenenberger (2024) supports these findings, emphasizing that badges provide immediate recognition, fostering motivation and creating a competitive yet collaborative learning environment.
The wide range of digital badge designs presents a challenge in establishing a universally accepted classification system. While their positive impact on learning is acknowledged, a clearer classification framework is needed to understand how different types of badges influence learning motivation and outcomes. Additionally, quantifying the effects of various badge designs is essential to grasp their distinct contributions in educational contexts. Badge designs can be categorized by learning objectives: absolute evaluation badges focus on mastery of skills against standards, whereas relative evaluation badges promote achievement by comparison with peers (Dweck, 1986). These categories aim to facilitate different aspects of student motivation and educational goals.
Absolute evaluation badges assess performance against predetermined criteria, similar to gaming trophies like those in Grand Theft Auto and World of Warcraft (Montola et al., 2009). Although these systems enhance gaming enjoyment, some research indicates that when viewed as mere rewards, they can reduce intrinsic motivation due to competition (Chan et al., 2024). In contrast, relative evaluation badges foster competition by comparing performance with peers, such as the seasonal ranking in Apex Legends. Zhang and West (2023) used competency-based badges to monitor teaching skills development in a multimedia lab, showing their potential for tracking progress while highlighting the need to increase their relevance for participants. While absolute badges offer consistent benchmarks, relative badges drive motivation through competition. Strategically combining both types could benefit educational settings, addressing gaps in understanding how game-based badge classifications influence motivation and learning.
2.3 Integration of digital badges in gamified TELEs
By examining the factors of rewards and badge types on intrinsic motivation, extrinsic motivation, and academic performance, this study delves into the application effects of digital badges within gamified Technology-Enhanced Learning Environments (G-TELEs).
2.3.1 The comparison of reward types
In analyzing the effects of reward types on motivation, digital badges are typically provided as rewards upon achievement. Hence, it is essential to compare them with material rewards and no rewards to assess their impact on motivation.
Firstly, the study initiates the comparative analysis with a focus on intrinsic motivation. According to Cognitive Evaluation Theory (Deci et al., 1975; Deci and Ryan, 1980; Deci and Ryan, 1985; Ryan and Deci, 2000), rewards consist of two aspects: informational and controlling. Deci and Ryan (1980) assert that the impact of rewards on intrinsic motivation depends on the dominance of these two aspects. When control is predominant, intrinsic motivation declines, whereas when information is predominant, intrinsic motivation increases. Since digital badges convey substantial information, their use enhances intrinsic motivation, strengthening their influence on the informational aspect. Furthermore, an in-depth analysis of intrinsic motivation reveals that digital badges, as gamification elements, can stimulate curiosity and excitement. The visual presentation of badges upon attainment enhances learners’ desire to pursue subsequent badges. Digital badges provide clear evidence of learners’ achievements, promoting their perception of competence when badges symbolize mastery or acquired skills.
Moreover, by associating badges with specific learning tasks or milestones, learners are more inclined to exert effort and recognize the significance and value of their efforts. However, it’s essential to consider the potential impact of badges on learners’ sense of pressure and tension. In certain situations, badge introduction might inadvertently lead to performance-related stress or excessive focus on external rewards. Nevertheless, compared to material rewards, badges typically induce less stress because they are unrelated to tangible items. Digital badges have the potential to enhance learners’ perception of the value and utility of their learning experiences. When badges align with real-world skills, competencies, or goals, learners recognize the practical applicability of their learning and appreciate its relevance to their personal or professional lives.
Material rewards are often criticized for potentially undermining intrinsic motivation. If individuals perceive their participation in an activity solely as a means to obtain external rewards, their intrinsic motivation and enjoyment of the task may diminish. This phenomenon is known as the “over justification effect,” indicating that external rewards can sometimes weaken intrinsic motivation. However, when material rewards serve as external reinforcement, acknowledging and reinforcing positive behavior or achievements, especially when material rewards and feedback are closely related, they can enhance intrinsic motivation. In this scenario, material rewards act as positive feedback, indicating that individual efforts are recognized and valued (Deci et al., 1999).
According to Cognitive Evaluation Theory (Deci et al., 1975; Deci and Ryan, 1980; Deci and Ryan, 1985), the influence of rewards on intrinsic motivation is governed by two primary dimensions: informational and controlling. Rewards perceived as controlling tend to undermine intrinsic motivation by exerting pressure on individuals to conform to specific behaviors. In contrast, rewards that are primarily informational enhance intrinsic motivation by providing feedback about competence and progress. Digital badges exemplify the latter, as they offer substantial informational value through their visual representation and their direct association with the learning process. This focus on information reinforces their positive impact on intrinsic motivation, making digital badges akin to tangible rewards that evoke a sense of achievement. In the context of digital badges, intrinsic motivation reflects how learners perceive the attainment of these badges as beneficial to their learning outcomes (Pangaribuan et al., 2021). This intrinsic motivation is fueled by internal factors and personal satisfaction, symbolizing an individual’s recognition of their own accomplishments. Thus, while digital badges may also serve as external rewards, their underlying structure and purpose facilitate a deeper sense of personal achievement and motivation in learners. By effectively balancing informational content with the desire for recognition, digital badges not only enhance engagement but also promote a lasting commitment to learning.
As Razhkou (2024) points out, games provide external rewards like achievements, trophies, and prizes, which encourage continued play. When players complete tasks, they experience a combination of satisfaction and reward. The sense of fulfillment upon completing an external task creates an intrinsic feeling of success, driving the desire to repeat the process. Game mechanics can thus support both intrinsic and extrinsic engagement simultaneously. Motivational elements like challenges are particularly powerful, as they are closely linked to a state of flow (Csikszentmihalyi, 1988). This state, characterized by deep immersion and enjoyment in an activity, can significantly enhance intrinsic motivation. Therefore, incorporating digital badges as part of learning strategies may not only provide external incentives but also deeply engage learners by tapping into their internal drive for achievement and growth.
Applying the Intrinsic Motivation Inventory (IMI) for an in-depth analysis of intrinsic motivation, it is evaluated across several scales (Monteiro et al., 2015): Interest/Enjoyment, Perceived Competence, Effort/Importance, Pressure/Tension, Value/Usefulness. The validity and reliability of the IMI have been well-established across various studies and contexts. Ostrow and Heffernan (2018) conducted an extensive review, confirming that the IMI maintains high internal consistency and stability across diverse settings, including educational and experimental research. Their validation involved examining the instrument’s factor structure, convergent validity (alignment with other established measures of motivation), and discriminant validity (the ability to differentiate between related but distinct constructs). The IMI’s reliability was demonstrated through consistent results across multiple samples and repeated administrations, ensuring that the scales accurately reflect different components of intrinsic motivation. The IMI’s robust validation supports its widespread use as a reliable measure for assessing intrinsic motivation in both educational and psychological research. This makes it a valuable tool for understanding how different interventions, such as digital badges or other motivational strategies, impact various aspects of learners’ motivational experiences.
Interest/Enjoyment: Digital badges, as gamification elements, can spark curiosity and excitement. The visual presentation of badges after attainment enhances learners’ desire to pursue subsequent badges.
Perceived Competence: Digital badges provide clear evidence of learners’ achievements, enhancing their perception of competence when badges symbolize mastery or acquisition of skills. Effort/Importance linking badges to specific learning tasks or milestones encourages learners to exert effort and recognize the significance and value of their efforts.
Value/Usefulness: Digital badges may enhance learners’ perception of the value and usefulness of their learning experiences. When badges align with real-world skills, abilities, or goals, learners recognize the practical application of their learning and appreciate its relevance to personal or professional life. Examining these four scales, digital badges and positive material rewards have similar effects, enhancing several dimensions of intrinsic motivation. However, considering the dimension of Pressure/Tension, the introduction of material rewards may inadvertently lead to performance-related pressure or excessive focus on external rewards in certain situations. Nevertheless, compared to material rewards, badges typically do not induce such pressure, as they are unrelated to tangible items.
On the other hand, extrinsic motivation involves participating in learning tasks to obtain external rewards, which act as positive reinforcement for the desired behavior (Filgona et al., 2020). Examples of such motivations include tuition subsidies, competitions among peers, student grading, research grants, as well as completing tests and assignments (Bandhu et al., 2024). Digital badges are a typical form of extrinsic incentive, providing learners with external factors that can boost morale. Learners may be drawn to the skills associated with earning badges (Robert et al., 2024). When examining the impact of different reward types on extrinsic motivation, research suggests that digital badges can significantly enhance extrinsic motivation when used as rewards (Shields and Chugh, 2017). By providing visible recognition of learners’ accomplishments, digital badges satisfy the need for external validation, which can lead to increased engagement and effort. This recognition encourages learners to strive for more badges, thereby reinforcing extrinsic motivation and driving continued participation in learning activities.
On the other hand, material rewards, such as tangible items or monetary compensation, have long been considered effective external incentives (Deci et al., 1999). Learners are motivated by the prospect of obtaining tangible rewards, especially when these rewards align with their needs or desires. Various fields, including education and business, frequently adopt this form of external motivation.
In conclusion, both digital badges and material rewards can meet learners’ needs for external recognition and motivation, effectively enhancing both intrinsic and extrinsic motivation. In summary, digital badges are rewards that have the same effect as positive material rewards (i.e., material rewards with feedback), elevating learner performance by boosting both intrinsic and extrinsic motivation. Additionally, digital badges have a slight advantage over material rewards; in evaluating the dimension of Pressure/Tension related to intrinsic motivation, digital badges, as virtual badges, do not impose pressure on learners.
2.3.2 The comparison of evaluation types
Relative evaluation badges, as indicated by Festinger’s (1954) Social Comparison Theory, involve comparing learners to their peers, fulfilling humans’ needs for social recognition and achievement. Previous research by Deterding et al. (2011) suggests that these badges create a competitive environment where learners strive to outperform their peers. This competitive motivation satisfies the need for social recognition and the desire to excel within the community, driving intrinsic motivation. In analyzing the effects of evaluation types on extrinsic motivation, absolute evaluation digital badges are typically awarded based on specific, predefined criteria or achievements (Abramovich et al., 2013). These badges explicitly recognize learners’ accomplishments and provide a clear, tangible reward for reaching specific goals. Moreover, learners who receive absolute evaluation badges can compare their achievements to the predefined standards or expectations. This external comparison provides a benchmark for success, thereby enhancing extrinsic motivation (Hamari et al., 2014).
In contrast, relative evaluation badges often emphasize peer comparisons. These badges consider learners’ achievements relative to their peers, fostering competition and social comparison. Learners can earn relative evaluation badges based on their performance compared to others within a group. Furthermore, relative evaluation badges can increase extrinsic motivation by introducing a competitive element (Deterding et al., 2011). Learners may strive to outperform their peers to earn badges, thereby increasing their effort and engagement. While both types of badges enhance extrinsic motivation, they operate through different psychological mechanisms. Absolute evaluation badges emphasize individual achievements and goal attainment, whereas relative evaluation badges harness the desire for competition and social recognition. Therefore, in conclusion, when comparing different evaluation types of digital badges, namely absolute evaluation and relative evaluation, it can be inferred that relative evaluation digital badges are more effective in enhancing learners’ intrinsic motivation, while both types contribute to an increase in extrinsic motivation.
2.4 Hypothesis
Based on the exploration of digital badges within gamified Technology-Enhanced Learning Environments (G-TELEs), the following integrated hypothesis on reward types can be proposed:
Hypothesis on Reward Types and Motivation: Digital badges, as a form of informational reward, enhance both intrinsic and extrinsic motivation more effectively than material rewards or the absence of rewards. By providing feedback on learners’ competence and progress, digital badges stimulate curiosity and excitement, fostering a desire for further engagement and reinforcing the perception of mastery and skill acquisition. Unlike material rewards, which may lead to performance-related pressure and stress, digital badges promote a sense of accomplishment without inducing the same level of tension. Additionally, they offer visible recognition of achievements, fulfilling learners’ needs for external validation and encouraging increased engagement and effort in learning activities. Overall, digital badges effectively balance the enhancement of intrinsic motivation with the fulfillment of extrinsic motivational needs, reinforcing learners’ commitment to the educational process.
Hypothesis on Evaluation Types: Relative evaluation badges are more effective than absolute evaluation badges in fostering intrinsic motivation among learners. Relative evaluation badges, which emphasize peer comparisons, tap into the competitive nature of learners, fulfilling their need for social recognition and achievement. In contrast, absolute evaluation badges focus on individual accomplishments, which may enhance extrinsic motivation but not to the same extent as relative evaluation badges enhance intrinsic motivation.
3 Methodology
3.1 Sites and participants of experiment
The study involved 95 first-year Chinese university students who were enrolled in either Japanese or English majors at the same institution. Both groups had been learning Japanese at a comparable pace, resulting in an equivalent level of proficiency in the language. None of the participants had prior exposure to classical Japanese grammar, which was the primary subject of the instructional materials used in the experiment. Therefore, all students were considered beginners in the study of classical Japanese grammar. The participants were equally divided by gender (47 males and 48 females). Previous studies on motivation in similar educational contexts have not indicated significant gender-related differences, so gender effects were not expected to influence the outcomes of this experiment. The group was gathered in a classroom setting, where they participated in the study for one class period (within a maximum of 90 min). They accessed the online learning activities, which included digital game-based tasks, via their personal mobile devices (either smartphones or laptops). Given that classical Japanese grammar is a distinct subject that requires new learning separate from modern Japanese language skills, it was assumed that participants’ prior Japanese knowledge would not affect their performance in this experiment. Thus, all participants were starting from a similar baseline with regard to the content covered.
3.2 Research tools
The game provided in this study is a text-based adventure comprising four essential components: a background story, a knowledge introduction section, a knowledge application section through practice exercises, and a final test. The subject matter focuses on classical Japanese, and the choice of classical grammar aligns with the study’s objectives, aiming to investigate the impact of digital badges on motivation and performance. By deliberately selecting a task with relatively weak intrinsic motivation, variations in learning outcomes become more pronounced and easier to analyze. Based on Haruguchi (2010) survey, where only 15 out of 37 students in a classical Japanese grammar class at a Chinese university claimed to be “very motivated” during class, this confirms classical Japanese as a low-motivation task, with less than half of voluntary participants expressing high motivation.
The foundational game structure involves students initially learning classical Japanese grammar through a story relevant to classical Japanese language learning. The grammar section introduces five rules governing sound changes in classical Japanese when translated into modern Japanese. After each rule introduction, exercises aid in memory retention. Upon completing the learning of all five rules, a transfer test is conducted. This test consists of 20 questions, providing ancient Japanese vocabulary. Applying the learned rules, students must convert the pronunciation of these classical Japanese words into modern Japanese to assess memory retention and application. Each question carries 5 points, totaling 100 points. Following the completion of the transfer test, three types of rewards are offered: no reward, material reward, and digital badge reward. The material reward comprises a ¥500 voucher, while the digital badge reward includes a badge containing information signifying completion of the learning. Evaluation of the transfer test involves both absolute and relative assessments based on students’ scores and rankings, respectively. The game is segmented into six versions, each corresponding to six distinct groups based on these two primary factors. The variations include no-reward absolute evaluation, no-reward relative evaluation, material reward absolute evaluation, material reward relative evaluation, digital badge reward absolute evaluation, and digital badge reward relative evaluation. Each group receives feedback and, if applicable, rewards based on their performance in the final test. In the no-reward absolute evaluation group, learners receive their individual scores as feedback upon completing the test but do not receive any rewards. Similarly, in the no-reward relative evaluation group, learners receive feedback in the form of their ranking among all participants without any accompanying rewards. For the material reward absolute evaluation group, learners receive their individual scores as feedback after completing the test. If their score is 90 or higher, students are eligible to receive a ¥500 voucher as a reward. In the material reward relative evaluation group, learners receive feedback regarding their ranking among all participants after completing the test. Those ranking in the top three are entitled to a ¥500 voucher as a reward. As for the digital badge reward absolute evaluation group, learners receive their individual scores as feedback after completing the test. Achieving a score of 90 or above allows students to receive a digital badge as a reward. Lastly, in the digital badge reward relative evaluation group, learners receive feedback on their ranking among all participants after completing the test, with the top three ranked students earning a digital badge as a reward.
Specific examples of the game’s design are illustrated in the subsequent experimental procedure. These examples highlight how various game elements were structured and integrated into the experimental process, providing a clear picture of how the game was used as a tool to achieve the research objectives. The design details, including game mechanics, tasks, and participant interactions, are laid out step by step in the following sections to demonstrate their role in shaping the experimental flow and facilitating the data collection process.
3.3 Data collection
The data collection for the experiment consisted of two parts: a pre-questionnaire and a post-questionnaire. The pre-questionnaire aimed to assess the learners’ prior knowledge to ensure a uniform baseline understanding of the topics among all participants. In contrast, the post-questionnaire focused on evaluating the learners’ intrinsic motivation, utilizing the Intrinsic Motivation Inventory (IMI) measurement scale (Ryan et al., 1983). Intrinsic motivation was measured in this study using a 7-point Likert scale, examining five distinct dimensions through the IMI scale, which captures various aspects of intrinsic motivation.
Below are detailed descriptions of the five dimensions within the IMI scale:
Interest/Enjoyment: This dimension assesses how interesting and enjoyable learners find the activity. Participants rated their enjoyment and personal interest in the learning experience on a scale from 1 (not at all enjoyable/interesting) to 7 (very enjoyable/interesting), with items like “This activity was fun to do.”
Perceived Competence: This dimension evaluates learners’ perceptions of their own competence and effectiveness in completing the learning tasks. Participants rated how capable they felt during the activities on a scale from 1 (not at all competent) to 7 (very competent), with items such as “I think I am pretty good at this activity.”
Effort/Importance: The Effort/Importance dimension measures the effort learners exert and the significance they attach to the learning tasks. Participants indicated their level of effort and the importance they assigned to the activities on a scale from 1 (very little effort/very unimportant) to 7 (a lot of effort/very important), with items like “I put a lot of effort into this.”
Pressure/Tension: This dimension assesses the pressure or tension learners feel while engaging in the learning activities. Participants rated their experience of pressure or tension on a scale from 1 (not at all pressured/tense) to 7 (very pressured/tense), with items such as “I felt pressured while doing these.”
Value/Usefulness: The Value/Usefulness dimension evaluates learners’ perceptions of the value and usefulness of the learning content and activities. Participants rated the perceived value and usefulness of the tasks on a scale from 1 (not at all valuable/useful) to 7 (very valuable/useful), with items like “I believe this activity could be of some value to me.”
Collectively, these five dimensions offer a thorough insight into learners’ intrinsic motivation, highlighting various aspects of their engagement with the learning activities. By examining dimensions such as engagement, enjoyment, competence, effort, perceived pressure, and perceived value, the IMI scale enables researchers to capture a holistic view of how learners interact with educational content. This multifaceted approach not only reveals the depth of learners’ intrinsic motivation but also identifies specific areas where enhancements can be made to improve the overall learning experience.
The reliability of the IMI scale has been previously established through prior research, which has empirically validated its consistency and trustworthiness. These studies confirm that the IMI scale is a robust tool for measuring intrinsic motivation, ensuring that the insights derived from its application are both accurate and meaningful in understanding learners’ experiences.
For extrinsic motivation, the study employed the extrinsic motivation scale developed by Buckworth et al. (2007). Their research found that the overall extrinsic motivation inventory (EMI) demonstrated good internal consistency (α = 0.75) and adequate reliability (α = 0.70).
3.4 Experiment process
Step 1: Explain the experiment to the participants and have them sign the consent form.
Before commencing the experiment, participants were provided with a detailed explanation of the study, and they were asked to sign the consent form. Figure 1 illustrates the game’s consent form interface, encompassing the experimental procedure sequence, the duration allocated to each step, and the precautions associated with participating in the experiment.
Figure 1. For Step 1 The game’s consent form interface.
The consent form, available in both Chinese and Japanese, outlined the purpose of the study, emphasizing its focus on assessing the learning effects of the game. The experimental procedure was elucidated as follows:
Participants were instructed to start playing the game. They were to read the instructions within the game and complete the pre-game questionnaire, with an estimated time requirement of approximately 10 min. The game, characterized as a novel game with a storyline, involved practicing questions embedded within the narrative. Participants were encouraged to take notes during this segment. Participants were informed about the upcoming word game, scheduled to begin at 14:10 and requiring around 30 min. A final test was announced, allowing participants a single attempt. Once initiated, there would be no revisiting previous sections of the story or practice questions. The final test would take place from 14:40 to 15:00, with the scores uniformly tallied at the conclusion. Following the final test, participants were required to complete a post-questionnaire within an estimated time frame of 15–20 min. Participants were instructed not to communicate with other students during the entire process and were encouraged to raise their hands if any questions arose. The overall time commitment for the entire process was communicated as approximately 1 hour and 30 min. Upon agreement with the outlined terms, participants were prompted to enter their names and check the consent box, signaling the commencement of the experiment, which had a total duration of 85 min.
Step 2: The participants enter the game via a QR code (5 min).
In Step 2 of the experiment, participants accessed the game by scanning a QR code, initiating a process illustrated in Figure 2, depicting the game’s start interface.
Figure 2. For Step 2 The game’s start Interface.
Upon entering the game, the system automatically categorized participants into six distinct groups without revealing this information to them. This grouping was conducted using a random assignment method, where participants were allocated to groups based on numbers generated randomly by the system. The use of random assignment was intentional, as it effectively minimizes the potential for human bias that can occur when groups are formed by individuals. By relying on a systematic randomization process, the study ensured that the distribution of participants across the six groups was impartial and balanced, thereby enhancing the validity of the experimental results. This method not only fosters a fair comparison among groups but also strengthens the overall reliability of the findings by mitigating confounding variables that could arise from non-random grouping. The groups were named as follows: “No-reward feedback-only absolute evaluation group,” “No-reward feedback-only relative evaluation group,” “Material reward absolute evaluation group,” “Material reward relative evaluation group,” “Digital badge absolute evaluation group,” and “Digital badge relative evaluation group.” Each participant was briefed about the type of reward they could potentially receive at the end of the game and the conditions under which they would qualify for that reward.
The “No-reward feedback-only group” participants were not informed about any reward. In contrast, participants in the “No reward feedback-only absolute evaluation group” were informed that they would receive feedback on their final test scores, while those in the “No-reward feedback-only relative evaluation group” were told they would receive feedback on their final test ranking. For the “Material reward absolute evaluation group,” participants were informed that they would receive monetary compensation if they scored 90 or higher on the final test. The “Material reward relative evaluation group” participants were informed that monetary compensation would be awarded to the top three performers in the final test. In the “Digital badge absolute evaluation group,” participants were told that if they scored 90 or higher on the final test, they would receive a digital badge providing information on when it was earned, proof of grade, and proof of completion of the course (studied classical Japanese). The design of the digital badge image would be revealed after earning it. Similarly, participants in the “Digital badge relative evaluation group” were informed that placing in the top three on the final test would grant them a digital badge with similar details. The content from Figure 2 indicated that participants in the no-pay condition group would engage in a game comprising a story, exercises, and a final test, emphasizing that the correctness of exercise answers had no impact on the final score. Additional declarative information, based on the group, informed participants about potential rewards or competition. The right side of the figure depicted the game loading process, transitioning to the commencement of the game after loading completion.
Step 3: The participants complete the pre-questionnaire that will be distributed via an in-game link (5 min).
In Step 3 of the experiment, participants were directed to complete a pre-questionnaire distributed via an in-game link. This pre-questionnaire aimed to assess participants’ prior knowledge of the task and required approximately 5 min to complete. Figure 3 illustrated the pre-questionnaire interface, featuring four questions related to gender, major, prior study of the subject, and interest in the subject.
Figure 3. For Step 3 the pre-questionnaire interface.
Step 4: After answering a pre-questionnaire, the game begins. The participants read the game novels, learn knowledge, and complete the exercises. The exercises can be repeated (40 min).
Following the completion of the pre-questionnaire, Step 4 commenced. Participants entered the game, initiating a 40-min session where they read game novels, acquired knowledge, and completed exercises. The exercises were designed to be repeatable, allowing participants to reinforce their understanding. Figure 4 presented the study interface, showcasing the ongoing progress of the knowledge learning part of the story, featuring one of the knowledge points of ancient Japanese language. Simultaneously, Figure 5 depicted the comprehension test interface, displaying the answer section of the comprehension level, which assessed participants’ understanding of the knowledge points introduced during the lesson. The lesson on the study of classical Japanese was presented in the form of a novel game, followed by drill exercises. Participants were provided time to practice the initial drill exercises and review the acquired knowledge, with a total of 5 lessons. The contents of the left side of Figures 4 and 5 mirrored those of Figure 2, emphasizing the key elements of the game.
Figure 4. For Step 4 the study interface.
Figure 5. For Step 4 the comprehension test interface.
Step 5: The participants complete the final test. The final test can only be participated in once (20 min).
In Step 5 of the experiment, participants proceeded to complete the final test, which was a one-time opportunity lasting 20 min. Figure 6 depicted the interface for the final test, displaying five of the questions participants encountered during this assessment. The final test comprised a total of 20 questions, and upon answering these questions, the participants’ scores were immediately displayed on the screen, with each question contributing 5 points and a perfect score totaling 100 points.
Figure 6. For Step 5 the final test interface.
Upon completing the final test, participants in the “No-reward feedback-only absolute evaluation group” received feedback solely on their scores, while those in the “No-reward feedback-only relative evaluation group” received feedback exclusively on their rankings. Participants in the “Material reward absolute evaluation group” received score feedback, along with information on the material rewards available when achieving a score of 90 or higher. Similarly, those in the “Material reward relative evaluation group” received feedback on their ranking, with details about the material rewards available for the top three performers. For the “Digital badge absolute evaluation group,” score feedback was provided, and participants were shown a digital badge upon reaching a score of 90. Conversely, the “Digital badge relative evaluation group” received feedback on their ranking. After all participants completed and submitted their final tests, scores were tallied, and a digital badge was displayed on the screen for the top three ranked participants.
The final test interface, as shown in Figure 6, exhibited five of the questions from the test. Participants could read the questions and select the corresponding word as the correct answer.
Step 6: The participants complete a post-questionnaire that will be distributed via an in-game link (15 min).
In Step 6 of the experiment, participants proceeded to complete a post-questionnaire, which was distributed via an in-game link and lasted for approximately 15 min. Figure 7 depicted the interface for the post-questionnaire, showcasing some of the questions participants encountered during this phase. The post-questionnaire, as illustrated in Figure 7, included a 7-segment rating scale where participants could rate their responses on a scale from 1 (indicating not feeling a certain way at all) to 7 (indicating feeling very much this way). The questions covered various aspects, with a section specifically addressing the intrinsic motivation of the participants. Participants were prompted to provide their feedback and responses on the post-questionnaire, offering insights into their experiences, feelings, and levels of intrinsic motivation related to the gamified learning environment they had just engaged with. This data was crucial for evaluating the effectiveness of the gamification elements, including digital badges and different reward structures, on the participants’ overall learning experience and motivation.
Figure 7. For Step 6 the post-questionnaire interface.
4 Results analysis and discussions
4.1 Data analysis
The first is an analysis of intrinsic motivation. A two-factor repeated measures ANOVA on the conditions was conducted, and the result showed a significant difference in the “types of rewards” factor (F(2,89) = 6.277, p = 0.003**, partial η2 = 0.124). No significant differences was found in the “types of evaluation” factor (F(2,89) = 1.149, p = 0.287, partial η2 = 0.013), and the interaction effects (F(2,89) = 1.172, p = 0.314, partial η2 = 0.026). Table 1 demonstrates the results of the analysis of intrinsic motivation. The analysis revealed a highly significant difference in the “types of rewards” factor for overall intrinsic motivation. The partial η2 value of 0.124 indicates a moderate effect size.
Table 1. Two-factor ANOVA results (intrinsic motivation).
Next, the five dimensions of intrinsic motivation were further analyzed and multiple comparisons were made. In the dimension of interest/enjoyment, the analysis also showed a highly significant difference in the “types of rewards” factor (F(2,89) = 4.936, p = 0.009**). This finding supports the hypothesis that digital badges positively impact the interest/enjoyment dimension of intrinsic motivation. The partial η2 value of 0.100 indicates a moderate effect size. For perceived competence, the analysis indicated a significant difference in the “types of rewards” factor (F(2,89) = 3.176, p = 0.047*). This result provides support for the hypothesis that digital badges have a positive effect on the perceived competence dimension of intrinsic motivation. The partial η2 value of 0.067 indicates a small effect size. In the dimension of effort/importance, the analysis revealed a significant difference in the “types of rewards” factor (F(2,89) = 3.333, p = 0.040*) and the interaction effects (F(2,89) = 3.679, p = 0.029*). This finding supports the hypothesis that digital badges impact the effort/importance dimension of intrinsic motivation. The partial η2 value of 0.070 for the interaction effect indicates a moderate effect size. Regarding the pressure/tension dimension, the analysis showed marginal significance in the “types of rewards” factor (F(2,89) = 2.617, p = 0.079+). This result suggests that digital badges may not increase the pressure/tension dimension of intrinsic motivation to the same extent as material rewards, partially supporting the hypothesis. For the value/usefulness dimension, the analysis revealed a significant difference in the “types of rewards” factor (F(2,89) = 3.101, p = 0.050*). This result provides support for the hypothesis that digital badges have a positive impact on the value/usefulness dimension of intrinsic motivation. The partial η2 value of 0.065 indicates a small effect size.
Figure 8 illustrates comparative line graphs of multiple comparison test for each scale of intrinsic motivation.
Figure 8. Line graph of the comparison in the dimension of intrinsic motivation.
Then comes the analysis of extrinsic motivation. A two-factor repeated measures ANOVA on the conditions was conducted, and the result did not show significant differences in the “types of rewards” factor (F(2,89) = 0.163, p = 0.85, partial η2 = 0.004), in the “types of evaluation” (F(2,89) = 1.034, p = 0.312, partial η2 = 0.011), and the interaction effects (F(2,89) = 0.268, p = 0.766, partial η2 = 0.006). Table 2 demonstrates the results of the analysis of extrinsic motivation.
Table 2. Two-factor ANOVA results (extrinsic motivation).
Finally, there is an analysis of academic performance. A two-factor repeated measures ANOVA on the conditions was conducted, and the result showed a significant difference in the “types of rewards” factor (F(2,89) = 5.281, p = 0.007**, partial η2 = 0.106). No significant differences was found in the “types of evaluation” factor (F(2,89) = 0.013, p = 0.91, partial η2 = 0), and the interaction effects (F(2,89) = 1.159, p = 0.854, partial η2 = 0.004). Table 3 demonstrates the results of the analysis of learning outcomes.
Table 3. Two-factor ANOVA results (learning outcomes).
4.2 Data discussion
After the data were analyzed, they were analyzed and discussed against the hypotheses presented in the prior study. The two main factors, type of rewards and type of evaluation, are discussed separately here.
4.2.1 The comparison of reward types
Hypothesis on reward Types and motivation posited that digital badges, as an informational reward, would enhance both intrinsic and extrinsic motivation more effectively than material rewards or no rewards. The hypothesis suggested that digital badges provide feedback on learners’ competence and progress, stimulating curiosity and excitement, thus fostering intrinsic motivation. Additionally, by offering visible recognition of achievements, digital badges fulfill extrinsic motivational needs without inducing the performance-related pressure commonly associated with material rewards.
The experimental results partially supported Hypothesis on reward Types and motivation, showing a positive impact of reward type on intrinsic motivation and academic performance, but no statistically significant differences for extrinsic motivation. The findings align with prior studies suggesting that digital badges primarily influence intrinsic motivation due to their informational role, as opposed to merely serving as an external incentive. The analysis confirmed that digital badges significantly enhanced intrinsic motivation across several dimensions. Specifically:
Interest/Enjoyment: Digital badges increased learners’ interest and enjoyment more effectively than material rewards. This result supports the hypothesis that digital badges, by providing feedback and visible recognition, capture learners’ attention and engage them more deeply. Post-experimental comparisons indicated that while material rewards improved intrinsic motivation compared to no rewards, digital badges were more effective at sustaining learners’ interest.
Perceived Competence, Effort/Importance, and Value/Usefulness: In these dimensions, digital badges had similar effects to material rewards with feedback, indicating that both reward types were effective in reinforcing learners’ perceptions of competence, effort invested, and the perceived value of the tasks. These findings suggest that when feedback accompanies rewards, it can promote a sense of achievement and purpose, regardless of the reward type.
Pressure/Tension: A significant advantage of digital badges over material rewards emerged in the Pressure/Tension dimension. Unlike material rewards, which may inadvertently create stress due to their association with external expectations, digital badges did not induce tension, resembling the no-reward condition in this respect. This result highlights the unique benefit of digital badges in fostering a stress-free learning environment while still motivating learners.
The increase in intrinsic motivation associated with digital badges was further reflected in academic performance. The data supported the notion that heightened intrinsic motivation, driven by informational rewards such as digital badges, leads to improved learning outcomes.
Despite the positive impact on intrinsic motivation, the analysis did not find significant differences in extrinsic motivation across the different reward types. This finding is intriguing, given the hypothesis that digital badges would also fulfill extrinsic motivational needs by providing visible recognition. Several factors might explain this discrepancy:
Contextual Factors of the Experiment: The study was conducted in an online setting where participants did not have the opportunity to interact or share their rewards post-experiment. Previous research has indicated that social recognition and the sharing of rewards can significantly influence motivation, especially in terms of external validation. The lack of interaction may have limited the external impact of both digital badges and material rewards, thus reducing the observable differences in extrinsic motivation.
Timing of Reward Delivery: In practical scenarios, rewards are often delivered after a task’s completion, allowing time for participants to appreciate and internalize them. In this study, the immediate reward delivery might not have provided sufficient time for participants to process and value the rewards fully, potentially dampening their motivational impact. Delayed reward delivery in future studies might better simulate real-world conditions, potentially revealing different effects on extrinsic motivation.
The lack of a statistically significant main effect for the “types of rewards” factor suggests that the influence of reward types on motivation may be more nuanced than initially anticipated. Future studies could explore mechanisms for incorporating social recognition and delayed reward delivery to better understand the role of different reward types. For instance, facilitating post-experiment interactions where participants can share and discuss their digital badges may reveal latent effects on motivation. Additionally, replicating the study in an offline setting or integrating hybrid approaches could help to assess whether contextual factors played a significant role in these findings.
Overall, the results indicate that digital badges, when used as rewards in educational settings, have comparable positive effects on intrinsic motivation as material rewards. However, they also offer distinct advantages by being more effective at capturing interest and avoiding learner stress. Although the findings did not demonstrate significant differences in extrinsic motivation, the implications of the results suggest that digital badges still hold promise for enhancing learning outcomes by fostering a balanced motivational environment that supports both intrinsic and extrinsic needs.
4.2.2 The comparison of evaluation types
Hypothesis on evaluation types proposed that relative evaluation badges, which emphasize peer comparisons, would be more effective in fostering intrinsic motivation than absolute evaluation badges. The hypothesis was based on the idea that relative evaluation badges fulfill learners’ need for social recognition and achievement by tapping into their competitive nature, whereas absolute evaluation badges focus on individual accomplishments and may primarily enhance extrinsic motivation without significantly boosting intrinsic motivation. The results did not support Hypothesis on evaluation types, as the analysis did not show statistically significant differences in intrinsic motivation, extrinsic motivation, or academic achievement across the different evaluation types. Despite the lack of a significant main effect, the findings offer insights into the potential mechanisms and factors that may have influenced these results.
Although the data did not reveal statistically significant differences in motivation outcomes between relative and absolute evaluation badges, the results are somewhat consistent with prior research indicating that both types of badges can enhance extrinsic motivation. However, the mechanisms underlying this enhancement may vary depending on the evaluation type:
Shared Mechanisms in Extrinsic Motivation: The lack of significant differences suggests that the extrinsic motivation triggered by both relative and absolute evaluation badges may operate through shared mechanisms. For example, both badge types offer some form of recognition, which could fulfill participants’ needs for external validation and reward regardless of the specific evaluation type. This shared effect may have diluted any observable differences in the data.
Limitations in Measurement Sensitivity: The measures used to assess motivation in this study may not have been sensitive enough to detect the nuanced differences between the two evaluation types. While the badges may have influenced motivation in different ways, these distinctions may not have been adequately captured by the tools used for data collection. For instance, the competitive aspects of relative evaluation badges may not have been sufficiently highlighted in the survey items, potentially leading to an underestimation of their impact on intrinsic motivation.
Participant Characteristics and Task Nature: The specific characteristics of the participant group and the nature of the learning task could also account for the lack of significant findings. It is possible that the participants in this study did not respond to peer comparison-based evaluations in the same way as those in previous research. Factors such as prior competitive experiences, cultural attitudes towards competition, or the level of familiarity with the digital badges could have influenced how participants perceived the badges and affected the results.
The absence of a significant main effect in this study raises interesting questions about the underlying processes by which different types of evaluation badges affect motivation. To gain a deeper understanding of these mechanisms, future research should consider adopting a more detailed approach:
Segmented Experimental Phases: By dividing the study into pre-intervention, during intervention, and post-intervention phases, future research can track changes in extrinsic motivation over time. This segmented approach may reveal subtler variations in how relative and absolute evaluation badges impact learners’ motivation at different stages of the learning process, potentially uncovering dynamic changes that were not captured in the current study.
Exploring Individual Differences: Investigating individual differences in response to evaluation types may provide valuable insights into why certain learners respond more positively to relative evaluation badges, while others benefit more from absolute evaluation badges. Factors such as personality traits, prior educational experiences, or intrinsic competitive drive could be examined to identify subgroups that are more sensitive to the effects of peer comparisons or individual achievement recognition.
Contextual and Social Factors: The study was conducted in an online environment, where participants may not have had the same opportunities for social interaction and recognition as in traditional classroom settings. The lack of social engagement could have weakened the motivational impact of relative evaluation badges, which often rely on peer recognition to drive competition and engagement. Incorporating social elements such as discussion boards or leaderboards in future studies may help to better simulate real-world learning environments and highlight the effects of social recognition.
Task Design and Reward Relevance: Future studies could also explore tasks that are inherently more competitive or cooperative to see if the effects of relative versus absolute evaluation badges differ depending on the task’s nature. Tasks designed to emphasize social comparisons may bring out the motivational differences more clearly than tasks that focus solely on individual performance.
While Hypothesis on evaluation types was not supported by the data, the findings align with some prior research suggesting that both relative and absolute evaluation badges can enhance extrinsic motivation, albeit potentially through different mechanisms. The lack of significant differences in this study may be attributed to shared motivational pathways, limitations in measurement sensitivity, participant characteristics, or the online nature of the experiment. Future research should take a more fine-grained approach to examine these factors, exploring changes in motivation over time, considering individual differences, and incorporating social elements to better understand the distinct impacts of evaluation types on learners’ motivation.
4.2.3 Additional analysis on individual differences
The additional analysis explored the impact of individual differences on motivation to explain the lack of alignment between the results for extrinsic motivation and the initial hypothesis. The analysis focused on two key variables: prior interest in the subject and preference for the game, examining their relationships with various aspects of motivation.
The correlation between prior interest in the subject (n) and five dimensions of intrinsic motivation—Interest/Enjoyment, Perceived Competence, Effort/Importance, Pressure/Tension, Value/Usefulness, and Extrinsic Motivation—was investigated. Table 4 presents the results, with Pearson correlation coefficients used to assess the strength of these relationships.
Table 4. Pearson correlation.
The findings indicate that the correlation coefficient between prior interest and Interest/Enjoyment was −0.041, close to zero, with a p-value of 0.692 ( > 0.05), showing no significant relationship. Similarly, the correlation with Perceived Competence was −0.059, with a p-value of 0.568 ( > 0.05), indicating no significant association. For Effort/Importance, the correlation was 0.009, with a p-value of 0.935 ( > 0.05), also showing no significant relationship. The correlation with Pressure/Tension was −0.042, with a p-value of 0.687 ( > 0.05), and with Value/Usefulness, it was −0.052, with a p-value of 0.614 ( > 0.05), both showing no significant associations. Lastly, the correlation between prior interest and Extrinsic Motivation was 0.112, with a p-value of 0.279 ( > 0.05), indicating no significant relationship.
These results are consistent with the selection criteria for participants, as all learners were beginners in classical Japanese, leading to a similar level of interest in the subject.
An additional analysis was conducted to examine the correlation between preference for the game and five dimensions of intrinsic motivation: Interest/Enjoyment, Perceived Competence, Effort/Importance, Pressure/Tension, Value/Usefulness, and Extrinsic Motivation. The results are displayed in Table 5.
Table 5. Pearson correlation.
The correlation analysis revealed that the coefficient between preference for the game and Interest/Enjoyment was −0.130, with a p-value of 0.210 ( > 0.05), indicating no significant relationship. The correlation with Perceived Competence was −0.054, with a p-value of 0.602 ( > 0.05), indicating no significant association. For Effort/Importance, the correlation was −0.049, with a p-value of 0.637 ( > 0.05), again indicating no significant relationship. The correlation with Pressure/Tension was −0.055, with a p-value of 0.597 ( > 0.05), and with Value/Usefulness, it was −0.189, with a p-value of 0.066 ( > 0.05), both showing no significant associations.
However, a significant positive correlation was observed between preference for the game and Extrinsic Motivation, with a correlation coefficient of 0.355 and a significance level of p < 0.01. This suggests that participants with a higher preference for the game tended to exhibit increased extrinsic motivation.
The findings indicate that while individual differences in prior interest did not significantly influence motivational outcomes, preference for the game had a notable effect on extrinsic motivation, suggesting the need for further investigation into how such preferences can impact motivation in educational contexts.
4.2.4 Other discussions and conclusions
In comparing this study with Tripon’s (2020) research on the challenges of using video tools to promote personalized student learning and enhance thinking skills, several key aspects emerge. Both studies employ quantitative methodologies, highlighting the necessity of adapting teaching strategies in an evolving educational landscape. Additionally, they acknowledge the transformative potential of technology; Tripon asserts that technology can revolutionize curricula, a notion supported by the current study, which demonstrates that gamified digital badges can enhance learner motivation and improve traditional classroom practices. Overall, the findings from this research closely align with Tripon’s, underscoring the importance of leveraging tools in technology-enhanced learning environments to boost motivation and enrich personalized learning experiences.
In summary, the non-significant findings regarding the types of rewards and evaluations in this study indicate a need for further investigation. Factors such as context, timing, and participant characteristics may significantly influence the effectiveness of motivational strategies. Future research should address these elements to develop a more thorough understanding of how to effectively boost motivation in educational environments. This study also demonstrates the positive impact of digital badges as a reward mechanism on academic performance, aligning with previous research. Digital badges represent a contemporary and effective approach to enhancing intrinsic motivation and improving academic outcomes. The study highlights their versatility, as their impact is not contingent upon specific designs. Future investigations could explore the nuanced mechanisms involved and examine how different types of digital badges may yield varying effects on motivation and performance. Additionally, the duration of the experiment—set at just 90 min—suggests that external motivation verification may require a more extended timeframe. Therefore, assessing the impact of external motivation on sustainable learning is essential.
5 Conclusion
This study aimed to investigate the effectiveness of digital badges as a design element for enhancing learning outcomes within gamified Technology-Enhanced Learning Environments (TELEs). The methodology included a series of experiments comparing the impact of digital badges with material compensation and no compensation, as well as assessing the effects of absolute and relative evaluations. Results revealed that both forms of digital badges, regardless of design, significantly enhanced learning motivation and outcomes. However, the study did not discern distinct mechanisms underpinning the enhancement of motivation between the two badge types. Furthermore, no multiplicative effect was observed when considering the interplay between badge types and reward categories. Limitations included insufficient time allocated for measuring external motivation and a lack of thorough investigation into the specific mechanisms through which the two digital badge types influenced learning.
Future studies are suggested to measure motivation both pre- and post-experiment to gain a deeper understanding of the extent and stages of motivation change attributable to different digital badge types. This approach could elucidate the mechanisms involved. The findings underscore the potential of digital badges as a valuable addition to gamified TELEs, providing educators and instructional designers with insights to optimize gamification in educational content. Interestingly, the absence of significant differentiation between the two badge types points towards flexibility in designing learning experiences. Instructors can opt for either type based on specific goals and learner preferences, accommodating diverse student needs. As digital badges gain traction in formal and informal learning environments, they can promote lifelong learning, encouraging learners to pursue knowledge for personal growth while recognizing achievements. This research contributes to the academic discourse on gamification and motivation within TELEs, positioning digital badges as a powerful tool in education.
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
The studies involving humans were approved by the Osaka University Research Ethics Review Committee. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author contributions
JL: Writing – original draft, Writing – review and editing.
Funding
The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.
Conflict of interest
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Abramovich, S., Schunn, C., and Higashi, R. M. (2013). Are badges useful in education?: It depends upon the type of badge and expertise of learner. Educ. Technol. Res. Dev. 61, 217–232.
Alexandra, W. (2024). How digital badges can be used within education. Available at: https://alexandraandrewjames.com/defining-digital-badges/ (accessed October 15, 2024).
Bandhu, D., Mohan, M. M., Nittala, N. A. P., Jadhav, P., Bhadauria, A., and Saxena, K. K. (2024). Theories of motivation: A comprehensive analysis of human behavior drivers. Acta Psychol. 244:104177.
Bower, M. (2017). Design of Technology-Enhanced Learning: Integrating Research and Practice. Bingley: Emerald Publishing Limited.
Buckworth, J., Lee, R. E., Regan, G., Schneider, L. K., and DiClemente, C. C. (2007). Decomposing intrinsic and extrinsic motivation for exercise: Application to stages of motivational readiness. Psychol. Sport Exerc. 8, 441–461.
Chan, G., Arya, A., Orji, R., Zhao, Z., and Whitehead, A. (2024). Increasing motivation in social exercise games: Personalising gamification elements to player type. Behav. Inf. Technol. 43, 2608–2638.
Clark, D. B., Tanner-Smith, E. E., and Killingsworth, S. S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. Rev. Educ. Res. 86, 79–122.
Csikszentmihalyi, M. (1988). The flow experience and its significance for human psychology. Optimal Exp. Psychol. Stud. Flow Consciousness 2, 15–35.
Deci, E. L., Cascio, W. F., and Krusell, J. (1975). Cognitive evaluation theory and some comments on the Calder and Staw critique. J. Pers. Soc. Psychol. 32, 81–85.
Deci, E. L., and Ryan, R. M. (1980). The empirical exploration of intrinsic motivational processes. Adv. Exp. Soc. Psychol. 13, 39–80. doi: 10.1016/S0065-2601(08)60130-6
Deci, E. L., and Ryan, R. M. (1985). Intrinsic Motivation and Self-Determination in Human Behavior. Berlin: Springer Science & Business Media.
Deci, E. L., Koestner, R., and Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychol. Bull. 125:627. doi: 10.1037/0033-2909.125.6.627
Delwiche, A. (2006). Massively multiplayer online games (MMOs) in the new media classroom. J. Educ. Technol. Soc. 9, 160–172.
Deterding, S., Dixon, D., Khaled, R., and Nacke, L. (2011). “From game design elements to gamefulness: Defining” gamification,” in Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, (Association for Computing Machinery), 9–15.
Dettori, G. (2009). “Narrative learning environments,” in Encyclopedia of Information Communication Technology, eds A. Cartelli and M. Palma (Hershey, PA: IGI Global), 575–583.
Dickey, M. (2005). Engaging by design: How engagement strategies in popular computer and video games can inform instructional design. Educ. Technol. Res. Dev. 53, 67–83.
Dror, I. (2008). Technology enhanced learning: The good, the bad, and the ugly. Pragmat. Cogn. 16, 213–215.
Erhel, S., and Jamet, E. (2013). Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness. Comput. Educ. 67, 156–167. doi: 10.1016/j.compedu.2013.02.019
Finkelstein, J., Knight, E., and Manning, S. (2013). The Potential and Value of Using Digital Badges for Adult Learners. Virginia: American Institutes for Research.
Filgona, J., Sakiyo, J., Gwany, D. M., and Okoronka, A. U. (2020). Motivation in learning. Asian J. Educ. Soc. Stud. 10, 16–37. doi: 10.9734/ajess/2020/v10i430273
Gibson, D., Ostashewski, N., Flintoff, K., Grant, S., and Knight, E. (2015). Digital badges in education. Educ. Inf. Technol. 20, 403–410.
Goulding, J., Sharp, H., and Twining, P. (2024). Awarding digital badges: Research from a first-year university course. High. Educ. Res. Dev. 43, 640–656.
Grant, S. L. (2016). “History and context of open digital badges,” in Digital badges in education, eds Y. Lin, M. Mullenberg, and Z. L. Berge (Milton Park: Routledge), 3–11.
Greenblat, C. S. (1981). “Teaching with simulation games: A review of claims and evidence,” in Principles and Practices of Gaming-Simulation, eds C. S. Greenblat and R. D. Duke (Beverly Hills, CA: Sage Publications), 139–153. doi: 10.1080/10872981.2018.1438718
Hakulinen, L., Auvinen, T., and Korhonen, A. (2013). “Empirical study on the effect of achievement badges in TRAKLA2 online learning environment,” in Proceedings of the Learning and Teaching in Computing and Engineering (LaTiCE), 2013, (Macau: IEEE Computer Society), 47–54.
Hamari, J., Koivisto, J., and Sarsa, H. (2014). “Does gamification work?–a literature review of empirical studies on gamification,” in Proceedings of the 2014 47th Hawaii International Conference on System Sciences, (Piscataway, NJ: IEEE), 3025–3034.
Haruguchi, J. (2010). Reconsidering classical Japanese language education for learners of Japanese - from the perspectives of learners, Japanese language teachers and Japanese language teachers. Nagasaki Univ. Foreign Stud. Rev. 141–152.
Hays, R. T. (2005). The Effectiveness of Instructional Games: A Literature Review and Discussion. Technical Report 2005-004. Orlando, FL: Naval Air Warfare Center Training Systems
Homer, B. D., Raffaele, C., and Henderson, H. (2020). “Games as playful learning: Implications of developmental theory for game-based learning,” in Handbook of Game-Based Learning, eds J. L. Plass, R. E. Mayer, and B. D. Home (Cambridge, MA: MIT Press), 25–52.
Kent, S. L. (2010). The Ultimate History of Video Games, Volume 1: From Pong to Pokemon and Beyond- the Story Behind the Craze That Touched Our Lives and Changed the World, vol. 1. New York, NY: Crown.
Ketelhut, D. J., and Schifter, C. C. (2011). Teachers and game-based learning: Improving understanding of how to increase efficacy of adoption. Comput. Educ. 56, 539–546.
Kim, M., and Shin, J. (2016). The pedagogical benefits of SimCity in urban geography education. J. Geogr. 115, 39–50.
Lu, S., Xie, Y., and Chen, X. (2023). Immediate and enduring effects of digital badges on online content consumption and generation. Int. J. Re. Mark. 40, 146–163.
Malone, T. W. (1981). Toward a theory of intrinsically motivating instruction. Cogn. Sci. 5, 333–369.
Marín, V. I., de Benito, B., and Darder, A. (2024). Technology-enhanced learning for student agency in higher education: A systematic literature review. Interaction Design Architecture J. 45, 15–49.
McDaniel, R., Lindgren, R., and Friskics, J. (2012). “Using badges for shaping interactions in online learning environments,” in Proceedings of the Professional Communication Conference (IPCC), 2012 IEEE International, (Piscataway, NJ: IEEE), 1–4.
Minnery, J., and Searle, G. (2014). Toying with the city? Using the computer game SimCity™ 4 in planning education. Plann. Pract. Res. 29, 41–55.
Monteiro, V., Mata, L., and Peixoto, F. (2015). Intrinsic motivation inventory: Psychometric properties in the context of first language and mathematics learning. Psicologia Reflexão e Crítica 28, 434–443.
Montola, M., Nummenmaa, T., Lucero, A., Boberg, M., and Korhonen, H. (2009). “Applying game achievement systems to enhance user experience in a photo sharing service,” in Paper Published in Proceedings of the 13th International MindTrek Conference, (New York, NY).
Muilenburg, L. Y., and Berge, Z. L. (2016). Digital Badges in Education: Trends, Issues, and Cases. Milton Park: Routledge.
Nadeem, M., Oroszlanyova, M., and Farag, W. (2023). Effect of digital game-based learning on student engagement and motivation. Computers 12:177.
Nardi, B., and Harris, J. (2006). “Strangers and friends: Collaborative play in World of Warcraft,” in Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work, (ACM), 149–158.
O’Donovan, S., Gain, J., and Marais, P. (2013). “A case study in the gamification of a university level games development course,” in Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference, (New York, NY: ACM), 242–251.
Ostrow, K. S., and Heffernan, N. T. (2018). “Testing the validity and reliability of intrinsic motivation inventory subscales within ASSISTments,” in Proceedings of the Artificial Intelligence in Education: 19th International Conference, AIED 2018, (London: Springer).
Pangaribuan, C. H., Hidayat, D., Putra, O. P. B., Aguzman, G., and Febriyanto, R. (2021). Digital badge from the perspective of selfdetermination theory. PalArch’s J. Archaeol. Egypt 18, 116–128.
Plass, J. L., Homer, B. D., and Kinzer, C. K. (2015). Foundations of game-based learning. Educ. Psychol. 50, 258–283.
Razhkou, I. (2024). In-Game Reward Systems and their Effect on the Player. Doctoral dissertation. Klagenfurt: Alpen-Adria-Universität Klagenfurt.
Robert, J. G., Zaldivar, M. R., Ajao, H., and Yang, Z. (2024). Enhancing real estate data analytics education with digital badges. J. Real Estate Pract. Educ. 26:2403792.
Mayer, R. E. (2014). Computer Games for Learning. An Evidence-Based Approach. Cambridge: The MIT Press.
Ryan, R. M., Mims, V., and Koestner, R. (1983). Relation of reward contingency and interpersonal context to intrinsic motivation: A review and test using cognitive evaluation theory. J. Pers. Soc. Psychol. 45, 736–750.
Ryan, R. M., and Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am. Psychol. 55, 68–78.
Schoenenberger, H. (2024). “The impact of digital badges on student learning and engagement,” in Connected Learning: Origins, Opportunities, and Perspectives of Contemporary Educational Design, ed. H. Schoenenberger (Cham: Springer).
Schweighofer, P., and Ebner, M. (2015). Aspects to be considered when implementing technology-enhanced learning approaches: A literature review. Future Int. 7, 26–49.
Shields, R., and Chugh, R. (2017). Digital badges–rewards for learning? Educ. Inf. Technol. 22, 1817–1824.
Shuler, C. (2012). “What in the world happened to Carmen Sandiego? Classroom aid,” in The edutainment era: Debunking myths and sharing lessons learned. New York, NY: The Joan Ganz Cooney Center at Sesame Workshop.
Tripon, C. (2020). “Challenges of teaching - personalized students learning by using video tools to improve thinking skills,” in Proceedings of the international conference on virtual learning, proceedings paper, WOS:000676171700016, 159–165.
Vogel, J. J., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K., and Wright, M. (2006). Computer gaming and interactive simulations for learning: a meta-analysis. J. Educ. Comput. Res. 34, 229–243. doi: 10.2190/FLHV-K4WA-WPVQ-H0YM
Wang, G., Zhou, G., and Li, Z. (2022). Research on the application of chinese teaching based on social media video platforms. Int. J. Distance Educ. Technol. 20:16. doi: 10.4018/IJDET.296699
Zhang, J., and West, R. E. (2023). Use of open badges for student instructor training: An evaluation case study. J. Formative Desig. Learn. 7, 126–138.
Keywords: gamification, technology-enhanced learning environments, motivation, digital badges, academic performance
Citation: Luo J (2024) Validating the impact of gamified technology-enhanced learning environments on motivation and academic performance: enhancing TELEs with digital badges. Front. Educ. 9:1429452. doi: 10.3389/feduc.2024.1429452
Received: 08 May 2024; Accepted: 30 October 2024;
Published: 13 November 2024.
Edited by:
Leman Figen Gul, Istanbul Technical University, TürkiyeReviewed by:
Chih Ming Chu, National Ilan University, TaiwanVanessa Camilleri, University of Malta, Malta
Cristina Tripon, Polytechnic University of Bucharest, Romania
Copyright © 2024 Luo. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Jiejin Luo, jiejin@hus-et.net