Research on an evaluation rubric for promoting user’s continuous usage intention: a case study of serious games for Chinese cultural heritage

The sustainable development of serious games dedicated to Chinese cultural heritage faces challenges due to the lack of prolonged user engagement and iterative optimization based on long-term user feedback. This situation not only hinders the sustainable growth of these games but also limits the transmission of Chinese cultural heritage, a problem that demands urgent attention yet remains under-acknowledged. This study synthesizes literature to unearth user needs from three dimensions: motivational use, quality requirements of games, and continuous usage intention. It identifies 14 influential factors, including cognitive satisfaction, immersion satisfaction, and achievement satisfaction. Drawing from the User Experience Rubric for Educational Games-CH (UEREG-CH) evaluation rubric and the Delphi method, these factors are expanded into an evaluation system model comprising six primary indicators, 14 secondary indicators, and 38 tertiary indicators. The analytic hierarchy process (AHP) is utilized for expert scoring to assign weights to each indicator. This is combined with the fuzzy comprehensive evaluation method to arrive at a final user score of Z = 3.4228, indicating that the evaluation rubric is close to ‘good’ and has received positive user feedback. Ultimately, this generates an evaluation rubric tailored to the specific context of serious games for cultural heritage. By integrating qualitative and quantitative methods, this paper confirms the scientific and rational nature of the evaluation rubric. The study aims to establish a user-approved rubric that encourages continuous usage intention, thereby providing effective guidance for game developers and assisting users in selecting appropriate games, while also addressing the theoretical gap in the field of evaluation for serious games related to cultural heritage.

1 Introduction

This research focuses on Serious Games for Cultural Heritage, which are games developed through the integration of cultural heritage and serious gaming. The primary purpose of these games is to protect and disseminate a specific theme of cultural heritage, primarily through digital interactive formats. They often simulate and emulate traditional cultural skills. In this study, they are abbreviated as CH-SGs (Zhou and Xu, 2017). The interest in the cultural heritage domain within serious games arises from the current social context, where protecting and inheriting cultural heritage is a crucial historical mission for the national defense of independence and sovereignty. Many Chinese cultural heritage projects have been lost due to historical reasons and the complexity of their forms. Therefore, there is an urgent need to broaden the channels for disseminating cultural heritage and to find carriers for its living inheritance and innovation (Teng, 2019). CH-SGs, with their unique functional attributes, make cultural heritage knowledge and skills more accessible to users through digital network technologies and entertaining gaming methods, thereby stimulating their autonomous learning (Qin, 2017). Thus, CH-SGs play an essential role in preserving the authenticity of cultural heritage, enriching its presentation forms, lowering the barriers for user engagement, and assuming the crucial responsibility for the transformative development of cultural heritage. Hence, this paper emphasizes the importance of optimizing and improving CH-SGs for significant social relevance (Cai, 2020). Although CH-SGs possess positive social value and have become a new medium for presenting Chinese traditional culture to the world, supported strongly by the state, their practical development is not optimistic. Their development and usage in the Chinese gaming market are relatively scarce (Chen and Liu, 2019). To date, only about twenty games have been developed by renowned companies like Tencent Games and the Palace Museum (Figure 1). These games only offer a “one-time” experience of cultural heritage, sparking only brief interest in users and failing to elicit long-term usage feedback and real, objective evaluation standards to effectively improve game quality (Li, 2023). Likewise, theoretical research on these games is also scarce. In previous literature, the Chinese scholar Xu Haifeng focused on the Shenyang Imperial Palace’s material culture as a theme, emphasizing the game’s functional and aesthetic attributes to enhance user experience (Xu H. F., 2019). Li Haishi, through an analysis of games for intangible cultural heritage, summarized three design transformation paths, emphasizing the design expression of intangible cultural heritage, its visual transformation, and experience design (Li, 2019). Yang Xiao took Shaman culture as a starting point to analyze the factors influencing cognition to behavior in serious games, proving that games play a more effective role in the dissemination of intangible heritage (Yang, 2020). Previous literature has focused on CH-SGs’ cultural themes, game element design, and dissemination impact, but lacks consideration on how to promote continuous usage intention and exploration of user evaluation of such games.

Figure 1

Figure 1. Serious games for Chinese cultural heritage.

The current evaluation theories for serious games generally adopt Gardner and Hatch, (1989) and Krathwohl et al. (1956). These evaluations focus on curriculum standards and the educational attributes of educations (Shen, 2015). However, established serious game evaluations are mostly limited to educational and medical fields, and their applicability in the cultural heritage domain remains unverified. Therefore, the research on a CH-SGs evaluation rubric in this paper aims to fill this gap, promoting the development of both theoretical and empirical research on CH-SGs evaluation. It also aids other types of serious games in exploring suitable evaluation indicators and assessment tools. This study posits that with the guidance of evaluation rubrics, the design process of CH-SGs is akin to having a measuring ruler and a direction, leading to breakthroughs in research on design, development, and application evaluations of CH-SGs (Wang et al., 2019a). An evaluation rubric is a scoring tool that includes standards for assessing a task, encompassing various indicators each with its specific weight and assessment criteria. It is a scoring tool that can present indicators in the form of charts or tables. The Chinese scholar Li and Zhu (2017) considers rubrics as structured quantitative evaluation tools, useful for assessing student digital works with strong operability and accuracy, allowing evaluations by teachers, students, and peers (Li and Zhu, 2017). The evaluation rubric in this paper includes various levels of indicators based on theoretical foundations, representing a comprehensive set of evaluation indicators. It revolves around the needs of CH-SGs users, serving as a reference for designers. Establishing a CH-SGs evaluation rubric requires more than a single theoretical construction; it necessitates examination from the user’s perspective (Xu L. Q., 2019). However, it is observed that the existing evaluation rubrics are predominantly used by teachers, designers, and parents, with few specifically formulated for users. The primary evaluators of these games are educators and developers, not the learners or users. This study argues that the users are the true evaluators of serious games, which requires developers to consider game content that matches user needs right from the design stage (Shen and Zhang, 2014). However, accurately grasping user needs is challenging, and past qualitative evaluations based on user experience have overly relied on the evaluators’ quality, often being subjective and lacking scientific rigor (Jin and Yang, 2017). Therefore, this study attempts a multidimensional exploration of user need indicators and a multi-method validation of the evaluation rubric.

In summary, this research focuses on two main questions: how to construct a multi-dimensional CH-SG evaluation rubric that meets user needs, and whether the constructed CH-SG evaluation rubric is accepted by users. This article summarizes 14 user demand factors through literature induction, from the perspectives of usage motivation, game quality, and continuous usage intention; refers to the User Experience Rubric for Educational Games-CH (UEREG-CH) evaluation scale to initially form a CH-SG evaluation system model with 6 primary indicators, 14 secondary indicators, and 32 tertiary indicators; revises and perfects this model in terms of dimensions and semantics through the Delphi expert consultation method, expanding the original 32 tertiary indicators to 38; then conducts Analytic Hierarchy Process (AHP) hierarchical analysis with expert scoring to assign weights to each indicator of the model; subsequently, through user questionnaire surveys, scores are assigned to this model, combined with the weights given by experts to perform fuzzy comprehensive calculations, obtaining the final user evaluation value to determine whether users accept the CH-SG evaluation scale (Figure 2).

Figure 2

Figure 2. Research flowchart.

This study constructs the evaluation rubric for Cultural Heritage Serious Games (CH-SGs) from three dimensions: usage motivation, game quality, and continuous usage intention. Initially, one issue with CH-SGs is that they fail to prioritize users’ motivational needs (Tang, 2022). Instead, from design to content, these games often align with the aesthetics of designers and professionals, as evidenced by games like “Tenon and Mortise,” awarded as an excellent app by the Apple App Store in 2014, “Folding Fan” winning the “2018 Zcool Award - Interface Design Category” special award, and “Nishan Shaman” receiving the “Lina and Haifeng, 2020 China Best Sound Effect Award” among others. While these games are recognized in the industry, they struggle to captivate users, often described as “one-time games” or “beautiful but useless,” indicating that designers are not fully aware of the real motivations of users. In terms of usage motivation literature, Chinese research on game users, especially focusing on the psychological motivations of adolescents using online games, is limited (Chen, 2023). The scholar Zhong Zhijin found that players engaged in online games are influenced by different motivations like personal achievement, enjoying a social life, and immersing in virtual worlds to escape reality, all significantly boosting game duration and stickiness (Zhong, 2011). Research by Ye Na, involving an empirical survey of 546 college students playing online games, showed that the relationship between achievement satisfaction and online game addiction was the strongest (Ye et al., 2009). This study references Zhang Wenmin’s research on Multiplayer online battle arena (MOBA) mobile game users’ continuous usage intentions, which posits that social, competitive, immersive, and cognitive motivations significantly impact users’ continued usage (Zhang, 2019). Furthermore, the scholar Holbrook noted that the product provided by online games is virtual, offering psychological and emotional satisfaction through emotional interaction with others, leading to leisure enjoyment (Holbrook and Hirschman, 1982). In her empirical study on the relationship between customer perceived value, satisfaction, and loyalty in online games, Hu (2013) also found that leisure enjoyment positively impacts customer retention. Therefore, based on the above discussions, this study incorporates cognitive satisfaction, immersive satisfaction, achievement satisfaction, social satisfaction, and leisure satisfaction into the user motivation indicators for measurement, defined as follows:

1. Cognitive satisfaction: Kahn believes this involves helping users develop specific intellectual abilities through games.

2. Immersive satisfaction: Yee defines this as the behavior or psychological state where users immerse themselves in the virtual world of the game, escaping reality.

3. Achievement satisfaction: This refers to the user’s strong desire to achieve victory in the game.

4. Social satisfaction: Yee sees this as the user’s expectation to achieve social activities on the gaming platform.

5. Leisure satisfaction: Holbrook suggests that through emotional interaction in online games, customers experience a sense of well-being in their consciousness, meeting their psychological and emotional needs and expectations.

Secondly, as products of the internet era, CH-SGs require support from various sectors of the industry chain, from pre-launch advertising by game operators, provision of downloads by distributors, and stable service from service providers to continuous updates and optimization by developers (Jiang, 2011). Users may choose to continue or abandon a game due to various factors such as server quality and system stability. A prominent issue in the current design of CH-SGs is the simplicity of their mechanism design, lacking in gameplay and intrigue, and their weak playability. Additionally, the integration of educational elements with gameplay is insufficient (Lei and Feng, 2020). For instance, games like “Tenon and Mortise” and “Folding Fan” by Tag Design team are termed as “serious games,” but they focus more on conveying the artistry of cultural heritage, with more emphasis on aesthetics than on interactive gaming, failing to convey information through game mechanics or narratives, which points to a deficiency in the game quality of CH-SGs (Zhang S. Y., 2023). Among the literature related to game quality, many scholars view games as a new type of information system and have incorporated the revised model of information system success into studies on game user engagement, confirming that system quality, information quality, and service quality influence user participation (Wang, 2010). This study refers to the scholar Zhang (2016), who analyzed mobile game user churn and attributed game factors to system quality, service quality, game design, and brand image, demonstrating significant impacts on the perceived usefulness and ease of use by game users (Fang, 2023). Furthermore, CH-SGs differ from traditional online games in that they are primarily designed to address real-world social and industrial problems. They are not solely for entertainment but are meant to achieve at least one additional goal, namely, the learning objective, without compromising the user’s gaming experience (Peng and Liu, 2020). Many studies have explored different quality standards for serious games, including effectiveness and attractiveness, but have not fully considered the dual mission of serious games, i.e., achieving both functional and gaming aspects. High-quality serious games must balance functionality and gaming; they must systematically support users in achieving learning goals (functional part) and must engage and maintain entertainment objectives (gaming part). These two aspects should be perfectly matched and integrated, rather than being addressed in isolation (Caserman et al., 2020). Therefore, this paper also considers learning and entertainment objectives as factors in game quality, defined as follows:

1. System quality: This refers to the stability and usability of the game system, essential for ensuring a normal gaming experience for users.

2. Service quality: This assesses the quality of services provided by companies to users, ensuring a comfortable and enjoyable gaming experience.

3. Game design: This involves the rational organization and arrangement of various elements of a game product, creation through game engines, and the final formation of the game product.

4. Brand image: The perceived influence of the game brand by users, including the brand image of the game itself and the company’s reputation.

5. Learning objectives: The learning objectives of CH-SGs should be closely related to their application fields. The learning or training effect should remain the focus throughout the game process, always supporting users in achieving this purpose, with game elements not interfering with the learning or training process.

6. Entertainment objectives: High-quality CH-SGs should be engaging and enjoyable, focusing on user involvement, positive gameplay experiences, fluidity, and feeling of control, as well as supporting social interaction to ensure users find enjoyment in the game.

Additionally, this study emphasizes the promotion of continued usage of CH-SGs. Parthasarathy once noted in the context of information systems that retaining an existing customer costs five times less than acquiring a new one (Yu and Cheng, 2012). Long-term user feedback helps understand the ever-changing needs of users, improving and innovating game products. Continued usage intention is defined as the subjective desire of users to continue using an information system after initial use (Niu, 2020). In the field of continued usage intention, the technology acceptance model (TAM) has been widely applied and has become one of the most mature theories in this area (Sun, 2023). As online games, fundamentally, belong to the category of information systems (Li, 2011), many scholars have used the TAM model in their studies of online game users’ continuous usage (Min, 2018). The TAM model predicts the degree to which a person accepts, uses, or rejects information technology, effectively explaining the behavior of users in adopting new technologies and its influencing factors (Sun and Guo, 2023), and is the most concise and universal model for studying the initial intention and continuous intention to adopt technology (Yao, 2023). Davis (1989) first proposed the TAM model while studying users’ acceptance of computers. He suggested that perceived usefulness is an extent indicator, reflecting the individual’s perception of how an information system enhances their work efficiency. Higher perceived usefulness increases the likelihood of taking actions; perceived ease of use reflects the individual’s perceived difficulty in using the system, with higher ease of use leading to more likely user behavior (Jia, 2019). Later in 2000, Davis and Venkatesh revised the TAM to include subjective norms based on numerous studies (Dong, 2017). Scholars like Sun Shaojun integrated the TAM and Stimulus-Organism-Response (SOR) models to establish a continuous usage behavior model for game live streaming (Sun and Zhang, 2017); Wu Xuanying integrated the TAM model and flow theory to construct a stickiness influence model for mobile game users (Wu, 2019); Zhang Di combined the SOR, TAM, information success model, and immersion theory to build a model for mobile game user churn factors (Zhang, 2016). These studies confirmed the positive impact of perceived usefulness, ease of use, and subjective norms on continuous usage. Therefore, this study incorporates these three variables into the continuous usage indicators, defined as follows:

1. Perceived usefulness: The degree to which a user perceives that using a particular information system will help them perform tasks better or gain other benefits. When users perceive that using such games meets their expected benefits, they will feel satisfied and continue using the games.

2. Perceived ease of use: The degree to which a user perceives the ease or difficulty of using a particular information system. If users encounter difficulties when experiencing a game, they may feel dissatisfied and abandon it, whereas if they find the game easy to operate, they will feel satisfied and continue using it.

3. Subjective norms: The degree to which a user’s behavior is influenced by group dynamics, such as conformity, suggestion, and obedience. Therefore, an individual’s behavior is often influenced by the group. If users perceive that no one in their social circle is using the game, they may feel that the game is futile and abandon it due to a lack of positive gaming experience.

2 Methodology

2.1 Constructing the CH-SG evaluation system model

This study posits that only by establishing an evaluation rubric fitting the CH-SG context can effective assessments be conducted in all aspects, referencing the UEREG-CH evaluation scale with high similarity prior to construction (Table 1). This scale was developed by the team at the Educational Game Center of Nanjing Normal University. It is guided theoretically by Hartmann’s improved model of user experience influencing factors, reviews the current state of research on factors influencing user experience, and analyzes the characteristics and elements of serious game user experiences. The rubric identifies key terms in user experience evaluation through user testing, and then refines the primary and secondary indicators of the index system through two rounds of expert consultation, thus ultimately determining this evaluation rubric (Shen, 2015).

Table 1

Table 1. The UEREG-CH evaluation rubric.

This paper, based on literature research, identifies 14 influencing factors: ‘Intellectual satisfaction, Immersion satisfaction, Achievement satisfaction, Social satisfaction, Pleasure satisfaction, System quality, Service quality, Game design, Brand image, Learning objectives, Entertainment objectives, Perceived usefulness, Perceived ease of use, and Subjective norms’. These factors are used as secondary indicators in the CH-SG evaluation system model. Referring to the dimensions in the UEREG-CH evaluation rubrics, these 14 secondary indicators are categorized into 6 primary indicators and further expanded into 32 tertiary indicators, thereby preliminarily forming the CH-SG evaluation system model (Liu et al., 2021) (Table 2).

Table 2

Table 2. Initial establishment of the CH-SG evaluation system model.

2.2 The Delphi method

The Delphi method is a group decision-making behavior that involves a series of concentrated expert surveys. Based on the collective professional knowledge and experience of experts, supplemented by controlled feedback of opinions, the process aims to achieve the greatest consensus among a group of experts (Dai, 2023). The advantage of the Delphi method is that it avoids direct confrontation between experts and structures the discussion process of the expert group, making them more effective when discussing complex issues (Liu et al., 2021). It is not only used in the field of prediction but also widely used in the establishment of various evaluation systems and the determination of specific indicators (Liu et al., 2011). The 10 experts who filled out the questionnaire on game studies were strictly selected and all had at least 2–3 years of research experience in serious games, educational games, and online games, with at least 2 years of game testing experience and at least one experience in educational game evaluation (Liu et al., 2023).

Ten experts were consulted over two rounds regarding the initially established CH-SGs evaluation system model, using both electronic mail and paper questionnaires to collect expert opinions and to add, delete, or modify indicators. The specific process included: clarifying evaluation objectives, selecting experts, issuing questions, experts’ assessment of questions, and revising based on expert feedback (Liu et al., 2011). The first round of questionnaires involved experts’ familiarity with each level of indicators, suggestions for additional or deletable indicators, and other questions or recommendations. Based on expert suggestions, similar indicators in the CH-SG evaluation system model were merged, unreasonable ones were deleted or broken down, and missing indicators were added (Dong, 2013). In the first round of questionnaire consultation, the primary indicator “Identity dimension” was not described accurately and was loosely connected logically with the secondary indicators “Brand image” and “Subjective norms,” hence it was changed to “Social dimension.” The secondary indicator “Game design” was too broad and encompassed too much, so it was changed to “Interface design” based on the tertiary indicators. Complex descriptions in tertiary indicators were further refined, such as “D1 Game interface operation setting is reasonable, with the characteristics of the cultural heritage theme,” which was split into “D1 Interface design features cultural heritage theme” and “D2 Interface menu layout is reasonable and orderly” for easier understanding and scoring by experts or users. “D4 Immersed in the game’s artistic roles, creating one’s own story background” was simplified to “D5 Immersion in creating one’s story with artistic roles,” etc. Other less comprehensive descriptions were expanded, thus increasing the original 32 tertiary indicators to 38. Overall, the experts acknowledged the feasibility of the indicators, with minimal modifications made (Table 3).

Table 3

Table 3. Revised CH-SG evaluation system model.

The revised CH-SG evaluation system model consists of 6 primary indicators, 14 secondary indicators, and 38 tertiary indicators. In the second round of questionnaire surveys, the opinions of the experts converged, with the agreement rate for each level of indicators reaching 100%. Therefore, no further modifications were needed, and the CH-SG evaluation system model was finally determined to be used as the AHP for the next phase of research.

2.3 AHP hierarchical analysis

The AHP is a decision-making method for complex and vague issues, particularly suitable for issues difficult to be fully quantified. It is a flexible, convenient, and practical multi-criteria decision-making method (Zhang et al., 2014). The principle of AHP is to organize and hierarchize problems, constructing a structured model with various levels. Complex issues are decomposed into elements, which are then grouped according to their characteristics and certain rules into several levels. Elements within each level have approximately the same status. An orderly, step-by-step hierarchy is established based on the affiliation relationships between levels, where elements of a higher level dominate related elements of the next lower level (Dong, 2013). In this step-by-step hierarchical model, decision-makers score the importance of each level quantitatively based on their judgment of certain objective facts. This is done by determining the relative importance of elements within each level through pairwise comparison and quantitatively expressing this importance, thereby establishing judgment matrices. Mathematical models are then used to calculate the relative importance weights of each indicator within the judgment matrices of each level. Finally, the relative importance weights of all indicators concerning the evaluation objective are calculated through corresponding operations (Xu, 2009). In the CH-SG hierarchical model of this paper, the relationship between primary, secondary, and tertiary indicators is one of containment and being contained, also known as the upper and lower level relationship in the decision-making field. In the hierarchical model, “CH-SG Evaluation System Model” A is the goal layer, B1-B6 are primary indicators, C1-C14 secondary indicators are criterion layers, and D1-D38 tertiary indicators are the scheme layer. Experts compare and score the indicators in the evaluation system (Table 4), generate judgment matrices, and calculate the weight values of each indicator.

Table 4

Table 4. Expert questionnaire scoring 1–9 scale method.

Steps for calculating weights using the AH involve the hypothetical judgment matrix A:

$A=\left[\begin{array}{cccc}\hfill a_{11}\hfill & \hfill a_{12}\hfill & \hfill \cdots \hfill & \hfill a_{1n}\hfill \\ \hfill a_{21}\hfill & \hfill a_{22}\hfill & \hfill \cdots \hfill & \hfill a_{2n}\hfill \\ \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill \cdots \hfill \\ \hfill a{}_{n1}{}^{}\hfill & \hfill a_{n2}\hfill & \hfill \cdots \hfill & \hfill a_{nn}\hfill \end{array}\right]$

The calculation steps for the root method are as follows:

(1) Calculate the product of the elements in each row of the judgment matrix A:

$m_t={\displaystyle \prod }_{j=1}{a}_{ij},i=1,2\cdots n$

(2) Calculate the nth root of mi;

$w_i^{\ast }=\sqrt[n]{m_i}$

(3) Normalize the vector $W^{\ast }={\left(w_1^{\ast },w_2^{\ast },\cdots ,w_n^{\ast }\right)}^T$

$w_i=\frac{w_i^{\ast }}{{\displaystyle \sum }_{i=1}^n{w}_i^{\ast }}$

The weights obtained are the desired weight vector, w1, w2,..., wn, which corresponds to the weight value of each element.

Matrix consistency test method:

(4) Calculate the consistency index

$CI=\frac{{\lambda }_{\text{max}}-n}{n-1}$

The calculation formula for the largest eigenvalue is:${\lambda }_{\text{max}}$

${\lambda }_{\text{max}}=\frac{1}{\mathrm{n}}{\displaystyle \sum }_{i=1}^n\frac{{\left(\mathit{Aw}\right)}_i}{w_i}$

(5) Look up the corresponding Average Random Consistency Index RI.

(6) Average Consistency Ratio (CR).

$CR=\frac{CI}{RI}$

When CR is less than 0.1, the consistency of the judgment matrix A is considered acceptable; otherwise, some adjustments to the judgment matrix are required (Zhu et al., 2020) (Refer to the appendix for the detailed calculation process).

Based on the scoring from 10 experts, a judgment matrix is constructed, and the results of the weight calculation using the AHP are as follows:

Table 5 shows that the CR value is <0.1, passing the consistency test. The smaller the CR value, the better the consistency of the judgment matrix, indicating the rationality of this weight determination method. There is no need to modify the judgment matrix. The derived weight set can reflect the importance of each indicator, and the weight distribution is reasonable (Li and Guo, 2004).

Table 5

Table 5. Results of the consistency test.

Through the above research, the weight coefficients of each level of indicators for the CH-SGs evaluation were determined (Table 6), forming a complete evaluation system. Subsequently, the fuzzy comprehensive evaluation method will be used by users to assess the overall quality of this system (Zhang X. W., 2023).

Table 6

Table 6. Weight values of each indicator in the CH-SG evaluation system model.

2.4 Fuzzy comprehensive calculation

Fuzzy comprehensive calculation is based on fuzzy mathematics and applies the principle of fuzzy relation synthesis to quantify factors with unclear boundaries and difficult quantification, conducting a comprehensive evaluation of the belonging level of an entity under evaluation from multiple factors (Ge and Sun, 2014). The basic procedure is to first determine the factor domain of the evaluation object, i.e., the number of indicators, and second to determine the evaluation level domain, typically taking integers, and for the convenience of expert judgment, the levels are divided as finely as possible (Li and Deng, 2019). This study adopts an evaluation set of levels: poor, below average, average, good, and excellent. Finally, the weights of each indicator obtained from the AHP are combined to calculate the final fuzzy evaluation result (Wang, 2017). The fuzzy comprehensive calculation is as follows:

1 First, determine the factor domain of the evaluation object (i.e., the fuzzy factor set): let there be p evaluation indicators, U = {u1, u2,..., up}.

2 Determine the weight vector of evaluation factors and the evaluation level domain (i.e., the evaluation set): in fuzzy comprehensive evaluation, determine the weight vector of evaluation factors: W = {a1, a2,..., ap}. The relative importance order of evaluation indicators is usually determined using the AHP or other methods to establish the weight coefficients, and normalize them before synthesis. Let the evaluation level domain (i.e., the evaluation set) be V = {v1, v2,..., vm}, with each level corresponding to a fuzzy subset, i.e., the level set.

3 Establish a fuzzy relation matrix (i.e., the membership degree matrix): after constructing the level fuzzy subsets, quantify the evaluated entity from each factor ui (i = 1, 2,..., p), i.e., determine the membership degree of the evaluated entity to the level fuzzy subsets (R|ui). Through survey questionnaires, obtain the scoring of each person for each level of the indicator, with the proportion of the number of people scoring each evaluation level to the total number of experts as the membership degree, thereby establishing the single factor fuzzy comprehensive evaluation matrix.

The membership degree matrix for a single element (R|ui) = (ri1, ri2,..., rim), where for each level of a single element, the membership degree is: $r_{ij}=\raisebox{1ex}{$c_{ij}$}\!\left/ \!\raisebox{-1ex}{$c$}\right.$ .

Cij represents the number of people who chose level vj for indicator i, and c represents the total number of experts participating in the evaluation.

Calculate the membership degree for p evaluation indicators to obtain the overall fuzzy relation matrix, as follows:

$R=\left[\begin{array}{l}R|u1\hfill \\ R|u2\hfill \\ \dots \dots \hfill \\ R|u_p\hfill \end{array}\right]=\left[\begin{array}{cccc}{r}_{11}& r_{12}& \dots & r_{1m}\\ r_{21}& r_{22}& \dots & r_{2m}\\ \dots & \dots & \dots & \dots \\ r_{p1}& r_{p2}& \dots & r_{\mathit{pm}}\end{array}\right]$

In this element, the element rij in the ith row and jth column represents the membership degree of a certain evaluated entity ui in terms of the factor to the level vj fuzzy subset.

4 Synthesize the fuzzy comprehensive evaluation result vector: combine W with the overall fuzzy relation matrix R of each evaluated entity to obtain the fuzzy comprehensive evaluation vector B for each evaluated entity, i.e.:

B=W*R = (a1,a2,…,ap)* $\left[\begin{array}{cccc}\hfill r_{11}\hfill & \hfill r_{12}\hfill & \hfill \dots \hfill & \hfill r_{1m}\hfill \\ \hfill r_{21}\hfill & \hfill r_{22}\hfill & \hfill \dots \hfill & \hfill r_{2m}\hfill \\ \hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill \\ \hfill r_{p1}\hfill & \hfill r_{p2}\hfill & \hfill \dots \hfill & \hfill r_{\mathit{pm}}\hfill \end{array}\right]$ =(b1,b2,…,bm).

bi represents the degree of membership of the evaluated entity to the level vj fuzzy subset from an overall perspective.

5 Analyze the fuzzy comprehensive evaluation result vector: the most commonly used method in practice is the principle of the maximum degree of membership. However, in some cases, its use can be forced, leading to the loss of a lot of information, and even producing unreasonable evaluation results. This paper uses the method of weighted average to determine the membership level, allowing for the ranking of multiple evaluated entities according to their level positions (refer to the appendix for the detailed calculation process).

To verify the scientific and practical nature of the CH-SG evaluation system model discussed earlier, this paper utilizes the weights of indicators determined by the AHP and applies fuzzy comprehensive calculation to evaluate the comprehensive score of this system. For this purpose, a questionnaire targeted at users of CH-SGs is designed (refer to the appendix for details). Evaluators simply need to select the corresponding levels for each indicator on the scale based on their judgment (Zhang, 2015). A total of 400 questionnaires were distributed, with 392 valid responses collected. To ensure the reliability and validity of the questionnaire, the sample data was analyzed using the Cronbach’s alpha coefficient in SPSS, yielding an overall Cronbach’s alpha of α = 0.962, indicating good internal consistency. Exploratory factor analysis was also conducted, with a Kaiser-Meyer-Olkin (KMO) value of 0.978 and a significant level of Bartlett’s test of sphericity at 0.000, indicating significant correlations between variables and passing the test of reliability and validity. This paper analyzes the collected questionnaires, calculating the frequency of each level—poor, below average, average, good, and excellent—and the corresponding values of each indicator for fuzzy comprehensive calculation. Through these calculations, the fuzzy evaluation values of each level of indicators in the CH-SGs evaluation system are derived, providing reference for further analysis of the evaluation rubric.

The fuzzy comprehensive calculation resulted in the evaluation values for each level of indicators in the CH-SG evaluation rubric (Table 7). The overall primary level fuzzy comprehensive evaluation value was 3.4228, indicating an overall level between average and good. This suggests that the evaluation rubric is recognized by users, and the descriptions of evaluations in these games closely align with user experiences (Liang, 2019). According to the data, the primary indicators are ranked by their importance as follows: B4 Gamification at 3.5371 > B5 Demand at 3.4546 > B6 Social dimension at 3.4473 > B3 Educational value at 3.4307 > B1 Artistic dimension at 3.4223 > B2 Usability at 3.2355. The B2 Usability at 3.2355 is notably lower than the overall evaluation value of 3.4228, indicating that this indicator is not ideal and suggests a mismatch between expert and user standards in evaluating the game.

Table 7

Table 7. Fuzzy evaluation values of each indicator in the CH-SG Evaluation Rubric.

3 Results

The primary indicator B4 Gamification at 3.5371 is higher than the overall comprehensive evaluation value, indicating that enhancing gameplay positively impacts the evaluation of the game and that users are most concerned with the gameplay aspect of CH-SGs. In the secondary indicators, C8 Perceived ease of use at 3.6105 and C10 Pleasure satisfaction at 3.5510 are higher than the gamification indicators, enhancing the gamification. In the tertiary indicators, D21 Easy download, installation, payment for the game at 3.6301, D22 Easy learning and operation of the game at 3.5714, D23 Establishing multi-modal experiences such as graphics, sound effects, and tactile at 3.7883, and D29 Participation in offline experiences and purchasing cultural peripheral products at 3.6633 are all higher than the gamification, having a positive impact. Designers should consider how to make the game user-friendly, aligning with the characteristics of CH-SGs as light-function games, requiring minimal time and effort from users, facilitating knowledge acquisition, character cultivation, enjoyment, and mental pleasure, and providing different experiences in sound effects, tactile modes, and offline activities, all of which will effectively increase users’ willingness to continue using the game (Gong and Liang, 2018).

The primary indicator B5 Needs at 3.4546 is higher than the overall evaluation value, positively impacting it. In its secondary indicators, C12 Achievement satisfaction at 3.4719 is slightly higher than the demand dimension. In the tertiary indicators, D31 Making many friends through cooperation and communication with other users at 3.4745, D32 Long-term contact with other users and feeling care among friends at 3.8954, and D33 The desire to win in challenges and competitions at 3.5434 are all higher than the demand dimension, positively impacting it. Designers should focus on expanding users’ social pathways and convenience, fully satisfying users’ desires for social interaction and emotional communication, and increasing the game’s competitiveness and positive feedback to stimulate users’ sense of achievement, thereby promoting their willingness to continue using the game (Chang, 2015).

The primary indicator B6 Social dimension at 3.4473 is higher than the overall evaluation value, positively impacting it. In its secondary indicators, C13 Brand image at 3.5485 is higher than the social dimension indicator. In the tertiary indicators, D36 Production companies with high recognition or liked experts and stars at 3.7474 and D35 High recognition or liked cultural IP themes at 3.4490 are both higher than the primary indicator. This indicates that production companies with high recognition are a guarantee of high-quality games for users. Choosing well-known experts and stars for advertising endorsements can effectively attract users’ attention. Alternatively, choosing popular, well-received IPs for game development can arouse users’ curiosity and encourage continued use (Wang, 2022).

The primary indicator B3 Educational value at 3.4307 is slightly higher than the overall evaluation value, also having a promotive effect. In its secondary indicators, C5 Learning objectives at 3.5879 and C6 Intellectual satisfaction at 3.6956, are higher than the primary indicator. In the tertiary indicators, D12 Continuous and systematic learning content at 3.7908, D15 Accurate and reliable cultural heritage knowledge in various forms at 3.8316, D16 Level design conforming to learning rules and user cognitive level at 3.4413, D17 Enhancing the ability to complete tasks and cooperate intellectually with others at 3.6939, and D18 Stimulating curiosity in learning traditional culture at 3.6990, all these indicators are higher than the primary indicator. The educational value is where serious games differ from other entertainment games in value, thus gaining more attention from users (Li and Sun, 2019). Designers, while developing the educational goals of the game, must first ensure that the integrated cultural heritage knowledge is accurate and diverse, maintaining users’ freshness toward knowledge. Secondly, they must consider whether the designed cultural knowledge and skills are systematic, enabling users to learn progressively rather than a one-time experience that cannot be sustained. They must also consider whether the knowledge broken down into stages of task levels conforms to the users’ cognitive level, ensuring that users can overcome the task with some effort or seeking help, without feeling a sense of failure and giving up. Finally, users should gain from the use and learning of the game, either through increased positive rewards or by setting stage-wise tasks and evaluation rewards, ensuring that users have gains at every stage after completing tasks or cooperating with other users, and developing new cognition and curiosity about cultural heritage. These aspects can effectively stimulate users to continue using the game (Huang, 2023).

The primary indicator B1 Artistic dimension at 3.4223 is close to the overall evaluation value. In its secondary indicators, C2 Immersion satisfaction at 3.6786 is higher than the artistic dimension, positively impacting it. In the tertiary indicators, D4 High clarity, exquisite realism, and interesting visual scenes at 3.6709, D5 Immersion in creating one’s own artistic characters and stories at 3.6378, and D6 Helps to forget problems and stresses of real life at 3.8010, all these indicators are higher than the primary indicator value. This suggests that users hope to immerse themselves in a virtual world through the game, temporarily escaping life’s troubles for mental rest. To achieve such immersion (Wang et al., 2019b), designers need to enhance the visual effects of game scenes and endow characters with artistry, strongly attracting users’ attention, and giving them more control to create their own characters and stories (Zhang, 2018). Stimulating these aspects can enhance users’ willingness to continue using such games.

The primary indicator B2 Usability at 3.2355 is below the overall evaluation value, indicating that this indicator is not ideal and that there is a mismatch between experts’ and users’ standards in evaluating the game. In its secondary indicators, C4 Service quality at 3.5000 is higher than both the primary indicator and the overall evaluation value, suggesting that users are more concerned about the service attitude and quality of the game production company. In its tertiary indicators, D10 Conducting promotional activities and fulfilling promises to users at 3.4745 and D11 Customer service staff being competent and promptly resolving user issues at 3.5510, these indicators indicate that if game production companies deceive users during use or fail to fulfill various promotional promises, or if customer service staff have a poor attitude, lack professional competence, or do not promptly resolve users’ issues in the game, it will lead to user dissatisfaction and the desire to quit the game.

4 Discussion

4.1 Comparative analysis of CH-SGs and UEREG-CH setting principles

This section compares and analyzes the CH-SGs evaluation model and UEREG-CH based on three principles of rubric setting: purposefulness, systematicity, and measurability. Purposefulness is the fundamental basis for establishing evaluation rubrics and directly affects the conduct of evaluation activities. Systematicity requires that the evaluation rubric should comprehensively reflect the requirements of the evaluation objective, without missing any significantly important indicators. Additionally, indicators should be mutually independent, without causal or overlapping relationships, clearly hierarchical, and collectively form an organic whole. Measurability demands that specific indicators should be defined in operationalizable language, allowing for conclusions to be drawn through actual observation or measurement. This principle concretizes and quantifies abstract evaluation objectives, and is one of the basic conditions for the practical application of evaluation rubrics and the generation of reliability (Shen, 2015).

Purposefulness: The UEREG-CH primarily reflects the sensory experiences and real needs of serious game users, including students and teachers. The purpose of the evaluation is to select educational games that meet the autonomous learning needs of learners and adapt to different learning styles. The CH-SGs in this study primarily focuses on user needs and experiences to promote continued use. It provides scientific and reasonable guidance and evaluation standards for designers and developers of these games and assists users and learners in selecting suitable CH-SGs.

Systematicity: The UEREG-CH, centered on user experience as the evaluation objective, sets up 5 primary indicators, 22 secondary indicators, and corresponding descriptions and weights, with a tightly interconnected hierarchical relationship. For example, under “Gamification,” there are secondary indicators like “Appropriate challenge” and “Reasonable incentives.” However, a shortcoming is the similarity or lack of distinction between some secondary indicators, such as “Organized interface menu” and “Clear interface design”; and “Easy to remember” and “Easy to learn.” CH-SGs in this study sets up 6 primary indicators, 14 secondary indicators, and 38 tertiary indicators, assigning weights to each. Focusing on user experience, it emphasizes promoting continued use and includes content related to user motivation and game quality. The considered range of indicators is broader, deeper, and more detailed, with each level of indicators being interrelated yet distinct and easy to differentiate.

Measurability: Both the UEREG-CH and CH-SGs achieve clarity and conciseness in the design of indicator descriptions. In designing the scale method, descriptive terms like “excellent,” “good,” “average,” “poor,” and “below average” are used. Evaluators score according to the degree to which the last level of indicators meets the description.

4.2 Comparative analysis of CH-SGs and UEREG-CH indicator dimensions

4.2.1 Gamification

The UEREG-CH breaks down “Gamification” into five secondary indicators: challenge, reward, content, plot, and rules, which lacks comprehensive consideration. This study dissects the “Gamification” into “Perceived ease of use,” “Entertainment objective,” and “Pleasure satisfaction,” along with nine tertiary indicators from D21 to D29 (Table 8). It posits that the prerequisite for user engagement in a game is its perceived ease of use, as simple operability leads users to further explore the game’s entertainment aspects. The “Entertainment objective” involves users empathizing with characters, controlling the game’s direction, achieving self-realization, and immersing in the experience, leading to the satisfaction of pleasure and making the game a part of their life. This study views gamification as a distinguishing feature of CH-SGs compared to other educational games, necessitating a multi-faceted consideration of users’ actual operational needs.

Table 8

Table 8. Comparison of indicator dimensions.

4.2.2 Demand

The UEREG-CH decomposes “Demand” into two aspects: “Game learning content meets or exceeds learner needs” and “Game functionality meets or exceeds learner needs.” This study finds this approach overly simplistic and lacking a clear distinction from “Educational” aspects. CH-SGs categorizes the “Demand dimension” into “Social satisfaction” and “Achievement satisfaction,” with five tertiary indicators from D30 to D34 (Table 8). It believes that users’ intrinsic demands stem from two sources: achievement satisfaction through winning in the game mechanics, and social satisfaction from connections made with other users. These needs are interdependent, as victory in the game leads to a desire for social display and sharing, while collaboration and communication with others facilitate winning.

4.2.3 Social aspect

The UEREG-CH does not set this indicator (Table 8). However, this study argues that games, as products, have strong social attributes, encompassing both users’ social attributes and the game’s societal aspects. CH-SG users often belong to culturally and economically higher echelons, greatly influenced by social norms, and their actions are affected by others in society. If close friends or experts recommend such games, users are likely to try them. The “Brand image” represents the game product’s social attribute. If the game’s production company is well-known, endorsed by celebrities or experts, or developed from a well-known IP, it tends to be favored and sought after by users.

4.2.4 Educational aspect

In the UEREG-CH, “Educational value” is divided into five secondary indicators: “Clear and timely knowledge feedback,” “Levels compliant with learning rules,” “Reliable content with flexible and diverse forms,” “Clear learning objectives,” and “Balanced skills and challenges.” This categorization appears somewhat generic and necessitates further detailed expansion. CH-SGs decompose the “Educational value” into three secondary indicators: “Learning objectives,” “Intellectual satisfaction,” and “Perceived usefulness,” along with nine tertiary indicators from D12 to D20 (Table 8). “Learning objectives” play a crucial role in the quality aspect of CH-SGs games and serve as a distinctive feature differentiating them from other games. Users experience intellectual satisfaction only after completing tasks set by the “Learning objectives.” This sense of intellectual satisfaction, being one of the primary motives for initial game usage, leads to the perception of usefulness, thereby encouraging users to continue using the game. Thus, the relationship among these three indicators is interlinked, each supporting and leading into the next, collectively embodying the educational aspect from multiple perspectives.

4.2.5 Artistic aspect

The UEREG-CH deconstructs “Perceived aesthetics” into interface design, menu, elements, production, etc., which this study believes lacks an overall requirement for the game’s artistry and attention to users’ perception of art. CH-SGs breaks down “Artistic aspect” into two secondary indicators: “Interface design” and “Immersive satisfaction,” along with six tertiary indicators from D1 to D6 (Table 8). This study suggests that the artistic indicators should reflect cultural heritage themes and emphasize creativity more than other educational games. Only when users perceive a unique artistic atmosphere in the interface can they immerse themselves in their game characters and forget the stresses of life.

4.2.6 Usability

The UEREG-CH decomposes “Usability” into indicators such as ease of memorization, operation, learning, absence of faults, and timely feedback. However, these indicators are subjective and semantically vague, overlapping with others. This study breaks down “Usability” into “System quality” and “Service quality,” with five tertiary indicators from D7 to D11 (Table 8). A game’s usability is first judged by system quality; a game is considered usable only when the system is stable, frequently optimized, and has good guidance and synchronous feedback. Additionally, customer service’s timely problem resolution and fulfillment of promises to users are essential; otherwise, users may develop negative perceptions of the game’s usability and abandon it.

4.3 Comparative analysis of indicator weights in CH-SGs and UEREG-CH

4.3.1 Weight structure

Both the UEREG-CH and CH-SGs have tree-like indicator systems, but the former does not highlight the characteristics of various serious games as effectively as the latter, which is tailored for CH-SGs. Regarding the method of weight determination, the UEREG-CH uses group AHP cluster analysis, while the CH-SG combines the Delphi method, AHP, and fuzzy comprehensive calculation, blending qualitative and quantitative research (Table 9).

Table 9

Table 9. Comparison of indicator weights.

4.3.2 Important indicator weights

In the UEREG-CH, “Usability” ranks first, followed by “Educational value,” “Demand,” “Gamification,” and “Perceived aesthetics.” This prioritization shows that serious games in China focus more on basic user needs, emphasizing usability and educational aspects but are weaker in gamification and artistic aspects, lacking a balance between learning and gaming. In CH-SGs, “Gamification” ranks first, followed by “Demand,” “Social aspect,” “Educational value,” “Artistic dimension,” and “Usability.” This suggests that CH-SGs users prioritize leisure and entertainment in games, personal needs fulfillment, and social interaction. However, the educational aspect is less emphasized, possibly because CH-SGs, while educational, are perceived as weaker in function compared to pure educational games. This perception might stem from the games’ design issues that fail to emphasize learning objectives through game mechanics, leading to the impression that their educational function is weak. The lesser importance of the artistic aspect could be due to the convergence of artistic styles in current CH-SGs, predominantly adopting traditional Chinese styles and lacking in diversity and innovation. Usability ranks last because system quality generally meets users’ needs in current CH-SGs, and service quality has a minor impact on user experience, thus receiving less attention from users (Table 9).

5 Conclusion and limitations

This study confirms that an effective and comprehensive evaluation rubric for CH-SGs can be constructed by focusing on usage motivation, game quality, and continuous usage. Previous research on serious games has focused on the educational aspect and the improvement of learners’ outcomes. However, empirical studies primarily aimed at collecting evidence of learning effectiveness have design flaws, leading to a shift in the research trend beyond just educational aspects. This study observes that the educational function of CH-SGs in China is relatively weak and needs to be integrated into the game mechanics to achieve learning objectives. Compared to other serious games, CH-SGs have a distinct cultural and artistic attribute. Therefore, evaluation rubrics developed for other serious games are not suitable for CH-SGs. To reflect the characteristics of CH-SGs and the needs of their user group, this study distinguishes them from other serious games across six dimensions: “Artistic,” “Usability,” “Educational value,” “Gamification,” “Demand,” and “Social aspect.” The secondary and tertiary indicators emphasize cultural heritage elements, such as breaking down the “Educational value” dimension into technical indicators like “Learning objectives” and non-technical indicators like “Intellectual satisfaction” and “Perceived usefulness,” with tertiary indicators reflecting elements of cultural heritage, such as “Accurate and reliable cultural heritage knowledge in various forms,” “Stimulating curiosity in learning traditional culture,” and “Enhancing the level of traditional cultural knowledge and skills.” Empirical research has revealed a departure from the traditional approach in serious game studies, which often emphasizes a balance between educational and gameplay aspects. It was found that users of CH-SGs tend to favor “Gamification” more, with “Educational” factors ranking only fourth in terms of importance. Therefore, it is recommended that designers of CH-SGs tailor the balance between gaming and learning aspects according to the specific theme and type of each game, rather than uniformly pursuing a balance between the two.

This study also validates that the CH-SGs evaluation rubric is recognized by users, not just experts. Involving users in the evaluation process is crucial, as they have the right to assess the games. Therefore, the evaluation rubric is developed around users’ diverse needs, formed through expert consultations and the AHP method, and further validated by user surveys and fuzzy comprehensive calculations. This approach ensures significant user participation. The study also believes that a game’s true appreciation by users depends on their continued use and the lasting, long-term impact it has on them. Thus, the rubric differs from previous studies by including long-term indicators, focusing on users’ experiences, attitudes, enthusiasm, persistence, and other non-technical aspects (Yu, 2016). Starting from the theory of continued usage intention, the study sets “Perceived usefulness,” “Perceived ease of use,” and “Subjective norms” as secondary indicators, and “Continuous and systematic learning content,” “Long-term contact with other users and feeling care among friends,” etc., as tertiary indicators to assess the long-term impact of CH-SGs on users. True feedback for game iterative improvement can only be obtained through long-term usage by users.

This study also has limitations in its research methodology. Although user needs and evaluations were incorporated in the fuzzy calculation process of establishing the evaluation rubric, there are shortcomings in the composition of the user sample data collected. The survey respondents were all users who had experienced CH-SGs, but many potential users were not surveyed. Additionally, the user groups need further segmentation, as different groups may have varying evaluations of serious games, leading to instability in results. The methods for collecting user needs should also be expanded beyond questionnaires, including in-depth interviews and group discussions for a more detailed understanding of user needs, thus improving the accuracy of the evaluation results. In the Delphi and AHP methods, the study included opinions from an expert panel, which was not fully comprehensive, being predominantly from the field of game studies. This led to subjectivity and bias in the evaluation system and weightings. The inclusion of experts from educational sciences, cultural heritage inheritors, game industry professionals, and other cultural institutions is needed for a more balanced perspective. Moreover, the indicators in the evaluation rubric are not fully comprehensive, with many derived from educational game evaluations. Regarding the design scales for evaluation indicators, this study used a 5-point rating scale, and the scoring of each indicator primarily relied on subjective user perceptions without more detailed criteria. Therefore, how to delineate evaluation levels more finely and completely is a topic for future discussion. Lastly, as the CH-SGs evaluation rubric developed in this study has significant cultural specificity within the serious games domain, whether it can serve as a referable and extendable evaluation paradigm for other types of serious games requires continuous validation in future research.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author contributions

PM: Writing – original draft. DC: Writing – review & editing.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2024.1300686/full#supplementary-material

References

Cai, J. X. (2020). A brief discussion on the development of multimedia Technology in the Protection and Inheritance of intangible cultural heritage. Popular Liter. Art. 2020, 10–12.

Google Scholar

Caserman, P., Hoffmann, K., Müller, P., Schaub, M., Straßburg, K., Wiemeyer, J., et al. (2020). Quality criteria for serious games: serious part, game part, and balance. JMIR Serious Games 8:e19037. doi: 10.2196/19037

PubMed Abstract | Crossref Full Text | Google Scholar

Chang, K. (2015). Research on user experience of online social networking platforms based on needs and behaviors. Master’s thesis. Huazhong Normal University

Google Scholar

Chen, Y. R. (2023). Use and satisfaction of Mobile online games - taking female users as an example. Master’s thesis. Southwest Jiaotong University.2023-12-01

Google Scholar

Chen, Y. H., and Liu, Y. Y. (2019). Analysis on the communication path of socialist core values based on serious games [J]. Modern Communication (Journal of Communication University of China). 41, 96–100. Available at: https://kns.cnki.net/kcms2/article/abstract?v=fCqJ37DMrBmO0j8ybP1wbOTkUhWQKOODHUFHq-VxLY04p21sd8h1mkMlzTiS-gL0FEWX3NRP1LjnvFgSHJqQVGxiyitJO47dKb5rh4CElgZ7X24ivdBkKO5DTWNnJL-RByq02Fuwfag=&uniplatform=NZKPT&language=CHS

Google Scholar

Dai, Q. (2023). Exploration of constructing an evaluation system for elementary school calligraphy teaching using the Delphi method. Master’s thesis. Hainan Normal University

Google Scholar

Davis, F. D. (1989). Technology acceptance model: TAM. Al-Suqri, MN, AS Al-Aufi: Information Seeking Behavior and Technology Adoption, 205–219.

Google Scholar

Dong, Z. Z. (2013). Construction research on user experience evaluation system of educational games. Master’s thesis. Nanjing Normal University

Google Scholar

Dong, Y. (2017). Research on the factors influencing the reaction intention of Mobile social network users to contact product recommendation information. Master’s thesis. Jilin University

Google Scholar

Fang, Y. C. (2023). Research on the factors influencing the continued use intention of game live streaming platforms. Master’s thesis. Nanchang University

Google Scholar

Gardner, H., and Hatch, T. (1989). Educational implications of the theory of multiple intelligences. Educ. Res. 18, 4–10. doi: 10.3102/0013189X018008004

Crossref Full Text | Google Scholar

Ge, Y. K., and Sun, Y. J. (2014). Quantitative analysis of the causes of credit risk in commercial banks - based on fuzzy analytic hierarchy process. Sci. Technol. Manag. 16, 106–109. doi: 10.16315/j.stm.2014.01.014

Crossref Full Text | Google Scholar

Gong, C. B., and Liang, P. P. (2018). From “user experience” to “media user experience” - an analysis of several basic issues on media user experience. Journal. Commun. Review 354, 67–74.

Google Scholar

Holbrook, M. B., and Hirschman, E. C. (1982). The experiential aspects of consumption: consumer fantasies, feelings, and fun. J. Consum. Res. 9, 132–140. doi: 10.1086/208906

Crossref Full Text | Google Scholar

Hu, B. J. (2013). An empirical study on the relationship between customer perceived value, customer satisfaction, and customer loyalty in online games. Master’s thesis. South China University of Technology

Google Scholar

Huang, Z. L. (2023). Research on the influence of user participation in Mobile game products from the perspective of use and gratification theory. Model. Simul. 12, 2505–2512. doi: 10.12677/MOS.2023.123230

Crossref Full Text | Google Scholar

Jia, J. P. (2019).Research on the continued acceptance intention of Mobile game users based on the expectation confirmation model. Master’s thesis. South China University of Technology

Google Scholar

Jiang, Y. W. (2011). Analysis of business model innovation of online game operators. Master’s thesis. Southwest University of Finance and Economics

Google Scholar

Jin, Y., and Yang, K. (2017). Research on the evaluation index system of information quality based on user experience: analysis from the perspective of user cognitive and emotional needs. Inform. Theory Pract. 40:5. doi: 10.16353/j.cnki.1000-7490.2017.02.019

Crossref Full Text | Google Scholar

Krathwohl, D. R., Bloom, B. S., and Masia, B. B. (1956). Taxonomy of educational objectives: the classification of educational goals. Affective Domain. 58–60.

Google Scholar

Lei, Y. G., and Feng, M. T. (2020). Research on the design and development of educational games for preschool children from the perspective of learning quality. China Distance Educ. 2020:8.

Google Scholar

Li, G. X. (2011). Research on the influencing factors of consumer behavior in online game products. Master’s thesis. Southwest University of Finance and Economics

Google Scholar

Li, H. S. (2019). Design and Research of Functional Games of Intangible Cultural Heritage, Master Dissertation, Chongqing University. doi: 10.27670/d.cnki.gcqdu.2019.002378

Crossref Full Text | Google Scholar

Li, X. Y. (2023). From inside to outside: A study on the communication and shaping of the cultural IP of the Beijing palace museum. Master’s thesis. Shandong University.

Google Scholar

Li, Y., and Deng, Q. Y. (2019). Analysis of the implementation effect of long-term care insurance system for the elderly based on fuzzy comprehensive evaluation method. Popul. Econ. 2019, 82–96. doi: 10.1007/978-3-319-94574-3

Crossref Full Text | Google Scholar

Li, W. H., and Guo, J. P. (2004). Interval weight of judgment matrix and its consistency test in AHP. J. Syst. Manag. 13, 530–532. doi: 10.3969/j.issn.1005-2542.2004.06.012

Crossref Full Text | Google Scholar

Li, F., and Sun, Y. (2019). Functional games: definition, value exploration, and development suggestions. Educ. Media Res. 2019:4.

Google Scholar

Li, B. M., and Zhu, Z. T. (2017). From “learning to learn” focusing on results to “learning to learn” focusing on the process - research on online Learners’ Online learning ability assessment and development strategies. Open Educ. Res. 23, 92–100. doi: 10.13966/j.cnki.kfjyyj.2017.04.009

Crossref Full Text | Google Scholar

Liang, H. Z. (2019). Research on the comprehensive benefit evaluation of cultural and tourism characteristic towns based on AHP-fuzzy comprehensive evaluation method. Master’s thesis. Guangxi University

Google Scholar

Lina, J., and Haifeng, W. (2020). Cultural value and interaction design of functional games - taking “Nishan Shaman” as an example [J]. J. Tonghua Normal Univ. 41, 1–7. doi: 10.13877/j.cnki.cn22-1284.2020.11.001

Crossref Full Text | Google Scholar

Liu, W. T., Gu, H., and Li, C. H. (2011). Expert evaluation method based on the Delphi method. Comput. Eng. 2011:4.

Google Scholar

Liu, L., Liu, Q., and Yuan, Y. (2023). Learning player learning analysis model for serious games. Software Guide 22, 286–290.

Google Scholar

Liu, W. H., Wang, Y. T., and Zhao, M. (2021). Design and development of evaluation indicators for educational games. 111–120.

Google Scholar

Min, M. Y. (2018). Research on the continued usage intention of Mobile network game users based on marketing perspective. Master’s thesis. Wuhan University of Technology

Google Scholar

Niu, Y. Y. (2020). An empirical study on the continued use intention of WeChat among college students based on the theory of use and satisfaction. Master’s thesis. Zhengzhou University

Google Scholar

Peng, H. P., and Liu, N. N. (2020). Cultivating “play merchants” among minors in the web 3.0 era. Youth Develop. Forum 30:8

Google Scholar

Qin, F. (2017). Research on the digital survival and development of intangible cultural heritage. University of Science and Technology of China, PhD dissertation. Available at: https://kns.cnki.net/kcms2/article/abstract?v=fCqJ37DMrBk-naE2hSj_xUlHtFubu4-XCtPfPhT-xubzlBcmc3Che1K9M6W8KGZHNJ7XYeKci5qpFVFTUUlXffb75mYTYOFYr9Rry099Lj8m0NllNfy3igK8zRrOo-Q5bUquR866Wlu7VlwyWfzKqg==uniplatform=NZKPTlanguage=CHS

Google Scholar

Shen, J. (2015). Comparative study on evaluation index system of educational games in China and the United States. Master’s thesis. Hangzhou Normal University

Google Scholar

Shen, J., and Zhang, S. J. (2014). Review of research on evaluation methods for educational games. J. Dist. Educ. 32:8. doi: 10.3969/j.issn.1672-0008.2014.03.015

Crossref Full Text | Google Scholar

Sun, F. (2023). Research on the influencing factors of intention to use massive open online course (MOOC)platforms. Master’s thesis. Hebei Normal University

Google Scholar

Sun, J. J., and Guo, Y. Z. (2023). Research on the impact of digitalization of cultural heritage on National Heritage Responsibility: model construction based on TTF and TAM. Tourism Sci. 37, 71–85.

Google Scholar

Sun, S. J., and Zhang, Y. H. (2017). Research on user participation motivation in social electronic commerce UGC platform: a case study of Xiaohongshu. Design 2017:2. doi: 10.3969/j.issn.1003-0069.2017.07.005

Crossref Full Text | Google Scholar

Tang, Z. (2022). Use and satisfaction: research on the motivation of Mobile internet entertainment user needs in the era of Mobile internet-taking “peace elite” as an example. News Vanguard. 2022:2.

Google Scholar

Teng, C. E. (2019). Research on document protection of intangible cultural heritage from the perspective of social memory. Doctoral dissertation, Jilin University. Available at: https://kns.cnki.net/kcms2/article/abstract?v=fCqJ37DMrBnJxhInz-u7QSdlzh3qVbHdlVEw5fJxzEmN95O90eJvqjBFnNTNXanji5UOrqT3J_wmc1NuwVw1c-FnxIAGipIM72d4-ytJvQK77Hxg__8RBCkOjUN77ja4k2k9myAKHjl9PanMCkb2cQ==uniplatform=NZKPTlanguage=CHS

Google Scholar

Wang, Y. W. (2010). Research on the factors influencing user participation intention in MMORPG online games. Master’s thesis. University of Electronic Science and Technology

Google Scholar

Wang, L. (2017). Design and implementation of comprehensive ability evaluation system based on fuzzy analytic hierarchy process. China Manag. Inform. 2017, 184–186.

Google Scholar

Wang, R. Q. (2022). On the application of cross-media narrative in game IP development - taking “Arknights” as an example. Writ. Edit. 2022, 16–17.

Google Scholar

Wang, W. J., Zhao, X. C., and Xie, H. X. (2019a). New advances in the evaluation of foreign digital educational games. Compar. Educ. Res. 41, 101–108. doi: 10.20013/j.cnki.ice.2019.03.014

Crossref Full Text | Google Scholar

Wang, Y. M., Zhu, X. D., and Fu, S. S. (2019b). Empirical study on the factors influencing users’ intention to use mobile games based on immersion theory. Electron. Commerce 2019:4.

Google Scholar

Wu, X. Y. (2019). Research on the factors influencing the stickiness of Mobile game users based on TAM and flow theory. Master’s thesis. Beijing Jiaotong University

Google Scholar

Xu, F. (2009). Quality evaluation method of distance education based on AHP-FCE model. Master’s thesis. Jiangnan University

Google Scholar

Xu, H. F. (2019). A study on the application of serious games in material cultural heritage Shenyang Aerospace University.

Google Scholar

Xu, L. Q. (2019). User Experience Evaluation Model and Product Design Based on Cognitive Theory, Master Dissertation, Qilu University of Technology. Available at: https://kns.cnki.net/kcms2/article/abstract?v=fCqJ37DMrBkFBn-GK2LMhLsi8s-mH_V7Tqa6cI9EDPhzDOnBJN43f56xpNTv48ak7D8__LCZqNG1TB2h6YxSRO7JkAjhM8c0M7RY9FS_WJriTc2wer45cRNQk2rK2G7Pck_Q83cDqA5fUAMapaFRCw==uniplatform=NZKPTlanguage=CHS

Google Scholar

Yang, X. (2020). Master Dissertation on Serious Games and Digital Protection of Intangible cultural Heritage, Huazhong University of Science and Technology. doi: 10.27157/d.cnki.ghzku.2020.001487

Crossref Full Text | Google Scholar

Yao, Z. F. (2023). Research on factors influencing the continued use intention of game-based WeChat Mini programs.

Google Scholar

Ye, N., Zuo, B., and Zhang, L. (2009). A study on the relationship between motivation and addiction to online games among college students. Heilongjiang Higher Educ. Res. 2009:3. doi: 10.3969/j.issn.1003-2614.2009.04.031

Crossref Full Text | Google Scholar

Yu, X. Y. (2016). Construction of evaluation criteria for educational games. Curric. Teach. Res. 2016, 30–33.

Google Scholar

Yu, Y. W., and Cheng, R. F. (2012). A comparative study on the cost of acquiring new customers and retaining old customers. Times Econ. Trade 2012:2.

Google Scholar

Zhang, H. (2015). Research on the use value of virtual game products. Master’s thesis. Shanghai Jiao Tong University

Google Scholar

Zhang, D. (2016). Analysis of factors leading to user churn in Mobile games. Master’s thesis. Nanjing University

Google Scholar

Zhang, D. J. (2018). The artistry and Functionality of casual game interface design [J]. Oriental Collection. 269.

Google Scholar

Zhang, W. M. (2019). Research on continuous use intention of MOBA mobile game users, Master Degree Thesis, Northwest University. Available at: https://kns.cnki.net/kcms2/article/abstract?v=fCqJ37DMrBk3qjzSH7e8cb_VEcbbt8FGXxDwD7xVL-G4LUKrV0kYIWv0v0l5qGQu_T1NeQVzY5 T47x9uqtZoG_HhTLaIEtV6wk3ifJHHMgtZuEHJIyGu2WP9iUKXXDvE0wTYAElmv611IYVBpx2JCA==uniplatform=NZKPTlanguage=CHS

Google Scholar

Zhang, S. Y. (2023). Research on the Design of Serious Games for Changsha kiln cultural heritage VR on VIVE focus

Google Scholar

Zhang, X. W. (2023). Research on the design index system of news APP products based on fuzzy analytic hierarchy process. Master’s thesis. Beijing University of Posts and Telecommunications.

Google Scholar

Zhang, K. Y., Mo, Q. Y., and Li, R. J. (2014). Research on the impact of driving safety based on AHP method. Sci. Technol. Innov. Appl.. 2014:2.

Google Scholar

Zhong, Z. J. (2011). Basic characteristics of online game players and socialization behavior in games. Modern Commun. 2011:5.

Google Scholar

Zhou, Y., and Xu, X. (2017). Review of digitalization research on intangible cultural heritage. Library Inform. Serv. 1, 6–15. doi: 10.13266/j.issn.0252-3116.2017.02.001

Crossref Full Text | Google Scholar

Zhu, Y. H., Xu, S. Y., and Wu, J. F. (2020). Research on the evaluation model of ride-hailing service experience based on analytic hierarchy process. Packaging. Engineering 41, 112–117. doi: 10.19554/j.cnki.1001-3563.2020.14.016

Crossref Full Text | Google Scholar