Ayaz Karimov
Alexander Nolte2
Mirka Saarela
94% of researchers rate our articles as excellent or good
Learn more about the work of our research integrity team to safeguard the quality of each article we publish.
Find out more
- 1Faculty of Information Technology, University of Jyväskylä, Jyväskylä, Finland
- 2Software Engineering and Technology Cluster, Eindhoven University of Technology, Eindhoven, Netherlands
- 3Department of Human-centered Computing and Cognitive Science, University of Duisburg-Essen, Duisburg, Germany
The localization of online educational platforms brings many benefits to the students and teachers such as access to different types of textual and video content. Nonetheless, it demands time and capital resources to localize any content. The aim of this research was to re-engineer the entire localization process of Khan Academy content into the Azerbaijani language and evaluate its impact on users’ learning experience. For this purpose, we implemented process re-engineering’s cycle of successive steps. Additionally, we carried out a survey to investigate the new localization process’s effect on users’ learning experience. Our study found that making the localization process more efficient decreased the time and resources needed. Additionally, this improved process positively affected how users experienced learning on the platform.
1 Introduction
Online education is an instructional delivery system that encloses any pattern of learning that happens via the Internet. A plethora of online educational platforms, as well as tools, emerged starting in the early 2010s, and these platforms are used in different situations to facilitate and even offer the entire learning experience. As online educational platforms gain increasing popularity, numerous areas for optimization are also surfacing, highlighting their potential weaknesses. One example of this can be educational content localization. In order to localize educational content, it is necessary to translate and carefully proofread the textual material, subsequently dubbing the translated text and editing the video as needed. Hence, the localization of one subject can sometimes take more than a year to prepare (De la Cova, 2016). In such situations, business process re-engineering can be beneficial to achieve such goals by optimizing the process (Reijers et al., 2010). This study aimed to achieve two primary objectives: firstly, to enhance the efficiency of the current content localization process through the implementation of business process re-engineering, focusing on reducing the number of personnel involved and shortening the overall duration of the localization process. Secondly, the research aimed to enhance users’ learning experience on the platform.
1.1 Business process re-engineering (BPR)
Business process re-engineering concentrates on using business processes as one of the crucial contributors to achieving the organizations’ aims through analyzing lasting performance management, improvement, and governance of crucial processes (Jeston, 2014). Hereby, business processes or workflows are considered as a series of activities that are performed by a stakeholder or a group of stakeholders. Each activity within the process is a task. The main goal of the BPR is to analyze the current situation and define the parts that can be improved within the business processes (Harmon and Trends, 2010). BPR was particularly utilized in the manufacturing industry to optimize the resources used during production. However, the application of BPM expanded from the industrial field to many sectors and education is one of these domains (vom Brocke and Mendling, 2016). Educational stakeholders, such as public and private educational organizations, implement BPR for improving their service or product (Rhee et al., 2010), but its primary implementation is observed in online educational organizations, that is organizations that offer web-based, or else e-learning (Drağan et al., 2014; Hannafin et al., 2009). In educational organizations, BPR was used for various reasons such as providing customized learning paths for students (Adesina and Molloy, 2011), improving the student admission process (Balaji, 2004), and the organization of official trips process in educational institutions (Pasaribu et al., 2021). The localization of content is also one of the processes by which educational organizations utilize BPR.
1.2 The content localization and Khan Academy
Localization can be defined as the translation of any content or platform into a language in a way that crosses cultural boundaries. The main difference between translation and localization is that during the translation process, content is directly translated, however, during the localization process, content is culturally adapted, for example, by using culturally-similar names and proverbs (Pym, 2004). In the 1980s and 1990s, localization became crucial for software and hardware companies when they wanted to internationalize their businesses. Once online learning started to grow rapidly in the 2000s, in parallel with the emergence of educational technology companies, their internationalization strategies were also developed (DeBry Sr, 2002). Moreover, Khan Academy is also one of these companies that concentrated on the localization of their content within their internationalization strategies. Even though the company was founded in 2008, its localization started in 2010 with the Turkish language. Furthermore, Azerbaijani advocates commenced the localization process in 2017, and currently, the team is actively localizing (2000 videos and 4000 exercises each year) the content from the original page of the Khan Academy1 into the Azerbaijani language.
Khan Academy is an online educational institution that provides free world-class education anywhere for anyone and there are more than 60 subjects under the major spheres such as science, computing, and history. The organization has language advocates in more than 50 countries who localize the content into different languages. Khan Academy Azerbaijan is not an official part of the Khan Academy company, and it has a formal agreement with two people who lead the localization process in Azerbaijan. Within this research, we collaborated with these people to re-engineer their localization process.
1.3 Motivation of the research
Khan Academy is one of the biggest online education platforms in the world, however, since most of the contents of Khan Academy are in English, it restricts the learners from using the platform if their English level is not sufficient. One of the goals of Khan Academy is to localize its content into other languages, and Azerbaijani is one of them. The localization of the Khan Academy content is conducted based on volunteer work by EdBall and Inskillz. Thus, carrying out continuous effective content localization is challenging due to the limited number of people and the time that they can dedicate to the project. Moreover, the localization of international education platforms into Azerbaijani is significant since there are limited platforms as such and even most of the current platforms provide paid service. This, again poses a challenge for the students to be able to use the platform’s all features. Since EdBall and Inskillz focus on localizing Khan Academy’s content into Azerbaijani and the platform is entirely free, it can help students and teachers who live in the edge parts of Azerbaijan to learn and educate at a world-class level anywhere without financial expenditures. Karimov et al. (2023) demonstrated in their research that utilizing Khan Academy in under-resourced communities can improve students’ learning outcomes and motivation.
Even though the localization of the Khan Academy content into Azerbaijani can bring a myriad of advantages, a major problem is the project‘s cost. Here, the cost is time and human resources allocated for the project. The localization procedure started in 2017, and the team encountered many bottlenecks.
Localizing the content fast and qualitatively is the main goal of the team. Nevertheless, according to the Language Advocate of Khan Academy in Azerbaijan, the team cannot meet the key performance indicators, such as the number of localized videos, and exercises currently. By conducting this research, we aimed to improve the current localization process and measure its impact on students’ learning experience, and for this, we defined the following research questions to answer:
• Research question 1 (RQ1): What are the main bottlenecks of the current localization process of Khan Academy content into Azerbaijan?
• Research question 2 (RQ2): How can we re-engineer current business processes to improve efficiency in terms of success (the number of localized content and the number of people involved in the localization)?
• Research question 3 (RQ3): What is the impact of the re-engineered localization process on the user’s learning experience?
2 Literature review
2.1 The implementation of BPR in the education field
Business process re-engineering is utilized in education for various purposes, including enhancing administrative tasks and student learning experiences. Adesina and Molloy (2011) applied business process re-engineering principles to develop a personalized learning roadmap, utilizing Business Process Modelling Notation (BPMN) for process modeling. Their research resulted in a prototype system and software framework enabling education specialists to design tailored learning workflows. Balaji (2004) implemented BPR in a New Zealand tertiary education institute to enhance performance in areas like student enrollment and operational management. Following BPR implementation, they observed increased student satisfaction, improved adaptability to enrollment fluctuations, and reduced operating costs. They suggested that their approach could be adopted by other tertiary institutions. Pramartha and Mimba (2020) measured the impact of using BPR to solve the student admissions process issue at Udayana University. They started the re-engineering by defining the problems and analyzing them through interviews. To solve a long queue problem, they developed an IT artifact and this helped them to decrease the queue waiting duration from 60 days to only 29 days. The new re-engineered process also created an opportunity for the university to increase its financial income. Furthermore, BPR has also been implemented in other parts of the education domain rather than student admission, learning path roadmap, and operation cost reduction. For example, Pasaribu et al. (2021) researched how BPR can optimize the organization of official trips process in educational institutions. Within this research, initially, they commenced by identifying processes within official trips at Telkom University, then implemented the results of BPR. After re-engineering the process, they found that the official trips process duration performed more effectively and it saved paper usage.
Pornphol and Tongkeo (2018) researched the components of how BPR can be utilized in the process of shifting from a traditional university to a smart university. While they propose that this shift should be done through a BPR concept, according to Pornphol and Tongkeo (2018), the usage of this concept should be controlled by the university council. Igwe et al. (2021) examined the improvement of higher education in Nigeria by using the BPR. They found missing elements to develop a framework for re-engineering and re-valuing higher education values. Based on this, they developed a re-engineering framework that includes high-quality structures and curriculum design; effective leadership and governance; and high-quality teaching and learning. Moreover, educational institutions that translate or localize their content into different languages may encounter bottlenecks and by using the BPR, they can improve their processes (Smirnov et al., 2019).
2.2 The localization of educational content
Recent studies have emphasized the critical role of effective content localization in improving student engagement and learning outcomes across online educational platforms. Al-Hunaiyyan et al. (2021) conducted extensive research in Kuwait examining mobile learning platforms, finding that properly localized content significantly increased student participation and comprehension rates. Their study highlighted that localization goes beyond simple translation, requiring careful consideration of cultural context, learning preferences, and local educational norms.
Similarly, Guo et al. (2020) investigated Chinese faculty members’ usage of Open Educational Resources (OER), revealing crucial insights into localization challenges in Asian educational contexts. Their findings demonstrated that successful content localization required understanding local technological infrastructure, teaching methodologies, and cultural learning preferences. The study particularly emphasized how mathematical and scientific content required special attention during localization to maintain technical accuracy while ensuring cultural relevance.
Building on these findings, Xie and Rice (2021) examined the role of instructional designers during emergency remote teaching, providing valuable insights into rapid content localization processes. Their research revealed that effective localization strategies must balance maintaining content integrity with cultural adaptation. They found that organizations implementing structured localization processes showed significantly higher student engagement rates and better learning outcomes compared to those using ad hoc translation approaches.
Previous research has focused on content localization from different perspectives. Ghuman et al. (2017) concentrated on the importance of localization in e-learning and they also mentioned what kinds of basic steps can be taken to start the localization process. They mentioned that by localizing international content, online learners get the opportunity to access well-designed courses in their native language. According to Ghuman et al. (2017), particularly online learners from rural regions will have access to this online educational content and it would help them to improve their skills. Ghuman et al. (2017) also highlighted that by using various localization standards such as INSCRIPT Keyboard Layout, Common Language Data Repository, or XML Localization interchange Format, it is possible to commence the basic localization process. Kuprina et al. (2020) also researched educational content localization but their primary focus was the methods of localization and translation of the educational program of “Software Engineering.” They mentioned that some platforms such as SeoTXT.com, Istio.com, Advego.com, or CAT can be utilized within the successful localization process.
Kuprina et al. (2020) concluded their research by highlighting that the solution to problems that emerge during the localization process should be carried out at an interdisciplinary level, uniting both linguists- translators and specialists in that field. Baharum et al. (2017) researched whether utilizing the localization guideline can be more efficient or not. For this, they used the e-learning website of Universiti Malaysia Sabah (Smart2ums). After re-designing this platform by using the localization guideline Aslina et al. (2016), Baharum et al. (2017) evaluated user interaction. They concluded their research by mentioning that the e-learning website could provide better user interaction if the localization guideline was used.
Chowdhury et al. (2011) analyzed the localization of digital content to use in secondary schools, and within this framework, various data visualizations were prepared based on the localized content. They analyzed the impact according to the region, localization pattern, and type of content. While the previous research was mainly focused on the localization tool or guideline analysis or implementation, in this research, our main focus was to re-engineer the content localization process of the online educational platform.
2.3 The learning experience improvement
The learning experience is the overall process an individual engages in learning activities and acquires knowledge and skills. It encompasses various factors, including learner engagement, instructional methods, and personalization of learning. Within the online educational platform context, the learning experience involves factors such as multimedia integration, user interface design, instructional strategies, learner engagement, and content organization (Pratiwi and Sudirtha, 2022; Gupta and Maji, 2020; Yoon, 2003).
Allcoat and von Mu¨hlenen (2018), Ying et al. (2017), and Parisi (2015) researched the learning experience from a virtual reality (VR) perspective. Ying et al. (2017) proposed an education platform, VREX, to improve the teaching experience curriculum building. Within this research, they tried to transfer digital instructional documents into VR scenes. Ying et al. (2017) found that while VREX can be utilized in various ways in the future, it also provides a distributed mode for students to experience an interactive learning process at anytime, anywhere. Allcoat and von Mu¨hlenen (2018) designed research where students were separated into three groups based on their learning conditions: traditional (textbook style), video (a passive control), and VR. All participants were asked to take the test before and after the research to measure the change in their learning. Allcoat and von Mu¨hlenen (2018) found that the participants in the VR condition demonstrated higher engagement and better performance for “remembering” than those in the traditional and video conditions. Furthermore, Georgiou et al. (2021) and Rivera-Vargas et al. (2021) researched designing learning experiences with immersive virtual reality and students’ learning experiences in online education respectively. Georgiou et al. (2021) evaluated that learning experience design in terms of the student’s learning gains is connected to their perceptions of the immersive VR simulation and the learning process. Rivera-Vargas et al. (2021) found that students positively accept the integration and adoption of technical skills but are concerned about pedagogical and institutional support.
Some research concentrated on the learning experience of online educational platforms. Miya and Govender (2022) conducted a literature review of 25 papers to determine the influence of user experience (UX) and user interface (UI) on eLearning. They concluded their systematic review by mentioning the importance and crucial effect of UX/UI on online educational platforms. They also highlighted that while the UX improves the usability of the system, the UI enhances the learning process. Mayer et al. (2020) analyzed the learning level of students from an instructional video when the onscreen instructor draws graphics on the board while lecturing considering the case of Khan Academy. Mayer et al. (2020) concluded the research that students do not learn better from a multimedia lesson when an interesting but extraneous video is added.
In previous research, scholars have explored the learning experience from diverse perspectives, including VR, and the use of platforms like Khan Academy. However, our study takes a unique approach by focusing on measuring the impact of re-engineering the content localization process on users’ learning experience specifically within the context of Khan Academy. By analyzing the changes resulting from this intervention, we aim to gain insights into how modifications to the content localization process can enhance the learning experience of users on the platform. This research contributes to the existing body of knowledge by shedding light on the importance of optimizing specific aspects of online educational platforms to improve the overall learning experience for users.
3 Methodology
In this research, our aim was to re-engineer the localization process of an online educational platform and measure its impact on the learning experience of users. To re-engineer the process, we implemented Business Process Management (BPM) lifecycle steps (Van Der Aalst et al., 2012) and to measure the impact of optimization on the learning experience, we collected users’ responses about the platform before and after the re-engineering process through surveys.
3.1 BPM lifecycle and research process
Using the BPM lifecycle as a methodology to re-engineer a process can help to ensure that changes are made in a systematic and structured way. This can help to reduce the risk of errors and ensure that the process is optimized for maximum efficiency. Additionally, the BPM lifecycle provides a framework for continuous improvement, which can help to ensure that the process remains optimized over time (Dumas et al., 2018). The BPM lifecycle consists of six major steps (process identification, process discovery, process analysis, process redesign, process implementation, process monitoring, and controlling), and within this empirical research, the first five steps were gone through (Table 1). The sixth step, process monitoring and controlling, was not included due to resource constraints or the scope of the study (Figures 1, 2). Because this requires the allocation of more time to effectively implement and monitor the control processes.
TABLE 1
Table 1. Data collection and analysis methods implemented during re-engineering process.
FIGURE 1
Figure 1. The BPM lifecycle.
FIGURE 2
Figure 2. Process redesign implementation workflow.
Process identification and discovery: The initial step involved conducting Phase 1 interviews with stakeholders to gather insights about their roles, the tasks they perform, and the time required for each task. These interviews were semi-structured and provided qualitative data to map out the existing localization process. The output of this phase was the creation of the “as-is2” process model, capturing the sequence of activities in the current localization workflow.
• Process analysis: To identify bottlenecks, we employed both qualitative and quantitative analysis techniques: Issue Register Analysis: This qualitative method was used to categorize recurring problems, capturing the specific nature and impact of each issue reported by stakeholders. For instance, delays in proofreading and inconsistencies in translation quality were highlighted.
• Flow analysis: We applied cycle time calculation to quantitatively measure the duration of each subprocess within the localization workflow. This allowed us to identify which steps contributed most to inefficiencies, such as the extended time taken in the translation phase.
Process redesign: Phase 2 interviews were conducted to collect feedback from stakeholders on potential solutions to the identified issues. The redesign focused on streamlining workflows, merging roles where possible (e.g., combining Crowdin and Amara proofreading tasks), and integrating improved coordination mechanisms. The proposed changes were documented in a “changes document” outlining the rationale behind each adjustment.
Process implementation: We trained the stakeholders on the re-engineered process, allowing them a trial week to familiarize themselves with the new workflow. Subsequently, stakeholders localized a pilot content piece using the updated process to ensure all improvements were practical and beneficial.
3.2 Data collection
We collected data both from stakeholders and platform users for different purposes. To re-engineer the process, we held 2 interviews (Phase 1 and Phase 2 interviews) and a workshop. In the Phase 1 interviews, we aimed to get stakeholders’ feedback about the current process, learn the average time that they spend on doing their tasks individually, and the main bottlenecks that they observe. In the Phase 2 interviews, we wanted to get their feedback about solving the current bottlenecks. Both Phase 1 and Phase 2 interviews were in a semi-structured format and the interviews continued for around 35–45 min on average. The participants of Phase 1 and Phase 2 interviews were the same eleven stakeholders who actively take part in daily localization activities. Moreover, the interviews were conducted in Zoom in the Azerbaijani language.
In addition to the interviews, we also held a workshop where we asked the stakeholders’ opinions about localizing content based on the re-engineered localization process. The workshop continued for 50 min, and it was in Azerbaijani.
Before re-engineering the localization process and after doing so, we shared the survey with Khan Academy Azerbaijan platform users to measure the impact of the re-engineered method on users’ learning experience. Our purpose was to understand how much the re-engineered localization process impacted users’ learning experience. Hereby, as a learning experience, we considered any interaction that contributes to students’ learning on the Khan Academy platform. The survey was shared through different channels such as the Khan Academy Azerbaijan Facebook community, the current email list of Khan Academy Azerbaijan users, and the social media channels of the Khan Academy Azerbaijan. Moreover, 127 users filled out the first survey (before re-engineering the localization process), and 107 users filled out the second one (after re-engineering the localization process).
3.3 Data analysis
To ensure robust results, we implemented both qualitative and quantitative analysis techniques. The coding scheme methodology was used for interview and workshop data to identify recurring patterns and themes. This method was selected due to its effectiveness in categorizing qualitative data and highlighting common issues, which were crucial for understanding stakeholder perspectives and identifying process bottlenecks.
For quantitative analysis, we employed descriptive statistics and comparative metrics to analyze the cycle time and survey results (Saarela, 2024). The decision to use these methods was driven by the need to measure both time reductions and improvements in user feedback following the re-engineered process. Descriptive statistics provided a straightforward approach to understanding participant demographics and response distributions, while comparative analyses allowed for a direct comparison of pre- and post-implementation data.
3.3.1 Interview and workshop data analysis
To analyze the interviews and workshop, we implemented the coding scheme methodology (Barendregt and Bekker, 2006). Because coding scheme methodology identifies recurring patterns and connections within the data. By categorizing similar data segments under specific codes, we were able to determine themes or emerging concepts. In the coding scheme3,4, we categorized the responses into three categories:
• Platforms: we defined this category to categorize the bottlenecks based on which platforms they occur
• Subprocess: we defined this category to categorize the bottlenecks based on which subprocesses they occur
• Subprocess edit: We created this category during the subprocess editing phase. After developing the initial “as is” process, we consulted with stakeholders to determine if the proposed localization process aligned with their current practices. Stakeholders provided suggestions for modifications in specific subprocesses where they carried out tasks differently. To organize these suggestions, we categorized them according to the corresponding subprocesses in our coding scheme.
In the coding scheme, the coding rule was decided in a way that if the stakeholder mentioned the same or similar comments about the given example, then we gave “1” for this stakeholder, and if the stakeholder never mentioned the same or similar comment, then we gave “0” for this stakeholder for that example.
For example, in Table 2, we give the example in which the Crowdin translator mentioned the bottleneck that occurs in the Crowdin platform. Thus, we put his example in the “Crowdin” category by adding his example in the “Examples” column. We mentioned his position and the ID of the interview. Then, while analyzing the project manager’s interview transcription, we observed a similar comment about the same example. Thus, we gave “1” to the project manager and “1” to the Crowdin translator since he also mentioned the same example in his interview. However, the dubber never mentioned the same or similar comment about that example. Therefore, we gave “0” for the dubber. Finally, the last column depicts the total number of codes which is “2” in this example.
TABLE 2
Table 2. The frame that was used within the coding scheme.
3.3.2 Process analysis
To analyze the process, we selected one qualitative and one quantitative process analysis method because we wanted to review the process from both of these perspectives to have complementary insights and define as many bottlenecks as possible. From the perspective of qualitative process analysis, we selected the “issue register” analysis. This is a type of qualitative process analysis to categorize identified issues in the process as part of “as is” process modeling (Dumas et al., 2013). The main reason we selected the issue register analysis for the qualitative process analysis was its ability to provide detailed information about each issue and its quantitative and qualitative impact.
From the perspective of quantitative analysis, we conducted the flow analysis. Flow analysis has different techniques such as flow analysis for cost, critical path method, cycle time efficiency, and calculating cycle time. We implemented the cycle time calculation because time is the main success factor of the stakeholders at Khan Academy Azerbaijan and they wanted to reduce the cycle time of the localization process.
3.3.3 Process redesigning
To redesign, we used the tool called “Signavio” and the technique was a flow chart. The reason why we selected the flow chart technique was that it lets us make any changes if needed in any parts of the process and we wanted this flexibility in this redesigning process. After defining the changes, the stakeholders were invited to participate in the localization of the new content. Moreover, to compare the impact of the re-engineered process, “Algebraic expressions” topics from the Math content on the Khan Academy platform were selected. The primary motivation for selecting the Mathematics content within this research was:
• Mathematics was the only content that the Khan Academy Azerbaijan team fully localized one of its subtopics (“Negative numbers”) into the Azerbaijani language.
• It was the most required and used topic for students in Azerbaijan, according to the Khan Academy Azerbaijani team.
For the comparison, we chose “Algebraic expressions” from “Algebra Basics” as a chapter to localize to compare the result of localization process re-engineering. The reason why we chose those subjects (“Negative numbers” and “Algebraic expressions”) to compare is that they include roughly the same number of words in exercises, articles, descriptions, and videos. They also possess similar difficulty levels.
Figure 3 depicts the difference between the number of words and videos in “Negative numbers” and “Algebraic expressions” contents.
FIGURE 3
Figure 3. The comparison of words, videos in “negative numbers” and “algebraic expressions.”
3.3.4 Participants
The participants’ profiles in the survey before the re-engineering process indicated an average age of 19, with a majority of respondents (73.8%) identifying themselves as women. Additionally, a significant portion (77.4%) of the participants were students. The main responses came from Baku (71 responses) and Ganja (seven responses), showcasing the geographical distribution of the participants. It was observed that around 92.8% of the respondents spent 60 min or less learning any topic on the platform on average.
In the survey conducted after the re-engineering process, the participants’ profiles revealed an average age of 21. Among the respondents, 62.2% identified themselves as women and 68.5% were students. Similar to the previous survey, the main responses were from Baku (66 responses) and Ganja (eight responses). The data indicated that approximately 95.4% of the respondents spent 60 min or less learning any topic on the platform on average.
4 Results
4.1 Current localization process and its bottlenecks
The current localization process5 of Khan Academy into Azerbaijani commences with selecting the content and it finishes when the content is localized totally. The entire localization the process consists of six major subprocesses, which are as follows:
1. Assign content tasks
2. Translate Crowdin content
3. Translate Amara content
4. Recreate videos
5. Control quality
6. Deliver video
Based on the issue register6, from the perspective of qualitative impact, we found that these issues cause demotivating translators, unsatisfied users, extra work for the quality control team, the bad learning experience of learners, and churned users. While calculating the time lost for each issue, we found out that the team wasted 14137.65 min translating Crowdin first, then Amara content. One other main finding was that the damage of Crowdin’s proposed translations on the quality of translation caused the loss of forty students each month and a waste of 30 min.
Having done the cycle time calculation technique of the flow analysis, we found the general time spent on the localization process and each subprocess cycle time duration. The general localization process took 48516.45 min which equals 33.7 days. Moreover, the cycle time for the “Assign content tasks” was 438 min, and for the “Translate the content” subprocess, it took 14137.65 min. Subsequently, it follows with 20199.8 min for the “Translate Amara content,” 4380.6 min for the “Recreate videos,” 5783.8 min for the “Control quality,” and lastly, 3576.6 min for the “Deliver video” subprocess.
4.2 Differences between current and re-engineered process7
The current and re-engineered localization processes were compared based on the two success metrics: the number of people involved in the localization of the content and the duration spent on the localization of the content. Because these were the main success metrics for the Khan Academy Azerbaijani team.
From the human capital perspective, there was a reduction, because previously there were two proofreaders: Amara proofreader and Crowdin proofreader. Nonetheless, in the re-engineered localization process, those two proofreader titles were removed, and there is only one position regarding the proofreading, which is “proofreader.”
Subsequently, we compared the total duration spent on the localization of the contents. Figure 4 depicts the comparison of the total time spent on the localization of each subprocess. Since there were no changes or improvements in the “Assign content tasks” subprocess, the cycle time duration remained stable. These differences between the “as is” and the re-engineered process were 11597.64, 1465.8, 749.6, 798.8, and 167.4 min for the “Assign content tasks,” “Translate Crowdin content,” “Translate Amara content,”
FIGURE 4
Figure 4. The cycle time of the “as is” and the re-engineered processes.
“Recreate videos,” “Control quality,” “Deliver video” subprocesses respectively. The cycle time of the current localization process was 48516.45 min, and it signifies that there is a 17319.25-min difference between the current and re-engineered localization process in general, which is roughly equal to 12 days.
4.3 The impact of the re-engineered process on the learning experience of users
We asked the same questions before and after re-engineering the localization process to observe the changes in the learning experience of users. In both surveys, the users selected “1” if they gave “Definitely not” feedback to the question, and the opposite was to provide “5,” which means “Absolutely yes” to the questions. Firstly, we asked to learn how often the users encounter translation errors on the platform, and we found that the score decreased from 2.5 to 1.9 (Figure 5). It signifies that the content localized according to the re-engineered process positively impacted the number of translation errors. Subsequently, there was a positive change in the adaptation of the learning materials to local culture from 3.6 to 4.1 (around 87.8%), and the translations of non-textual (i.e., pictures) materials from 3.7 to 4.5 (82.2%). There was no change from the perspective of the irrelevance of learning materials for Azerbaijan, it remained at 2.1.
FIGURE 5
Figure 5. Comparison of pre- and post-re-engineering localization process feedback.
Furthermore, in one of the questions, we asked the users whether they wanted to highlight anything else, and the respondents mentioned that there is a need to increase the number of localized content for higher or secondary high school students.
5 Discussion
While the results of this study indicate positive outcomes from the re-engineered localization process, it is important to interpret these findings with caution. The observed improvements in user experience, as reflected in survey responses, suggest a correlation between the enhanced process and better learning experiences. However, it is not possible to definitively establish causation based solely on this study’s data (Saarela, 2024). Other variables, such as user familiarity with the platform, changes in external educational resources, or individual learning preferences, could also contribute to the observed outcomes.
Additionally, the timing of the data collection could coincide with broader educational trends or seasonal variations in user engagement that might have influenced the results. To provide a more balanced interpretation, future research could include control groups or longitudinal studies to assess the consistency of the impact over time and isolate the effects of the re-engineered process more conclusively.
By acknowledging these potential confounding factors, we present a more nuanced interpretation of the findings that strengthens the credibility of our conclusions and invites further inquiry into related influences on user learning outcomes.
In RQ1, we focused on the design of the current localization process of the Khan Academy Azerbaijan team. At some points, the stakeholders mentioned opposite comments about the performance of the tasks.
However, at last, all of them agreed with the last process design. All the information collected from the interviews helped us to detect the bottlenecks. The current process contained similar subprocesses of the research by Vergidis et al. (2007). In that research, there are old and new models, and the oldest one follows the same processes as it was in the current localization process of Khan Academy into the Azerbaijani language. In addition to these similarities, in our research, the main differences in subprocesses emerged because of the specific activities such as localization through Crowdin and proofreading on Amara.
To find the bottlenecks, holding two analyses (issue register and flow) helped us to define which processes we needed to focus on for process re-engineering. We also considered the value-added, waste, and root cause analysis. However, since these analyses do not provide information about the impact of the issue in detail, we did not select them. Both simulation and queuing analysis were also considered. We did not choose the simulation though it is one of the most popular ones because of its design characteristics for the more extensive processes and organizations (Dumas et al., 2013). However, the localization of Khan Academy content into the Azerbaijani language consists of shorter processes compared to other corporations. Secondly, queuing analysis was not selected either because within the localization process, the stakeholders do not encounter certain queuing, and basically, they start working on the content whenever they finish the previous tasks. Last but not least, there was also the reason that we needed to hold the analysis for a shorter period so that we could have enough time to re-engineer and analyze the re-engineered process. Compared to queuing and simulation analysis, the flow analysis is more straightforward. Therefore, it can be conducted within a short period of time. It also gives information about cycle time which is the main factor we were looking for in this analysis.
In RQ2, we concentrated on the re-engineering of the process and the direct implementation of this re-engineered process. We re-engineered the process based on the process analysis and stakeholders’ proposals, and all stakeholders agreed with the updated localization process. After this, stakeholders localized another content, and before starting the localization process, we already expected to decrease the duration by 2 days. With the integration of the content specialists and quality controllers, the re-engineered process followed a similar update pathway as it was in the research carried out by Vergidis et al. (2007). Furthermore, there were different points between our research and the research by Vergidis et al. (2007). For instance, in that research, it was expected that content creators could move back and ask translators to retranslate the content. Nevertheless, in the localization process that we focused on in this research, all the subprocesses happen to be like sequences. It signifies that if the translator makes a mistake, it will be corrected by either proofreader or quality controller. We also analyzed the interviews held, particularly for getting the stakeholders’ proposals about process re-engineering. We could not implement the six proposals, and the main reasons for this were time, finances, and human resources limitations.
However, it would be quite possible to update some subprocesses if there were no such constraints. As mentioned by Melaõ and Pidd (2000), holding more than three process analyses may be time-consuming.
However, being able to carry out both quantitative and qualitative process analyses could explain different bottlenecks as was the case in our research.
Last but not least, in RQ3, we measured the impact of the re-engineered localization process from the learners’ perspectives. In these surveys, we observed positive changes in the responses of people. The survey also showed some points that we were not searching for in the beginning. For example, it depicted how the source of problems changed after this re-engineered process. Moreover, it also gave insights into the team’s future roadmap, such as the importance of focusing on the Mathematics, and Computer Science teams. While analyzing the survey results, we found that the users want to see more Mathematics content on the platform. It helped us to ensure our decision that the localization process re-engineering of the Mathematics content was a successful selection.
Our findings align with recent literature on educational content localization. Similar to Al-Hunaiyyan et al. (2021), we observed improved user engagement after implementing culturally-adapted content. Our results on translation quality impact support Guo et al.’s (2020) findings on Chinese localization efforts.
Additionally, our findings on process optimization align with Sun et al.’s (2022) work on efficient localization workflows, while reinforcing Xie and Rice’s (2021) emphasis on maintaining content accuracy during localization.
The research by Cheng (2005) and Bentley et al. (2005) emphasized the potential positive impact of the localized international content in the local language. Therefore, it is essential to ensure that the localization process re-engineering also brings advancements in the quality of localized content. Hereby, in this research, we reached this goal based on the responses of the platform users.
5.1 Limitations
We defined two main points that threaten the research’s validity: selecting subjects, and data collection both in the interviews and surveys.
In this research, we concentrated on the localization process of the Khan Academy Mathematics subject into the Azerbaijani language. The implementation of this re-engineered localization process for another platform or the subject can be subject to bias. Hereby, it is worth mentioning that the research findings are not generalizable beyond the context of this study. In some cases, the localization re-engineering conducted in the research can be applicable, or quite similar steps can be followed if they possess the same conditions (localizing Mathematics subject, translating into Latin script).
In the interviews and workshops, the collected responses can be subject to bias because, in this kind of research, it is anticipated that the interviewees may answer questions that could damage the quality unintentionally. Because stakeholders also responded based on their perspectives which are inevitably subjective and can be subject to bias. Additionally, workshops also induce bias because people react differently when they are in a group. To mitigate the bias in the workshops, we conducted the interviews along with the workshops to collect the feedback of respondents individually or in small groups. We also organized workshops and interviews in a semi-structured format to adjust the flow based on the responses.
Moreover, we also held one interview per one (maximum three) stakeholder. It helped us to understand the answers of each stakeholder in detail.
One notable limitation is the sample size of survey respondents. Although the surveys provided valuable insights, a larger and more diverse sample could improve the generalizability of the findings. Additionally, there is a risk of selection bias, as participants were primarily sourced from online communities and may not fully represent the broader population of Khan Academy users in Azerbaijan.
Cultural factors play a crucial role in shaping user experiences with localized content. The adaptation of educational materials to reflect local norms, idioms, and learning preferences can significantly affect user engagement and comprehension. While this study found positive user feedback post-localization, future research should delve deeper into how specific cultural nuances impact learning outcomes. Such an exploration would provide a richer understanding of how localization can be tailored to maximize its effectiveness across diverse cultural settings.
6 Conclusion
The localization of educational content creates opportunities for students who encounter challenges accessing world-class education for free. However, the localization process generally requires almost the same amount of time spent on creating the content. Because, while localizing the educational content many local factors are also taken into account which require additional effort. Within this framework, the importance of process re-engineering should be highlighted. Nowadays, educational organizations also commenced utilizing process re-engineering to accelerate their activities by keeping or improving the quality. Furthermore, educational content localization is one of the activities that can be improved by implementing the process of re-engineering to increase learners’ learning experience.
In this academic research, we focused on localization process re-engineering and its impact on the users’ learning experience. For this, we selected the localization process of Khan Academy into the Azerbaijani language. Having designed the current localization process, we focused on analyzing the bottlenecks both from qualitative (issue register analysis) and quantitative (flow analysis) perspectives. As a result, we found that the main bottlenecks were solving the daily problems of translators, checking the quality of the video, and misusing the localization tools. In the next step, we collected the potential changes to be made in the re-engineered process. Based on these changes, we redesigned the localization process, and stakeholders started localizing the selected content based on the redesigned localization process. When we compared the current and re-engineered processes, we found that by using the re-engineered localization process, we reduced the number of stakeholders (from thirteen to twelve), and the localization duration decreased by approximately 12 days. Last but not least, by organizing two surveys, we checked the difference in the learning experience of users. Here, we observed the positive changes in the users’ responses about the quality of the localized non-textual (i.e., pictures) materials and the adaptation of the learning materials to local culture.
From the perspective of future work, we defined that this research can be extended in two aspects:
(1) The localization process re-engineering research can be conducted in other subjects such as Chemistry, and Computer Science. In this research, we re-engineered the localization process for Mathematics, however, different subjects require various processes. For instance, in the localization of Computer Science content, it is essential to consider the localization of codes, but this is not even a topic to be considered in Mathematics. Thus, one of the future works can be done only focusing on Computer Science content localization.
(2) Additionally, this research can be elaborated by focusing on the localization in other languages such as Chinese, and Arabic. There are different grammar and lexical rules in each language, and it may change the localization process. For example, in some languages, such as Chinese, localization, and implementation in this language are more complicated than in other languages (Rao et al., 2017). Thus, as future work, the localization process of the same subject in other languages can be conducted.
Data availability statement
The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found in this article/supplementary material.
Ethics statement
Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The participants provided written informed consent to participate in this study.
Author contributions
AK: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Visualization, Writing – original draft, Writing – review and editing. AN: Conceptualization, Methodology, Supervision, Writing – review and editing. I-AC: Conceptualization, Methodology, Supervision, Writing – original draft, Writing – review and editing. MS: Writing – review and editing. TK: Writing – review and editing.
Funding
The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This work was supported by the Academy of Finland (grant no. 356314).
Acknowledgments
We would like to express our sincere gratitude to the team that localized Khan Academy into Azerbaijani and to Leyla Gasimova who is the Khan Academy Advocate for the Azerbaijani language for actively participating in data collection.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Footnotes
- ^ www.khanacademy.org
- ^ “As is” process refers to the current state of a business process as it currently exists and is being executed in an organization (Aguilar-Save’n, 2004).
- ^ “To be” business process defines the future state of a business process in an organization (Aguilar-Save’n, 2004).
- ^ The coding scheme can be accessed here: https://zenodo.org/record/4753755
- ^ https://zenodo.org/record/4761968
- ^ https://zenodo.org/record/4736540
- ^ https://zenodo.org/record/4762179
References
Adesina, A., and Molloy, D. (2011). “A business process management based virtual learning environment-customised learning paths,” in Proceedings of the 3rd International Conference on Computer Supported Education CSEDU, (Setúbal: Science and Technology Publications), 365–368.
Aguilar-Save’n, R. S. (2004). Business process modelling: review and framework. Int. J. Prod. Econ. 90, 129–149.
Al-Hunaiyyan, A., Alhajri, R. A., and Al-Sharhan, S. (2021). Perceptions and challenges of mobile learning in Kuwait. J. King Saud Univ. Comput. Inform. Sci. 33, 441–448.
Allcoat, D., and von Mu¨hlenen, A. (2018). Learning in virtual reality: effects on performance, emotion and engagement. Res. Learn. Technol. 26:2140.
Aslina, B., Grace, J., Nurul-Hidayah, M., Nordaliela, M., and Teo, J. (2016). “Localization for e-learning of comprehensive universities in malaysia,” in Proceedings of the International Conference on Sustainable Engineering and Technology (INCONSET), Kuala Lumpur.
Baharum, A., Ismail, R., Fabeil, N. F., Ab Fatah, N. S., Tanalol, S. H., Zain, N. H. M., et al. (2017). “Evaluating the localization for e-learning website: case study in universiti Malaysia Sabah,” in Proceedings of the 2017 International Conference on Platform Technology and Service (PlatCon), (Busan), 1–6. doi: 10.1109/PlatCon.2017.7883707
Balaji, M. (2004). Reengineering an Educational Institute: A Case Study in New Zealand. School of International Studies AIS St Helens, New Zealand, Working Paper No.2. Auckland: Centre for Research in International Education.
Barendregt, W., and Bekker, M. M. (2006). Developing a coding scheme for detecting usability and fun problems in computer games for young children. Behav. Res. Methods 38, 382–389.
Bentley, J. P., Tinney, M. V., and Chia, B. H. (2005). Intercultural internet-based learning: know your audience and what it values. Educ. Technol. Res. Dev. 53, 117–127.
Cheng, Y. C. (2005). New Paradigm for Re-Engineering Education: Globalization, Localization and Individualization, Vol. 6. Berlin: Springer.
Chowdhury, M., Al-Mahmood, A., Bashar, M., Ahmed, J. U., et al. (2011). Localization of Digital Content for use in Secondary Schools of Bangladesh. Online Submission. Dhaka.
De la Cova, E. (2016). Translation challenges in the localization of web applications. Sendebar 27, 235–266.
DeBry Sr, D. P. (2002). Analysis of emerging practices in globalizing instructional materials. Educ. Technol. Res. Dev. 50, 73–82.
Drağan, M., Ivana, D., and Arba, R. (2014). Business process modeling in higher education institutions. developing a framework for total quality management at institutional level. Procedia Econ. Finance 16, 95–103.
Dumas, M., La Rosa, M., Mendling, J., and Reijers, H. A. (2013). Fundamentals of Business Process Management. Berlin: Springer.
Dumas, M., La Rosa, M., Mendling, J., Reijers, H. A., Dumas, M., La Rosa, M., et al. (2018). “Introduction to business process management,” in Fundamentals of Business Process Management, eds M. Dumas, M. La Rosa, J. Mendling, and H. A. Reijers (Berlin: Springer), 1–33.
Georgiou, Y., Tsivitanidou, O., and Ioannou, A. (2021). Learning experience design with immersive virtual reality in physics education. Educ. Technol. Res. Dev. 69, 3051–3080.
Ghuman, A., Mahajan, J., Bhatia, S., Singh, J., and Kulkarni, M. D. (2017). “Empowering e-learning with localization,” in Proceedings of the 2017 5th National Conference on E-Learning E-Learning Technologies (ELELTECH), (Hyderabad), 1–6. doi: 10.1109/ELELTECH.2017.8074999
Guo, Y., Zhang, M., Bonk, C. J., and Li, Y. (2020). Chinese faculty members’ Open Educational Resources (OER) usage status and the barriers to OER development and usage. Int. J. Emerg. Technol. Learn. 15, 126–140.
Hannafin, M., Hannafin, K., and Gabbitas, B. (2009). Re-examining cognition during student-centered, web-based learning. Educ. Technol. Res. Dev. 57, 767–785.
Harmon, P., and Trends, B. P. (2010). Business Process Change: A Guide for Business Managers and BPM and Six Sigma Professionals. Amsterdam: Elsevier.
Igwe, P. A., Hack-Polay, D., Mendy, J., Fuller, T., and Lock, D. (2021). Improving higher education standards through reengineering in West African universities–a case study of Nigeria. Stud. High. Educ. 46, 1635–1648.
Jeston, J. (2014). Business Process Management: Practical Guidelines to Successful Implementations. Milton Park: Routledge.
Karimov, A., Saarela, M., and Ka¨rkka¨inen, T. (2023). “The impact of online educational platform on students’ motivation and grades: the case of khan academy in the under-resourced communities,” in International Conference on Educational Data Mining, eds M. Feng, T. Käser, and P. Talukdar (Montreal, CA: International Educational Data Mining Society).
Kuprina, T., Maltseva, A., and Petrikova, A. (2020). The method of educational programs localization under internationalization of academic environment. J. Lang. Cult. Educ. 8, 146–156.
Mayer, R. E., Fiorella, L., and Stull, A. (2020). Five ways to increase the effectiveness of instructional video. Educ. Technol. Res. Dev. 68, 837–852.
Melaõ, N., and Pidd, M. (2000). A conceptual framework for understanding business processes and business process modelling. Inform. Syst. J. 10, 105–129.
Miya, T. K., and Govender, I. (2022). UX/Ui design of online learning platforms and their impact on learning: a review. Int. J. Res. Bus. Soc. Sci. (2147-4478) 11, 316–327.
Parisi, T. (2015). Learning Virtual Reality: Developing Immersive Experiences and Applications for Desktop, Web, and Mobile. Sebastopol, CA: O’Reilly Media, Inc.
Pasaribu, R. D., Anggadwita, G., Hendayani, R., Kotjoprayudi, R. B., and Apiani, D. I. N. (2021). Implementation of business process reengineering (BPR): case study of official trip procedures in higher education institutions. J. Industr. Eng. Manag. 14, 622–644.
Pornphol, P., and Tongkeo, T. (2018). “Transformation from a traditional university into a smart university,” in Proceedings of the 6th International Conference on Information and Education Technology, (Osaka), 144–148.
Pramartha, C. R. A., and Mimba, N. P. S. H. (2020). Udayana university international student management: a business process reengineering approach. Comput. Math. Eng. Applic. 11, 57–64.
Pratiwi, P. Y., and Sudirtha, I. G. (2022). Identification of learning experience in online learning with user persona techniques based on learner-centered design concepts. SISTEMASI 11, 414–428.
Pym, A. (2004). The Moving Tex: Localization, Translation, and Distribution, 1st edn. Amsterdam: John Benjamins Publishing Company. 1–244.
Rao, A., Hilton, J. III, and Harper, S. (2017). Khan academy videos in Chinese: a case study in OER revision. Int. Rev. Res. Open Distrib. Learn. 18. doi: 10.19173/irrodl.v18i5.3086
Reijers, H. A., van Wijk, S., Mutschler, B., and Leurs, M. (2010). “Bpm in practice: who is doing what?,” in Proceedings of the Business Process Management: 8th International Conference, BPM 2010, Hoboken, NJ, USA, September 13-16, 2010, Vol. 8, (Hoboken, NJ: Springer), 45–60.
Rhee, S.-H., Cho, N. W., and Bae, H. (2010). Increasing the efficiency of business processes using a theory of constraints. Inform. Syst. Front. 12, 443–455.
Rivera-Vargas, P., Anderson, T., and Cano, C. A. (2021). Exploring students’ learning experience in online education: analysis and improvement proposals based on the case of a Spanish open learning university. Educ. Technol. Res. Dev. 69, 3367–3389.
Saarela, M. (2024). “On the relation of causality-versus correlation-based feature selection on model fairness,” in Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing, (Avila), 56–64.
Smirnov, A., Kashevnik, A., Petrov, M., Shilov, N., Scha¨fer, T., and Jung, T. (2019). “Competence-based language expert network for translation business process management,” in Proceedings of the 2019 25th Conference of Open Innovations Association (FRUCT), (Helsinki), 279–284. doi: 10.23919/FRUCT48121.2019.8981515
Sun, Z., Zhang, B., Gu, C., Xie, R., Qian, B., and Huang, H. (2022). ET2FA: A hybrid heuristic algorithm for deadline-constrained workflow scheduling in cloud. IEEE Trans. Serv. Comp. 16, 1807–1821.
Van Der Aalst, W., Adriansyah, A., De Medeiros, A. K. A., Arcieri, F., Baier, T., Blickle, T., et al. (2012). “Process mining manifesto,” in Business Process Management Workshops: BPM 2011 International Workshops, Clermont-Ferrand, France, August 29, 2011, Revised Selected Papers, Part I 9, eds F. Daniel, K. Barkaoui, and S. Dustdar (Berlin: Springer), 169–194.
Vergidis, K., Tiwari, A., and Majeed, B. (2007). Business process analysis and optimization: beyond reengineering. IEEE Trans. Syst. Man Cybernet. C 38, 69–82.
vom Brocke, J., and Mendling, J. (2016). Business process management cases–learning from real-world experience. BPTrends Column 1, 1–10.
Xie, X., and Rice, M. F. (2021). Instructional designers’ roles in emergency remote teaching during COVID 19. Dist. Educ. 42, 70–87.
Ying, L., Jiong, Z., Wei, S., Jingchun, W., and Xiaopeng, G. (2017). “Vrex: virtual reality education expansion could help to improve the class experience (vrex platform and community for vr based education),” in Proceedings of the 2017 IEEE Frontiers in Education Conference (FIE), (Indianapolis, IN), 1–5. doi: 10.1109/FIE.2017.8190660
Keywords: educational technology, Khan Academy, learning experience, localization, process re-engineering
Citation: Karimov A, Nolte A, Chounta I-A, Saarela M and Kärkkäinen T (2025) Improving learning experience through process re-engineering: Khan Academy localization into Azerbaijani. Front. Educ. 9:1419362. doi: 10.3389/feduc.2024.1419362
Received: 18 April 2024; Accepted: 20 November 2024;
Published: 30 January 2025.
Edited by:
Heidi Kloos, University of Cincinnati, United StatesReviewed by:
Mohammad Najib Jaffar, Islamic Science University of Malaysia, MalaysiaSheryl Morales, Polytechnic University of the Philippines, Philippines
Copyright © 2025 Karimov, Nolte, Chounta, Saarela and Kärkkäinen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ayaz Karimov, YWthcmltb3ZAanl1LmZp