Using the Theory of Practice Architectures to establish what it means to “do” learning design, and the arrangements that enable and constrain practice

In the past decade, learning design has become a widely adopted field of practice for higher education institutions (HEI) engaged with producing online and distance learning materials. To date, much has been written about the conceptual principles of guidance, representation, and sharing that underpin learning design, and the theoretical frameworks, models, tools, and instruments that have also been developed to support it. However, little analysis has been done to describe learning design in the specific sites of practice into which it has been introduced, or to describe the arrangements that might enable or constrain the embedding of this digital learning innovation by the people tasked with doing so. This original research article utilizes the Theory of Practice Architectures (TPA) as a theoretical approach to establish what learning design practice is composed of, and how that practice is shaped by its multiple sites of practice in the STEM faculty of a large open and distance learning HEI. The analysis draws on evidence—captured longitudinally over 4 years—from surveys (n = 43), learning design analytics (n = 20), in-depth interviews with key stakeholders (n = 14), document analysis, and learning design workshop data (n = 28) about the journey of 28 modules from conceptualization to faculty approval for full module production. The application of TPA to this extensive data set offers new and under discussed identification of key challenges experienced in the adoption of learning design approaches. In the sites of practice explored here, two specific arrangements are discussed: time, and the legacy of the Open University Learning Design Initiative (OULDI). Both can be seen to constrain and enable practice in different ways. This study will be relevant for scholars and researchers attempting to evaluate current learning design approaches or looking to explore more accurate ways of describing what it means to “do” learning design, both now and in the future.

1 Introduction

Over the past 10 years, the implementation and conceptual understanding of learning design has progressed considerably as it has been increasingly adopted as “a methodology for enabling teachers/designers to make more informed decisions in how they go about designing learning activities and interventions, which is pedagogically informed and makes effective use of appropriate resources and technologies” (Conole, 2012, p. 3). Early work in the field focused efforts on trying to develop practical visualizations that accurately represented teaching and learning, would enable a sharing of pedagogical designs, and would encourage the increased incorporation of technology in education (Bennett et al., 2022). This type of work “has been carried out under such names as pedagogical patterns, learning patterns and pattern language” and has been supported by substantial work in the related field of learning analytics (Lockyer et al., 2013, p. 1,441; Bakharia et al., 2016; Rienties et al., 2017). In 2012, leading researchers met in Cyprus to try and address the question of “What is Learning Design?” and as a result they produced the vitally important Larnaca Declaration on Learning Design. This work proposed three conceptual approaches that when taken together would “…provide a foundation for the field of learning design” (Dalziel et al., 2016, p. 21). The authors established the idea of a Learning Design Framework (LD-F) as the core concept, which linked the wider educational context and the design activity of educators [via the concepts of a Learning Design Conceptual Map (LD-CM) and Learning Design Practice (LD-P)] and is reproduced in Figure 1.

FIGURE 1

Figure 1. The three conceptual approaches to learning design (Adapted from Dalziel et al. (2016), licensed under CC BY 4.0 International).

This influential result reflects the thinking at the time because it deliberately primarily orientates learning design as a product (i.e., “a” learning design—a plan or recorded sequence of teaching and learning activities) and the tools that could be used to capture such products. There is no doubt that establishing a LD-F is important to any learning design approach but in retrospect and with the benefit of hindsight this pre-occupation of researchers with orientating learning design overwhelmingly as a product has been described as naïve, and future research work in need of a refocusing to also take the other two orientations into account (Bennett et al., 2022). For instance, learning design can also be orientated as process (i.e., one or more events or stages that are attended or completed to assist in the development of a piece of learning) and/or as a practice (i.e., the action of applying learning design concepts to the creation and implementation of a piece of teaching and learning). To date, these orientations have received less attention in the academic literature, and arguably this has been to the detriment of the progress of the field.

One way in which this deficit could be addressed is firstly by acknowledging that learning design does not exist in a vacuum, is situated, and will require much more than the creation of new online tools or visualizations in order to progress. By paying specific attention to historical evolution and situational circumstances such approaches can be viewed as part of growing body of educational technology research that is utilizing more socially driven and relational approaches to understanding digital educational innovations, of which learning design can be considered one (Castaneda and Williamson, 2021). Using an approach based in this tradition this study sought to gather evidence from all three orientations of learning design, (product, practice, and process) to enable a rich, contextualized case study to emerge of what it means to “do” learning design in the STEM faculty of The Open University, United Kingdom (UKOU), something which we believe has not been done before.

For example, in a wide-ranging review of European approaches to learning design, Wasson and Kirschner (2020) acknowledge the challenge of research needing to find ways to better support educators as learning designers. They draw an illustrative parallel between the role of a learning designer at a HEI and the role of a chef in a restaurant who is required to not only utilize the tools, ingredients, and techniques of cooking but also to organize, manage, and collaborate with the restaurant staff and environment in order to gain a Michelin star. Such an analogy is useful in highlighting the complexity of the learning designer’s situated role by comparing it with a role more commonly understood by most people. However, when the discussion turns to solutions that meet the challenges these appear to be found either in improved techniques (e.g., pedagogies), better tools (e.g., educational technologies), or more ingredients (e.g., learning analytics) rather than a more accurate understanding of the workings of the restaurant/HEI itself (Wasson and Kirschner, 2020). Research undertaken by Halupa (2019) and Seeto and Vlachopoulos (2015) have explored the important relationship(s) between designer and faculty in some detail but do not attempt to situate their findings within either a detailed historical/social context or with evidence from any learning design driven outputs.

Other studies agree that refocusing to explore these, “new” spaces in learning design research would be advantageous. Agostinho et al. (2018) interviewed 30 university teachers about the kinds of support they accessed to help them with their learning design work. They found a wide variety of sources which included: colleagues, literature, workshops, seminars, conferences, and other institutional support services. A literature review looking at the adoption of learning design tools and methods found that while there had been a focus on the usability of specific tools there was a lack of studies that investigated barriers to adoption such as institutional support (Dagnino et al., 2018). Mosley (2023a) has highlighted the challenge of being able to embed learning design meaningfully and effectively into institutional culture and other researchers have advised, “…leaders within institutions need guidance to address constraints and enhance enablers. Some of this work sits in policy change, some in configuring systems, services, and procedures to support teachers as designers, and some lies in fundamentally shifting management views to engage with the positive capacity of their workforce” (Bennett et al., 2022, p. 158).

To tackle this deficit, a call for the application of Theory of Practice Architectures (TPA) as a theoretical approach in documenting learning design practice in higher education and the structures and networks that support that practice, has been made as a way to prevent stagnation and ensure institutional arrangements are transparent and versatile (Bennett et al., 2018).

Theory of Practice Architectures has been described as “a basis for a contemporary theory of education appropriate for the modern world.” (Mahon et al., 2017, p. 16) and it has been widely applied as a tool for the analysis of such varied projects as Education for Sustainability (Kemmis and Mutton, 2012), English language teaching (Edwards-Groves and Grootenboer, 2015), doctoral education (Rönnerman and Kemmis, 2016) teacher education (Sjølie and Østern, 2021), and leading in educational contexts (Reich and Lizier, 2023).

Therefore, this study sought to address the following research questions:

RQ1: What does it mean to “do” learning design in the Faculty of STEM at the UKOU?

RQ2: What are the arrangements that enable and/or constrain the practice of learning design in the Faculty of STEM at the UKOU?

2 Materials and methods

For this study, “materials” refers to the background, context, sites of practice, and the instruments used to gather data and information. “Methods” refers to the way in which the instruments were applied, and the overall theoretical approach used (TPA) to organize and analyze the gathered data and information.

2.1 The Open University UK education model

The UKOU has around 170,000 students studying with it at any given time and as such is the largest university in Europe. Founded in 1969, its mission statement “Open to People, Places, Methods, and Ideas” reflects the values of inclusivity, innovation, and responsiveness. In the design, development, and production process academics and academic-related staff are responsible for designing and creating the learning materials that form “modules” and do this collaboratively in “module teams.” Every module team contains a lead academic, the Module Team Chair (referred to from here as Chair)—and a lead academic-related administrator, the Curriculum Manager (referred to from here as Manager); both of whom are based in faculty. The wider module team can also include other academics, project managers, learning designers, graphic designers, librarians, editors, employability experts, accessibility experts, and a variety of other staff some of whom are based in faculty and some of whom are based in other central units.

Modules are typically live or “in presentation” for a duration of 24 or 32 weeks and take the place of conventional lectures. While the module is being studied, Chairs and Managers are also interested in the tasks of monitoring, refinement, and improvement that goes along with any other teaching situation but do not have regular contact with students. Due to the distributed nature of the UKOU model, day to day academic support for students is provided by tutors. Tutors typically work part-time for the UKOU, often combining this work with other employment. They work from home and keep in contact with their students via phone, email, and managed online spaces. Tutors help students to navigate the learning materials and support assessed work. Students typically study via a combination of printed books and online materials presented on a Moodle based Virtual Learning Environment (VLE) developed by the UKOU and need to complete 360 credits to pass their degree qualification.

In open and distance learning education models such as the one described above, where an institutionally based interaction between educator, learner, and content exists despite the educator and learner being physically separated, learning design is of particular interest and importance (Mosley, 2023a). Unlike face-to-face synchronous teaching environments, the design of content and pedagogy are not delivered by the same person, and opportunities for feedback and immediate adjustments to teaching are limited. This places an increased emphasis on the quality of the design being of the very highest standard (Olney et al., 2021). Also, from the point of view of academics, tutors, and designers, both delivery and communication technologies are fundamental to the open and distance learning model, and therefore engagement is unavoidable. Such technologies have evolved through five generations, which have been identified as (1) postal correspondence, (2) broadcast radio and TV, (3) open universities, (4) teleconferencing, and (5) the internet (Moore and Kearsley, 2011). Distance education that takes place in the fifth generation, via the internet, and is typical at the UKOU, can be referred to as online and distance learning (ODL) (Martin et al., 2022).

2.2 Theory of Practice Architectures

The Theory of Practice Architectures (TPA) is described as “an account of what practices are composed of and how practices shape, and are shaped, by the arrangements in which they are enmeshed in a site of practice” and is shown in Figure 2 (Mahon et al., 2017, p. 6).

FIGURE 2

Figure 2. The Theory of Practice Architectures (Kemmis et al. (2014), reproduced with permission from SNCSC).

The core concept of a “site of practice” has been described as a realm or set of phenomena (if any) of which practice is intrinsically a part (Schatzki, 2003). As such, TPA takes a site-orientated, ontological approach to investigating practices. This theoretical approach can be used to develop a holistic view of practices such as ODL, that are essentially social phenomena, and address questions from a personal and organizational point of view about what is being done, how it is being done, and why it is done like that (Bennett et al., 2018).

Theory of Practice Architectures calls for an exploration and identification of the arrangements that prefigure and shape such practices that exist across three mediums: material-economic, cultural-discursive, and social-political (Mahon et al., 2017). In the practice of ODL, material-economic arrangements exist in physical space–time. They are what make the activities of the practice possible, and in the project of implementing learning design, might include such things as financial resources, workshop agendas, time, frameworks, approval documentation, learning design analytics, etc. These arrangements constrain or enable the doings of the practice. Cultural-discursive arrangements exist in semantic space. They refer to the specialist language or discourse that prefigure, constrain, or enable the sayings of a practice. Like any other projects within ODL teaching, learning design has developed its own specific references and language that is used by practitioners to describe and justify what it is and does—some of which have already been referenced in the introduction. These sayings might be contained in documentation or might be live in discussion between those engaged in design activities. Thirdly, social-political arrangements exist in the social space. They shape and prefigure the relatings of a practice and are therefore concerned with how humans relate to one another, behave in the roles they are representing, exist in the power structures that the organization provides, and bring experience to group or team environments. New curriculum requires approval and contribution from a variety of stakeholders within university systems so the way these individuals and teams relate to one another is of utmost relevance.

In the project of learning design, practitioners engage in a social phenomenon which contains specialist discourse (sayings), activities and work (doings), and engagement of academics and non-academics in a complex ecology of power structures and individuals (relatings). While these separate arrangements can be considered in isolation, in reality they are inter-related and prefigured, but not predetermined. They should therefore be viewed as being in a constant state of flux, constantly shaped and reshaped by the passing of time, events, and individuals in a dynamic relationship (Sjølie and Østern, 2021). Indeed, analysis of how these arrangements come to hang together in different sites of practice should allow for the analysis of how one impacts on another and the identification of new and progressive research approaches to educational innovations (Mahon et al., 2017).

2.3 Sites of practice, participants, and data collection

Using the concept of “sites of practice” from TPA, and the perspective that “practices can be located in multiple sites at one time, and one practice can be the site of another practice” (Mahon et al., 2017: p. 7), we sought to firstly identify the sites appropriate to our research question. In order to do this, we first drew on definitions provided by other studies and discussed how they related to our context. For example, since the OULDI is now completed (rather than ongoing) we felt it had become a site of practice, rather than a project. These key terms, and the elements they refer to in the context of this study, have been reproduced in Table 1.

TABLE 1

Table 1. Key terms of TPA, their features/description, and what they refer to in the context of this study.

Subsequently, we applied instruments that could capture the kinds and scope of data and information we thought would be useful in addressing the research questions. These are summarized in Table 2.

TABLE 2

Table 2. Sites of practice, types of instruments, and scope.

In terms of materials, what follows is a detailed description of each of the sites of practice included in the study, and an expanded account of the various instruments that were utilized. Other examples of applying TPA in education (see Introduction) include data gathered from a wide range of sources that included case studies, questionnaires, interviews, focus groups, and historical documents. Since we could find no guidance on how to gather and analyze evidence, we applied a range of approaches which are described below. Also included is a description of the methods used to analyze the findings from the application of the instrument.

2.3.1 The Open University Learning Design Initiative

Between 2007 and 2012, the OULDI sought to explore and eventually establish an agreed approach to learning design at the UKOU. It drew on many wide-ranging interviews with staff as part of the Institutional Approaches to Curriculum Design and Delivery program, which was co-funded by JISC and the European Union (EU) (Conole and Wills, 2013). The OULDI pilots led to the integration of the recommended approaches into a new phased approvals and governance process. This stage-gate process introduced the idea of “gatekeepers” who authorized the progress of curriculum between a series of “stages.” Learning design was, “… designed to further promote creativity and innovation, and introduce a consistent, structured design, specification, and review process to support the new approvals process” (Galley, 2015, p. 5).

While the learning design model recommended by OULDI was firmly orientated toward process it also aimed to establish design approaches that were student-focused, collaborative, and characterized by the three principles of:

i. mechanisms to encourage design conversations across disciplines and expert roles;

ii. the use of tools and instruments as a means of describing and sharing designs; and

iii. the use of information and data to inform the conceptual tools and frameworks that guide the decision-making process (Galley, 2015, p. 6)

Between 2014 and 2016, an implementation project team located within the Institute of Educational Technology (IET) piloted and trialed the learning design approaches established by OULDI with faculties (for a fuller description of the OULDI recommendations see Cross et al., 2012. For a fuller description of later implementation see Olney et al., 2019).

We reviewed the recommendations made by OULDI (Cross et al., 2012) and considered how these recommendations were now being implemented in STEM using the three learning design conceptual approaches of LD-F, LD-CM, and LD-P identified by Dalziel et al. (2016) and described in the introduction. By doing so we have been able to situate OULDI recommendations in the thinking from that time and also highlight how the process orientation of OULDI (see section 2.3.1) is absent from the conceptual approaches.

2.3.2 Curriculum governance processes in the Faculty of STEM

In 2016, the OU was restructured into four faculties and the Faculty of STEM was created from the merger of the Mathematics, Computing & Technology Faculty, and the Science Faculty. STEM currently manages a curriculum offering of around 155 modules which comprises about 40% of the overall curriculum at the UKOU. A centrally located, permanent Learning Design Team, based in Learner and Discovery Services (LDS), was created from the pilot team described above, and the activity of learning design was moved into a business-as-usual format. Among many other things, each new faculty was tasked with developing structures, governance, and procedures that would support their academics and module teams in designing teaching and learning appropriate to their own context and establishing their own relationship with the Learning Design Team. Compared to the other new faculties, STEM retained significant control over learning design. Responsibility was located within the portfolio of the Associate Dean, Student Experience (AD-SE). Until faculty approval is achieved learning design support is provided by the Senior Manager, Learning & Teaching (SM-L&T) after which the Learning Design Team in LDS support further, more granular, learning design activities.

Learning design that took place in the period from initial conception to faculty approval was in scope for this project. Learning design that took place after faculty approval was not in scope.

The primary mechanism for STEM faculty approval is via the consideration of a Module Specification Document which contains key learning design outputs. This needs to be approved at a School Board of Study (BoS) and Faculty Scrutiny Group (SG) before resources can be made available for further development. This approval process is referred to as “Stage 4: Module Specification and Production” of the stage-gate approvals process.

An interview instrument was collaboratively designed by the members of the project team based around the principles of quality, confidence, and efficiencies in module specifications being approved over time. Participants were 14 members of either Board of Studies (Academic 2, Professional 3) or Scrutiny Group (Academic 4, Professional 5) with years of experience ranging from 1 to 5 years were identified and interviewed by two members of the project team. Interviews were recorded in MS Teams, transcribed, and imported into NVivo12 for coding and analysis by the authors. Interviewees were anonymized and are referred to in the results section by an identifying number in square brackets.

2.3.3 Learning design intention

As with other learning design initiatives a fundamental recommendation of the OULDI was the adoption of the Activity Types Classification Framework (Conole, 2012). This framework encourages learning to be divided into seven student focused categories (Table 3) to which time spent is then allocated at different levels of granularity to create shareable visualizations that reflect the learning design as product orientation. Once captured, these learning design analytics have been combined with VLE behavior (or other forms of learning analytics) to provide detailed insight into the pedagogies adopted by academics when designing ODL and to shed light on the student experience (Rienties et al., 2017).

TABLE 3

Table 3. The Activity Types Classification Framework.

One of the parts of the module specification document that is considered by BoS and SG is a completed Activity Planner. This learning design product visualizes the design intention of the module team by capturing the proposed learning activities, as classified by the Activity Types Classification Framework, and the expected time students will spend studying in each (Olney et al., 2019).

In this study, 20 modules submitted completed Activity Planners that could be said to represent the learning design intention of the module team. These completed Activity Planners were either downloaded from the online Learning Design Tool or data was extracted from module specification documentation and combined in a MS Excel spreadsheet. Comparison of the Activity Type distribution for the 20 modules was calculated. Mean averages for expected student workload in each Activity Type was then calculated and the totals were compared against the mean averages from 151 UKOU modules (inc. STEM) which was extracted and analyzed in 2015 (Rienties and Toetenel, 2016).

2.3.4 Learning design workshop agendas

Learning design workshops (LDW) provide a focal mechanism for bringing together members of the module team to reach consensus of key elements of the design of new curriculum. In STEM, LDW are facilitated by the SM-L&T.

Analysis of 28 LDWs are included in this study. The average time of the 28 LDW was 4.8 h, with each individual LDW varying in length from 2.5 to 7 h. 25 LDW were face to face, three were online. LDW agendas (tailored to the specific needs and context of the module team) are set by the SM-L&T, the Chair and the Manager who meet 5–10 days before the LDW to do this. The agendas for these workshops were downloaded and the frequency of the different activities or topics were collated and summarized.

2.3.5 Module team perceptions of components of STEM learning design

A survey instrument was created in MS Forms which was made up of seven positive statements about various “components of STEM learning design.” Each statement was designed to draw on the principles established by OULDI and outlined by Galley (2015). Respondents were asked to indicate the extent to which they agreed with each statement on a Likert scale with four options. There was also a general, open-ended question for other comments.

A link to the survey instrument was sent to the Chair and Manager of new modules which had engaged with learning design support, had submitted a module specification document, and had that specification approved by faculty Scrutiny Group. The survey was sent shortly after documentation had been approved at Scrutiny Group.

First module approved Scrutiny Group: July 2017.

Last module approved Scrutiny Group: Mar 2021.

Total modules: 28.

Total surveys sent: 58 (Manager: 28; Chair: 30).

Survey responses: 43 (Manager: 21; Chair: 22).

Response rate: 74%.

The results for Q1-7 (Likert responses) were exported from MS Forms into MS Excel and numerical values were allocated, that is; disagree = 0, slightly agree = 1, mostly agree = 2, and completely agree = 3. Therefore, the highest possible score for any possible response was 21. The results for Q8 (general open comment) were imported into NVivo 12 for analysis.

3 Results

For simplicity, in the following section, we have presented the results from each of the sites of practice that were identified as being included in the study. The analysis of these results using TPA are then presented and discussed in section 4.

3.1 Open University Learning Design Initiative

The central concept of a LD-F is based on the idea that educators should be able to share good examples of practice in learning and teaching, much like a musical notation system. Like learning and teaching, musical representations cannot capture everything about music—the “performance” is still essential—but by writing down music great works of art can be shared and valued across cultures and time (Dalziel et al., 2016). At the OU, this notation system is the Activity Types Classification Framework (Conole, 2012) and its representation is the Activity Planner.

Outputs of OULDI that are reflected in the concept of a LD-CM could be found in the establishment of Compendium DL learning design mapping software and the suggested use of the “Module Map” visualization. Before the use of Compendium DL was discontinued in 2019, it allowed for module teams to electronically visualize the component parts of a piece of teaching and learning. The Module Map visualization is still available for module teams to document their discussions in four areas of the student experience: guidance and support, content and activities, reflection and demonstration, communication, and collaboration.

As has been described earlier, in STEM, the Activity Planner is required to be completed for the module approvals process, but the use of the Module Map is sporadic, and has fallen out of common use. In reality, it is the Module Specification Document that bears the closest comprehensive resemblance to the LD-CM as laid out by Dalziel et al. (2016) despite it not being a visual representation. It contains details of proposals for (among other things): staffing, curriculum rationale, learning outcomes, registration requirements, student profiles, projected enrolments, external recognition, learning and teaching design, tuition, assessment strategies, delivery methods, accessibility, and risk assessment. After approval, module teams continue to engage with the Activity Planner to create more granular visualizations of the student experience on a module as the design develops.

The OULDI recommended and piloted approaches to LD-P which required low levels of orchestration in the shape of a voluntary design community. Sharing examples of practice and having access to a network of other design orientated academics and learning designers was encouraged through the establishment of the Cloudworks website in 2009 (Conole, 2010). However, in 2019 this early social networking platform was eventually archived as funding for its maintenance was discontinued and to date has not been replaced. In contrast, more orchestrated approaches, in the shape of facilitated LDW, were also recommended and piloted by OULDI. These continue to be a very common feature of the STEM approach to learning design.

3.2 Curriculum governance in the Faculty of STEM

The interviews with 14 members of SG and BoS tasked with reviewing module specification documents and ultimately approving module designs, revealed a range of responses. When asked about their perceptions of learning design support several reported how some academic colleagues had described the experience in negative ways, describing learning design as “hurdles to get over,” “what a waste of time, people come in here and tell me how to write a module, we know how to write a module” or hearing, “moans and groans.”

In general, the interviewees struggled to find ways to articulate how learning design support may have specifically impacted on positive changes to module specification quality, their confidence in the readiness of module teams to move on, or efficiencies of process. As one interviewee put it, “I would not necessarily be able to look at a spec. and say oh you know I can see that got there because of those discussions in a learning design workshop. I might be able to, but I might not.” Also, many expressed difficulties in recalling details of module specifications from the past with anything like the required accuracy to be confident about commenting.

The experience of the module team, rather than learning design support, emerged as an important factor for the interviewees in evaluating the process of reviewing and approving module specifications (seven comments). Several interviewees also referenced other complexities, such as: the timescales module teams were working within, or the staffing resources available to module teams, or the specific context of the module they were designing, that did not allow them to compare circumstances and isolate learning design.

Nevertheless, when pushed to assign values or asked directly about their perceptions of change over their time reviewing and approving module specifications, the analysis found that the interviewee’s responses were positive about the implementation of learning design in STEM.

In terms of responses to questions about their perception of a change in the quality of module specifications: zero interviewees thought that quality had declined, nine interviewees thought there was no change to the quality, and five interviewees felt the quality had improved.

When asked about any change in their confidence that module teams were ready to move to the next stage: three interviewees said they felt less confident, five said there was no change in their confidence, while six said they felt more confident (The reasons given for less confidence now were to do with increased expectations around the use of technology, and compressed timelines and tighter deadlines for presentation, rather than learning design).

In responses to questions about their perception of changes to efficiencies of process as a result of learning design: zero felt the process was less efficient, nine said there was no change, and three thought that efficiencies had now improved.

Five interviewees saw improvements in at least two of the three areas of quality, confidence, and efficiencies they were asked about.

Many positive comments and areas of impact associated with the learning design support provided were identified by the interviewees. These included module teams now being well placed to reduce student workload (two comments), improve learning outcomes (two comments), articulate a coherent rationale, and think through what the student experience would be, irrespective of the content (six comments). Interviewees explained they perceived the value of the support was in being able to “ringfence” or “protect” time (two comments) in order to have “a very productive conversation,” about elements of the student experience that were not content driven but needed to be “directly questioned,” “challenged,” or addressed as a “step in the process.” One interviewee highlighted that, due to learning design support, they felt they were now, “less in the position of signing a blank cheque of approving a module where there are lots of unknowns,” while for another learning design support, “meant that they [the module team] just had a clearer idea about what they were going to produce.” Despite five of the interviewees having never attended a LDW their perceptions of the activities that took place there were broadly accurate, specifically referencing the development of learning outcomes (two comments) assessment strategies, and “how to teach,” as being among the expected activities.

3.3 Learning design intention

20 STEM modules submitted usable Activity Planners with their module specification documents. Figure 3 shows the learning design intentions of these modules combined to provide mean averages for each Activity Type. This has then been compared with mean averages for 151 modules from the whole of the UKOU calculated by Rienties and Toetenel (2016) in 2015. The comparison shows that mean averages for the 20 STEM modules are lower for assimilative (−3.0%), productive (−3.9%), and assessment (−11.3%) categories than the UKOU mean averages were in 2015 but were higher for finding & handling information (+3.3%), communication (+3.3%), experiential (+3.7%), and interactive/adaptive (+8.0%) categories.

FIGURE 3

Figure 3. The mean averages of the learning design intentions of 20 STEM modules compared with the mean averages of 151 UKOU modules.

3.4 Learning Design Workshop (LDW) agendas

Figure 4 shows how often particular learning design activities were chosen for inclusion in a LDW through a consultation between SM: L&T, Chair and Manager based on the priorities of that module. It shows that the Activity Planner, setting the context and vision and next steps were included in every LDW. Learning outcomes, student profiles, and assessment design were also very frequently included, while skills mapping, employability, ed. tech tools/interactives and subject specific content also figured in repeated LDW. Accessibility and design challenges were included rarely.

FIGURE 4

Figure 4. Frequency of learning design activities included in 28 LDW.

3.5 Module team perceptions of components of STEM learning design

Figures 5, 6 show the results of the survey instrument where values have been allocated to the responses of Chair and Manager to the seven “components of STEM learning design” statements. This has allowed for a level of agreement value to be calculated between 1 and 21.

FIGURE 5

Figure 5. Level of agreement with components of STEM learning design’ statements over time by module specification approval date for Chair and Manager (43 responses).

FIGURE 6

Figure 6. Level of agreement with “components of STEM learning design” statements over time between LDW and module specification approval for Chairs and Managers (43 responses).

In Figure 5, the trendline shows there has been increasing agreement with the “components of STEM learning design” over time for the 43 Chair and Manager who responded to the survey.

Figure 6 shows the relationship between agreement with the seven “components of STEM learning design” statements and the time between LDW and Scrutiny Group for Chair and Manager. Despite some outliers, it shows there is a cluster of high level of agreement responses (12–21) with the “components of STEM learning design statements” when the LDW workshop takes place 2–6 months before module specification approval at Scrutiny Group. This cluster of responses is indicated by the circle.

4 Discussion

In order to address both the RQs we sought to apply TPA as a theoretical approach to structure our analysis, and as a transformative framework that would reveal future perspectives (Reich and Lizier, 2023). That analysis is presented here.

4.1 RQ1: What does it mean to “do” learning design in the Faculty of STEM at the UKOU?

In order to address RQ1 we have provided evidence of the interconnected sayings, doings, and relatings that constitute the project of adopting learning design within the practice of open and distance learning and provide a picture of what it means to “do” learning design in the Faculty of STEM. To do this we identified five sites of which the practice is intrinsically a part, developed instruments to capture information from them, and then tracked how doings, sayings, and relatings manifested across those sites.

For example, for an academic tasked with the design of a new module it can be seen that they would, at a minimum, engage with doings in the form of establishing an agenda and participating in a LDW, producing a learning design intention, and attendance at Board of Study and Scrutiny Group. They would also be required to use the specific language of learning design in documents and conversation that might include references to Activity Type Classification Frameworks, learning design analytics, vision, learning outcomes etc., as sayings, that may well be unfamiliar and/or no longer in use (e.g., OULDI). They would need to develop relatings with other academics in positions of power (such as those on Board of Study or Scrutiny Group) or possibly subordinate (module team members) as well as academic-related members of staff, both inside and outside the faculty, such as the Manager, SM-L&T or, in some cases, for example, interactive developers.

Taken on their own these features may seem unremarkable, but by gathering and organizing the data in this way we have been able to describe what it means to “do” learning design in all three orientations, product, practice, and process from five situated sites of practice. TPA has provided a useful theoretical approach in which comparisons across different sites or projects could be made.

4.2 RQ2: What are the arrangements that enable and/or constrain the practice of learning design in the Faculty of STEM at the UKOU?

In order to address RQ2, we have analyzed the sayings, doings, and relatings, and identified examples of the material-economic, cultural-discursive, and social-political arrangements that prefigure and shape them. Then we have considered, how the arrangements hang together, and the extent to which they enable and/or constrain learning design practice. An approach such as this is important in pushing the field forward, since providing guidance for senior leaders, who can shape policies, systems, services, and procedures, is vital in realizing future perspectives of learning design (Dagnino et al., 2018; Bennett et al., 2022). Further, if researchers at multiple HEI undertake scholarship in this way results can be pooled and the impact of different arrangements in different projects better understood.

However, for reasons of space not all of these complex arrangements can be presented in detail here. So, in order to demonstrate the value of TPA, we will consider two contrasting but complementary examples of arrangements, OULDI, and time, and then discuss future improvements and areas for research that they suggest.

4.2.1 OULDI as an arrangement

In this study OULDI has previously been considered as a site of practice, but it is legacy can also be considered as an enabling arrangement that contains multiple examples of practice.

For example, as a social-political arrangement, OULDI recommended the creation of specialized learning design roles and expertise that could bring together and facilitate module team conversations (relatings). As a material-economic arrangement, it enabled LDWs (doings) as a practical, accepted mechanism to facilitate learning design. This enabling influence is evidenced in three different sites of practice: by increasing levels of agreement by Chairs and Managers with the “components of STEM learning design statements” (relevant to these doings and relatings), the LDW workshop agendas, and positive perceptions of learning design from curriculum governance stakeholders. A 2018 set of interviews found that in “all forms of support those participants reported they drew on [for learning design], were inherently social” and concluded that “…these findings suggest the university teachers value support from credible others” (Agostinho et al., 2018, p. 9, 11). Supporting the development of communities of learning design practice was evidenced in at least two sites of practice (OULDI, curriculum governance). As a social-political arrangement, OULDI was initially able to enable this practice, but since the demise of the Cloudworks platform no other arrangement has taken its place.

However, evidence was also gathered from the curriculum governance and perceptions of learning design sites of practice that demonstrates how the influence of OULDI has been a constraining social-political and cultural-discursive arrangement. Negative comments that associated learning design with the introduction of the Stage-gate process and the language of “gatekeepers” and “quality assurance” served to position learning design as a process driven concept, rather than a creative, practice driven one, and may have resulted in a perception that learning design is something that is done “to” module teams, rather than “with” them. Other evidence found in the perceptions of learning design site of practice also suggested that there were mixed attitudes toward the use of the Activity Planner as a way to represent intended learning designs as product. Some Chairs and Managers considered this example of OULDI as a material-economic arrangement a constraint on practice because it limited creativity and acted as a pedagogical restriction. However, several others explicitly commented that they found it useful, helpful, and student-focused. Failing to provide opportunities for true collaboration between all stakeholders in learning design has been identified by several researchers as a barrier to the maturity of the field (Seeto and Vlachopoulos, 2015; Halupa, 2019; Mosley, 2023b).

Either way, the evidence shows that perceptions toward learning design as a product became more positive over time and served to support the design of more student-focused ODL. Intended learning designs produced by STEM module teams during this period contained less assessment and assimilative activities, and more examples of “active learning” than those intentions previously captured from the UKOU. Another study used learning design analytics to assert that the pedagogy of UKOU modules could be described as fitting into one of four design clusters based on the learning design intention: (1) constructivist, (2) assessment driven, (3) balanced variety, and (4) social constructivist (Rienties et al., 2015). It could be claimed that STEM module teams are increasingly intending to use fewer cluster 1 and 2 type designs, and more cluster 3 and 4 type designs, when compared against the OU averages in 2016.

From this analysis and discussion of OULDI as a constraining and/or enabling arrangement other possible routes of further inquiry can be suggested. Specifically, one avenue could be to consider influencing or adapting historical sayings, doings, and relatings that pivot learning design away from an association with process, while still maintaining a product orientated approach, but also promoting a more practice orientated perception. One way to do this, without damaging the gains associated with the product approach, could be to explicitly reference within which orientation a learning design conversation or document is situated. This approach could raise awareness about the theoretical underpinnings of learning design and shift sayings and language choice during governance. The evidence collected here suggests that this is relevant for the sites of practice in this context, but it might not be in other sites or projects. By applying TPA to different contextualized projects, researchers could compare approaches to learning design and consider what works where, and why.

4.2.2 Time as a material-economic arrangement

Time also emerged from the collected data as an important material-economic arrangement that could be seen to enable and/or constrain learning design practice in several different ways.

Firstly, evidence from the perceptions of learning design site of practice demonstrated it was advantageous to hold a LDW 2–6 months prior to the approval of the module specification since responses to the “components of STEM learning design statements” were most positive when this window was realized (Figure 6). This was supported by seven Chair or Manager comments who perceived their LDW as being too early in the design process, and two who said it came too late. All nine regarded the timing as important and a constraint on practice. Secondly, the LDW Agenda analysis (Figure 4) revealed that the time spent at LDW varied and was mentioned by seven Chairs or Managers in the comments. For four of those, the LDW was not long enough to fulfill expectations around what could be achieved. However, being able to shape how that time was spent by contextualizing the agenda and having enough structured preparation time for the LDW were considered to be enabling arrangements by both Chair and Manager. Some comments from the Board of Study and Scrutiny Group stakeholders also presented the time spent at LDW as an enabling arrangement, since it facilitated time spent considering non-content elements of the student experience which could otherwise be lost. Thirdly, previous time spent by Chairs and Managers on designing and creating ODL was highlighted by several Board of Study and Scrutiny Group stakeholders as being a constraining or enabling arrangement that shaped the quality, confidence, and efficiency of the approval of module specification documents.

The application of TPA as a theoretical approach has enabled the identification of these three examples of time as material-economic arrangements. Further, all three relate to learning design as a process that deserves consideration among the other two orientations and could well lead to improvements in how learning design is implemented in this, and potentially, other sites of practice.

For example, learning design support could start to be offered on a more ongoing, asynchronous basis that complements any single event LDW. Other researchers have highlighted how placing emphasis on one-off events such as LDW can be problematic when circumstances (such as, leave, research commitments, or emergencies) get in the way of the timing and limit learning design effectiveness (Mosley, 2023b). Also, module teams could agree a preferred governance route and work backwards in order to maximize the learning design support provided to them. Drawing on the second example, learning design support should encourage high levels of contextualization that is guided by the module team. In order to be successful, research by Halupa (2019) highlights the need for the roles of both designers and faculty members to be clearly delineated with control over content being reinforced as the responsibility of the academics. In regard to the third example, Chao et al. (2010), have also shown how the level of the experience of the faculty member is key to the level of collaboration in developing online courses. Those in positions of power over process could focus future efforts on allocating staff to module teams and considering the timelines associated with them more closely.

4.3 Future work and limitations

The scope of this study is limited. It covers the period of time between initial agreements among stakeholders that a new module or significant module rewrite is desirable, to the point of faculty approval for resources to be released. At this point there is still much learning design work to be done and this support is provided largely from outside the faculty. For this reason, future perspectives could include developing case studies of modules that provide several different evaluation points. A comparison of final learning designs against intended learning designs using the Activity Types Classification Framework may also illuminate how design decisions and compromises need to be made by design teams. Establishing a link between this early phase of learning design practice and final design output would also allow for a closer examination of the impact of learning design on students, which we have not been able to do here.

4.4 Conclusion

In this study, we have used TPA as both a theoretical approach and a transformative framework (Reich and Lizier, 2023) to describe what it means to “do” learning design in the Faculty of STEM in terms of the sayings, doings, and relatings that exist in five sites of practice. We have also identified and discussed two specific examples of arrangements, OULDI and time, and the ways in which they constrain or enable learning design. Through the analysis of these arrangements using TPA, we have been able to identify and suggest future directions for learning design that relate specifically to this situated project but will be applicable to other projects. As Reich and Lizier (2023) have suggested, building an account of practice with TPA as a lens in this way could allow for further comparative studies to take place, generate interesting insights into the practice architectures that are enmeshed in other sites or projects and help to explain why they have developed in those ways. Other HEI will have comparable sites of practice and projects to implement learning design with comparable features. Ultimately, a collection of such studies could contribute to building a culture of teacher design and progress research in the field (Bennett et al., 2022).

Crucially, this study has also demonstrated how TPA, if applied consistently, can be used as a way to review, and describe learning design implementations that encourages all three orientations of learning design, practice, process, and product to be considered together, and in a situated way. This opens up the prospect of new avenues of inquiry and opportunities for solutions to challenges that are not necessarily technology based or focused on learning design as product. In conclusion, we encourage other researchers to consider applying TPA as a theoretical approach to ensure the next level of maturity for the field of learning design.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

Ethical approval was not required for the studies involving humans in accordance with the local legislation and institutional requirements. The studies were conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required from the participants or the participants’ legal guardians/next of kin in accordance with the national legislation and institutional requirements.

Author contributions

TO: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Writing – original draft, Writing – review & editing. CW: Conceptualization, Investigation, Methodology, Resources, Writing – review & editing.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Acknowledgments

The authors would like to acknowledge and thank The Center for Scholarship in STEM (eSTEeM) at the UKOU for their support with the development of this project. The original working report prepared for eSTEeM can be found at: https://www.open.ac.uk/scholarship-and-innovation/esteem/sites/www.open.ac.uk.scholarship-and-innovation.esteem/files/resources/Tom%20Olney%2C%20Learning%20Design%20Approaches%20in%20STEM.%20eSTEeM%20Final%20Report%20.pdf (Accessed September 11, 2023). This material has not been peer-reviewed and has not been published within academic media.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The handling editor CH declared a shared affiliation with the authors at the time of review.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Agostinho, S., Lockyer, L., and Bennett, S. (2018). Identifying the characteristics of support Australian university teachers use in their design work: implications for the learning design field. Australas. J. Educ. Technol. 34, 1–14. doi: 10.14742/ajet.3776

CrossRef Full Text | Google Scholar

Bakharia, A., Corrin, L., de Barba, P., Kennedy, G., Gašević, D., Mulder, R., et al. (2016). “A conceptual framework linking learning design with learning analytics” in Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, Edinburgh.

Google Scholar

Bennett, S., Agostinho, S., Lockyer, L., Jones, J., and Gigliotti, A. (2022). “Chapter 12. From learning design to teacher design practice: researching how teachers design for technology integration” in Handbook of Digital Higher Education. eds. D. Persico, F. Pozzi, and P. Goodyear (Edward Elgar Publishing), 149–160.

Google Scholar

Bennett, S., Lockyer, L., and Agostinho, S. (2018). Towards sustainable technology-enhanced innovation in higher education: advancing learning design by understanding and supporting teacher design practice. Br. J. Educ. Technol. 49, 1014–1026. doi: 10.1111/bjet.12683

CrossRef Full Text | Google Scholar

Castaneda, L., and Williamson, B. (2021). Assembling new toolboxes of methods and theories for innovative critical research on educational technology. NApprEdR 10, 1–14. doi: 10.7821/naer.2021.1.703

CrossRef Full Text | Google Scholar

Chao, I. T., Saj, T., and Hamilton, D. (2010). Using collaborative course development to achieve online course quality standards. Int. Rev. Res. Open Dist. Learn. 11, 106–126. doi: 10.19173/irrodl.v11i3.912

CrossRef Full Text | Google Scholar

Conole, G. (2010). Facilitating new forms of discourse for learning and teaching: harnessing the power of web 2.0 practices. Open Learn 25, 141–151. doi: 10.1080/02680511003787438

CrossRef Full Text | Google Scholar

Conole, G. (2012). Designing for Learning in an Open World. Explorations in the Learning Sciences, Instructional Systems and Performance Technologies, 4. New York: Springer

Google Scholar

Conole, G., and Wills, S. (2013). Representing learning designs—making design explicit and shareable. Educ. Media Int. 50, 24–38. doi: 10.1080/09523987.2013.777184

CrossRef Full Text | Google Scholar

Cross, S., Galley, R., Brasher, A., and Weller, M. (2012). OUlearning design I-JISC project evaluation report: The impact of new curriculum design tools and approaches on institutional process and design cultures. OUlearning design I Project (Open University).

Google Scholar

Dagnino, F., Dimitriadis, Y., Pozzi, F., Asensio-Pérez, J., and Rubia-Avi, B. (2018). Exploring teachers’ needs and the existing barriers to the adoption of learning design methods and tools: a literature survey. Br. J. Educ. Technol. 49, 998–1013. doi: 10.1111/bjet.12695

CrossRef Full Text | Google Scholar

Dalziel, J., Conole, G., Wills, S., Walker, S., Bennett, S., Dobozy, E., et al. (2016). The Larnaca declaration on learning design. J. Interact. Media Educ. 2016, 1–24. doi: 10.5334/jime.407

CrossRef Full Text | Google Scholar

Edwards-Groves, C., and Grootenboer, P. (2015). Praxis and the theory of practice architectures: resources for re-envisioning English education. Austral. J. Lang. Liter. 38, 150–161. doi: 10.3316/informit.480213065682061

CrossRef Full Text | Google Scholar

Galley, R (2015). Learning Design at the Open University: Introducing methods for enhancing curriculum innovation and quality. Quality Enhancement Report. The Open University.

Google Scholar

Halupa, C. (2019). Differentiation of roles: instructional designers and faculty in the creation of online courses. Int. J. High. Educ. 8, 55–68. doi: 10.5430/ijhe.v8n1p55

CrossRef Full Text | Google Scholar

Kemmis, S., and Mutton, R. (2012). Education for sustainability (EfS): practice and practice architectures. Environ. Educ. Res. 18, 187–207. doi: 10.1080/13504622.2011.596929

CrossRef Full Text | Google Scholar

Kemmis, S., Wilkinson, J., Edwards-Groves, C., Hardy, I., Grootenboer, P., and Bristol, L. (2014). “Praxis, practice and practice architectures,” in Changing practices, changing education. (Singapore: Springer), 25–41.

Google Scholar

Lockyer, L., Heathcote, E., and Dawson, S. (2013). Informing pedagogical action: aligning learning analytics with learning design. Am. Behav. Sci. 57, 1439–1459. doi: 10.1177/0002764213479367

CrossRef Full Text | Google Scholar

Mahon, K., Kemmis, S., Francisco, S., and Lloyd, A. (2017). “Introduction: practice theory and the theory of practice architectures” in Exploring Education and Professional Practice. eds. K. Mahon, S. Francisco, and S. Kemmis (Singapore: Springer)

Google Scholar

Martin, F., Sun, T., Westine, C., and Ritzhaupt, A. (2022). Examining research on the impact of distance and online learning: a second-order meta-analysis study. Educ. Res. Rev. 36:100438. doi: 10.1016/j.edurev.2022.100438

CrossRef Full Text | Google Scholar

Moore, M., and Kearsley, G. (2011). Distance Education: A Systems View of Online Learning Cengage Learning.

Google Scholar

Mosley, N (2023a) What’s behind the growth and interest in learning design? Available at: https://www.neilmosley.com/blog/whats-behind-the-growth-and-interest-in-learning-design (Accessed August 07, 2023).

Google Scholar

Mosley, N (2023b). How do universities achieve learning design maturity? Available at: https://www.neilmosley.com/blog/how-do-universities-achieve-learning-design-maturity (Accessed August 07, 2023).

Google Scholar

Olney, T., Li, C., and Luo, J. (2021). Enhancing the quality of open and distance learning in China through the identification and development of learning design skills and competencies. Asian Assoc. Open Univ. J. 16, 61–78. doi: 10.1108/AAOUJ-11-2020-0097

CrossRef Full Text | Google Scholar

Olney, T., Rienties, B., and Toetenel, L. (2019). “Chapter 6: gathering, visualising and interpreting learning design analytics to inform classroom practice and curriculum design” in Learning Analytics in the Classroom: Translating Learning Analytics for Teachers. eds. J. Lodge, J. Horvath, and L. Corrin (London: Routledge Press)

Google Scholar

Reich, A., and Lizier, A. L. (2023). De-centering the leader: using the theory of practice architectures in a postgraduate education course. J. Educ. Administr. Hist. 55, 6–22. doi: 10.1080/00220620.2022.2114435

CrossRef Full Text | Google Scholar

Rienties, B., Nguyen, Q., Holmes, W., and Reedy, K. (2017). A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK. Interact. Design Architect. 33, 134–154. doi: 10.55612/s-5002-033-007

CrossRef Full Text | Google Scholar

Rienties, B., and Toetenel, L. (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Comput. Hum. Behav. 60, 333–341. doi: 10.1016/j.chb.2016.02.074

CrossRef Full Text | Google Scholar

Rienties, B., Toetenel, L., and Bryan, A (2015). “Scaling up learning design: impact of learning design activities on LMS behavior and performance” in Fifth International Conference on Learning Analytics and Knowledge. Amanager, New York, NY, USA. 315–319.

Google Scholar

Rönnerman, K., and Kemmis, S. (2016). Stirring doctoral candidates into academic practices: a doctoral course and its practice architectures. Educ. Inq. 7:27558. doi: 10.3402/edui.v7.27558

CrossRef Full Text | Google Scholar

Schatzki, T. (2003). A new societist social ontology. Philos. Soc. Sci. 33, 174–202. doi: 10.1177/0048393103033002002

CrossRef Full Text | Google Scholar

Seeto, D., and Vlachopoulos, P. (2015). “Design develop implement (DDI)—a team-based approach to learning design” in THETA 2015 Conference. Gold Coast, Australia.

Google Scholar

Sjølie, E., and Østern, A. L. (2021). Student teachers’ criticism of teacher education–through the lens of practice architectures. Pedagog. Cult. Soc. 29, 263–280. doi: 10.1080/14681366.2020.1729842

CrossRef Full Text | Google Scholar

Wasson, B., and Kirschner, P. (2020). Learning design: European approaches. TechTrends 64, 815–827. doi: 10.1007/s11528-020-00498-0

CrossRef Full Text | Google Scholar