An Integrated Environment for Learning Design

Learning design tools aim at supporting practitioners in their task of creating more innovative and effective computer-supported learning situations. Despite there being a myriad of proposed tools, their use presents challenges that recent studies link with practitioners' varied pedagogical approaches and context restrictions, as well as with barriers to practical application derived from the fact that most tools only cover limited functionality and do not support cooperation between practitioners. In this paper we investigate whether it is possible to provide a flexible community system that supports multiple learning design tasks. We propose an Integrated Learning Design Environment (ILDE), which is a networked system integrating collaboration functions, design editors and middleware that enables deployment of the designed learning situations into Virtual Learning Environments. We describe the iterative user-centered process adopted in the design of ILDE as well as its architecture. The architecture is implemented to show its feasibility and that it is capable of providing the targeted functionality. We also present the results of its use in training workshops with 148 practitioners from five different institutions in vocational training, higher and adult education. Some of the learning designs were deployed in VLEs and enacted with students in real learning situations.

Introduction

Teaching as a “design science” (Laurillard, 2012) aims at creating effective conditions for learners to learn (Mor et al., 2013). Practitioners design learning situations using the resources available, making design decisions that are informed not only by their knowledge of learning theories, but also by their past experience, by their assumptions about the learners, and by the formal or informal help coming from other colleagues (Bennett et al., 2017). The field of learning design (Lockyer et al., 2008; Conole, 2012) has the ultimate goal of improving teaching quality by supporting practitioners along the process of designing innovative and more effective learning situations (that is, producing “learning designs”). Learning design has its roots in instructional design, with the shared aim of systematizing the process of finding effective solutions to educational problems. However, learning design heavily relies on tools that allow the development of a participatory culture of design to encourage the sharing of all the half fabricates of the pedagogical design. The underlying assumption is that a participatory learning design culture can change the way teachers work, transforming them from lonely practitioners into networked professionals, engaged in pedagogical inquiry (Persico and Pozzi, 2015). A “learning design” is an artifact that explicitly documents a set of learning tasks (different granularities are possible, from a single task to a course, Hernández-Leo et al., 2007) with the set of resources and tools that support the realization of the tasks. Persico et al. (2013) and Prieto et al. (2013b) provide an example of a learning design represented with different approaches and tools.

Learning design research has contributed with a profuse set of representations and computer tools (Persico et al., 2013; Prieto et al., 2013b; Celik and Magoulas, 2016) to help practitioners¹ make their learning designs explicit and, thus, eventually sharable and reusable [see e.g., “The Larnaca Declaration on Learning Design,” (Dalziel, 2015), for an account of the evolution of the field]. Besides, part of the learning design community advocates the need for computer-interpretable representations of learning designs, in order to enable the automatic configuration of technology-enhanced learning environments according to higher level design decisions made by practitioners (see e.g., Dalziel, 2003; Koper and Olivier, 2004). All in all, training in learning design has been proposed by several authors as a necessary investment to help practitioners improve teaching quality, innovate in their practice, adapt it to contextual changes, and incorporate technology-enhanced educational materials and learning platforms (McKenney et al., 2015; Svihla et al., 2015; Bennett et al., 2017).

Despite all the accumulated research evidence about the benefits of learning design, as well as the wealth of existing support tools and a few exceptions of institutional adoption cases (Rienties et al., 2017), learning design proposals present challenges. Recent studies aimed at better understanding how practitioners work are shedding light on potential reasons for challenges. Bennett et al. (2017), for instance, identify the need for flexibility in learning design tooling: the way different practitioners design is influenced by their disciplines and teaching contexts (Kali et al., 2011), thus making flexible tools more likely to be adopted. Another problem is that the functionality of most learning design tools only covers part of the learning design needs and work of practitioners. Mor et al. (2013) underline the “shortage in full-cycle integration and compatibility with institutional systems,” i.e., the lack of support by learning design tooling to enable practitioners not only to make their design decisions explicit, but also to automatically implement the technological learning environments to be used by the students, for example, those based on mainstream Virtual Learning Environments (VLE, also referred as Learning Management Systems, LMS) (Caputi and Garrido, 2015). Implementation of learning designs is the process that involves instantiation of the learning tasks, described in the learning design, in a particular learning setting and VLE (Prieto et al., 2013a). This process can be described following the “recipe” metaphor. A “recipe,” as a learning design, contains all the information to prepare a culinary dish. Different cooks will then make different instantiations of the recipe depending on their setting, tools available, etc. As in learning design, when reusing the recipes cooks may slightly change the recipes to make them theirs and adapt them to the needs and characteristics of their educational situations.

For instance, LAMS (Dalziel, 2003) is considered one of the learning design tools with most complete support to different learning design tasks; however, while it allows detailed authoring and automatic configuration of a learning environment, it does not support early analysis or conceptualization, only focusing on particular pedagogies/representations (hence hindering flexibility). In the case of most other learning design tools, the coverage is even more limited, to only one task or representation. Furthermore, the professional development of practitioners needs to be practice-based, underpinned by bespoke technological environments (Persico et al., 2015). Last, but not least, several studies confirm the importance of the social perspective in learning design: there is an increasing need for practitioners to design in teams, as well as for communities of practitioners to be able to share their learning designs and experiences (Voogt et al., 2011; Bennett et al., 2017). However, current support to sharing and co-creation of designs is very limited in the landscape of learning design tooling (Hernández-Leo et al., 2011). Only a few attempts, such as LdShake (Hernández-Leo et al., 2011) and Cloudworks (Conole and Culver, 2010), enable exchange of learning designs; however they integrate limited authoring features and do not support the implementation of designs in VLEs.

In order to contribute toward overcoming the challenges mentioned above, this paper aims at advancing the state of art in the learning design technologies field by investigating whether it is possible to provide a community system for practitioners that flexibly supports the full learning design life cycle, by articulating an integrative approach that leverages a variety of learning design tools and middleware solutions. The paper presents the design-based research methodology (Peffers et al., 2007; Amiel and Reeves, 2008) followed to iteratively design and refine an Integrated Learning Design Environment (ILDE), as a community system that satisfies users' needs in terms of effectively supporting the whole learning design life cycle (from conceptualization to deployment) and the practitioners' community development. The proposed architecture for ILDE architecture is implemented to show its feasibility and that it is capable of providing the targeted functionality. Final users (educational practitioners) also experience the provided functionality both in workshop settings and in follow up implementation of the learning designs with their students. In these settings we also evaluate the extent to which ILDE is supporting the desired functionality in a useful and usable way.

Section Prior Learning Design Tools presents an overview of the learning design tools that were available previously to ILDE. Section User-Centered Design Methodology explains the design-based iterative methodology followed to build ILDE, including the outcomes from the first and intermediate phases. ILDE architecture and features are detailed in section ILDE: Iterated Version. Section Use in Practice discusses the latest round of ILDE evaluation. Finally, section Conclusions summarizes the main contributions and conclusions of the research.

Prior Learning Design Tools

As explained above, learning design research has contributed with a myriad of representations and computer tools (Persico et al., 2013; Prieto et al., 2013a; Dalziel, 2015; Celik and Magoulas, 2016) that enable making the planning of learning tasks explicit so they can be shared, reused and implemented with students (for example, through its instantiation in VLEs). Existing representations and tools are varied in pedagogical approaches and in the phases of the learning design process that they support. For example, some tools are generic or pedagogically independent in that they support the creation of learning designs compliant with any pedagogical approach. Other tools are specific as they guide practitioners in the design of learning tasks following a particular pedagogy. Also, the level of details reflected in the learning designs varies depending on the phase of the learning design process that a particular representation and tool supports (from early conceptualizations to detailed plans of learning tasks).

Tools such as Course Map, Course Features, Persona Card, Design Narrative, Factors and Concerns, Scenario, Generating the problem (Cross et al., 2012; Hernández-Leo et al., 2013; Mor, 2013) provide text-based templates devoted to early stages of the learning design process. The information that is typically reflected in these templates are conceptualizations of high-level overviews of ideas for a course or set of activities. For example, the Course Map template gives an “at a glance” view of a course across four dimensions. It enables a brief textual overview of the course activities in terms of the types of learning experiences the learner will have, how they will communicate and collaborate with educators and peers, as well as the support and guidance provided and the nature of assessments (Cross et al., 2012).

Several graphical learning design tools also support conceptualization phases in the learning design process and provide a visual representation of overall plans for learning activities. Examples of this type of learning design software tools include Compendium LD (Brasher et al., 2008), ScenEdit (Emin, 2008), IAMEL (Bottino et al., 2010), or LDTool (Agostinho, 2011). For instance, CompendiumLD uses a flexible visual interface that enables practitioners to articulate their ideas and map out the learning design as a set of learning outcomes, a sequence of activities, and indications of tasks times. The tool provides a set of icons to represent the components of learning activities. The icons can be dragged and dropped, then connected to form a map representing a learning activity (Brasher et al., 2008). IAMEL provides ad hoc representations allowing the visualization of a course or module as a sequence of activities, some of which may be optional, while others may represent alternative routes, etc. (Bottino et al., 2010).

Authoring tools in learning design are those that support the production of full-fledged definitions of learning designs so that they are ready to be implemented with learners. Web Collage and PyramidApp are cases of learning design authoring tools that are specific to the collaborative learning pedagogy (Villasclaras-Fernández et al., 2013; Manathunga and Hernández-Leo, 2018). For example, Web Collage offers a graphical user interface that is based on visual representations of collaborative learning flow patterns. These visualizations guide practitioners along the learning design process and hides the technical complexities associated to underlying computational representations of the learning designs, which enable its eventual automatic instantiation in VLEs (Villasclaras-Fernández et al., 2013).

OpenGLM (Derntl et al., 2011), CADMOS (Katsamani and Retalis, 2013), LAMS (Dalziel, 2003), and LDSE (Laurillard et al., 2013) are generic (i.e., pedagogically independent) authoring tools for learning design. They also generate computational representations for learning designs that are detailed enough to be implemented in virtual learning environments. OpenGLM and CADMOS offer an activity-based visual modeling metaphor and designer-friendly terminology on top of the internal computational representation used to facilitate the authoring process. OpenGLM, for example, offers the high-level abstraction and representation elements prominently, while the low-level details are “disguised” behind wizard and detailed property pages.

Finally, some learning design tools are focused on supporting sharing and collaboration between practitioners. SyncrLD (Nicolaescu et al., 2013) enables synchronous co-editing of learning designs. Martinez-Maldonado et al. (2017) proposes a multi-surface approach to support face-to-face collaboration between practitioners when designing for learning. LdShake and Cloudworks are social networking platforms for sharing and discussing learning designs (Cross et al., 2012; Hernández-Leo et al., 2014b). Cloudworks enable users to view and follow recent activity on a particular discussion, or of a particular person. LdShake handles different levels of access rights to the learning designs, and a flexible approach for their exploration and reuse.

The field of learning design is therefore rich in the variety of its tools. However, each tool only covers limited functionality. Moreover, even if existing learning authoring tools provide a computational representation of the learning designs, their automatic implementation in learning environments is not straightforward. There is a shortage of full-lifecycle integration and compatibility with mainstream VLEs. The Integrated Learning Design Environment aims at providing a flexible community system that supports different approaches to learning design, collaboration and sharing, across the different stages of the learning design life cycle.

User-centered Design Methodology

The nature of the aim of this research, addressing practical barriers and needs, calls for a design-based research approach framing an iterative process of design, development and evaluation with continual interactions between practitioners (users) and researchers (Peffers et al., 2007; Amiel and Reeves, 2008). This user-centered approach is intended to ensure the desired flexibility in terms of practitioner approaches and contexts. In this section, we describe the iterative phases followed in our approach, and summarize the intermediate outcomes that led to the current design of ILDE.

Design-Based Research Iterations

We have followed Design-Based Research iterations that involve users in the definition of the use cases, aim at understanding broad interest by practitioners not involved in the definition of use cases, and iteratively evaluate and refine a prototype as it is used in practice. Figure 1 shows the three iterative phases followed, summarizing its objective, techniques and the informants involved.

FIGURE 1

Figure 1. Design-based research iteration phases. Section 2.2 corresponds to “Use cases iteration, early feedback,” Section 2.3 to “Broad interest in use cases,” Section 2.4 to “Testing in practice,” and Section 4 to “Use in practice.”

ILDE has been developed within the context of the METIS EU-funded project. Several groups of practitioners (user groups, from now on) were involved in this project, representing three educational sectors: Higher Education, Vocational Training and Adult Education. A total of five educational institutions were involved, based in different European countries (Spain, Greece, United Kingdom). In addition, practitioners from diverse context and educational sectors were approached at intermediate stages of the research to gain a wider understanding of interests beyond the project user groups. We followed a pattern of deeper inquiry with fewer participants from three of the institutions (one per sector) in the first two phases (A1, B1/B2 in Figure 1), later extending participation to understand wider interest in the use cases and in the evaluation of ILDE in the last phase (B2, C1/C2). The five educational institutions involved were:

- A distance university, characterized by a multidisciplinary team approach to course development, with university professors involved in the design of such courses.

- A traditional face-to-face university (only involved in the evaluative phase of the research, phase C in Figure 1).

- A vocational training center that offers specialized training to address the evolving labor potential of the market and the needs of the per-sons who want to upgrade their qualifications.

- A non-profit association for adult education that is dedicated to non-formal training of lifelong learners, especially socially excluded ones (e.g., people coming from school failure or lacking basic education). Association volunteers are in charge of designing the training activities, considering learners' preferences.

- A public organization in charge of regional adult education (only involved in the evaluative phase of the research).

The first phase (A, in Figure 1) was focused on iterating the use cases and requirements for the solution, by starting conversations with users (A1). High-level use cases were formulated in a way that represented stakeholders' goals, as recommended in systems engineering for some contexts (Peffers et al., 2007). To facilitate the discussions, a preliminary scenario and set of use cases addressing the identified needs were initially proposed (Antón, 1996; Glinz, 2000). The discussions were elicited using several data gathering instruments, including online questionnaires and inter-views, as well as face-to-face meetings (Hernández-Leo et al., 2013). The analysis of this information led to a refined list of use cases for ILDE and a set of (prioritized) requirements and tooling (see main results in section 2.2, Figure 1).

To achieve further understanding about interest in the use cases beyond the METIS user groups, the refined list of use cases was published for wider scrutiny in the form of a questionnaire (A2 in Figure 1). The questionnaire was distributed to practitioners at diverse educational levels, inquiring about the frequency of appearance and desirability, in their own institutions, of the learning design use cases defined (findings in section 2.3, Figure 1).

In phase B (Figure 1) preliminary pilots were carried out to test the first ILDE prototype in practice (Hernández-Leo et al., 2014a). The pilots consisted in training workshops with 41 practitioners from METIS user groups, which included hands-on practice with ILDE. After the workshops, five participants were monitored in their use of ILDE after the workshop, which included the actual application or enactment of ILDE-created learning designs with their students. At the end of the pilots, ILDE was evaluated both by the workshop participants and those monitored beyond the workshops. The evaluation was based on an adaptation of the Technology Acceptance Model (TAM) (Davis, 1989; Chuttur, 2009; Persico et al., 2014; Pozzi et al., 2015), in combination with data tracked by the system and other data collected about the 5 levels of Guskey's model (Guskey, 2002). The main aim of this preliminary evaluation was to test a first version of ILDE, understand the perceptions of users and identify implications for improving it (see main results in section 2.4, Figure 1). Section 3 (Figure 1) is devoted to the refined version of ILDE resulting from this evaluation.

Finally, Phase C (Figure 1) evaluated the refined ILDE proto-type in a new round of training workshops. This iteration involved 107 practitioners from five institutions. Again, a subgroup of 13 participants was monitored in their use of ILDE and ILDE-created learning designs when deployed into VLEs and used with their students (see section 4, Figure 1).

Use Cases Iteration, Early Feedback

The first Design-Based Research phase was focused on iterating the use cases for ILDE from the perspective of the stakeholders' goals (phase A in Figure 1). Starting from the vision of an ILDE able to support the complete learning design life cycle using diverse tools in the context of teacher communities, a preliminary set of use cases was defined. These preliminary Use Cases (pUC) were:

pUC1 Produce a learning design, choosing a design tool among several available

pUC2 Co-produce a learning design with other practitioners

pUC3 Instantiate a learning design (associate a design to specific students and tools)

pUC4 Deploy a learning design (automatically do the technological set-up) into a chosen VLE

pUC5 Share a learning design

pUC6 Provide feedback and reflections on a learning design

pUC7. Explore learning designs, implementations and feedback

The combination of these use cases sketched a preliminary generic scenario in which practitioners within an institution author or co-author learning designs following a desired pedagogy, share them with other practitioners in their community, and implement them in VLEs for enactment with students.

The inquiry was then oriented toward understanding how the pUCs should be refined to better satisfy user groups' needs and what types of learning design tooling and VLEs should be prioritized for integration in ILDE (A1 in Figure 1). Initial data about contextual characteristics and relevance of pUCs to each user group context were collected using questionnaires. The analysis of these data was complemented with an activity focused on extending and particularizing the generic scenario into meaningful specific scenarios for user group contexts. All the above was used as the basis to unfold deeper conversations with user groups in interviews and in a face-to-face meeting.

The findings from this meeting indicated that informants from the three educational sectors were interested in using a variety of learning design tools supporting not only the authoring of detailed learning designs that are ready for implementation, but also of existing learning designs in previous stages of analysis and conceptualization. These early conceptualization actions required by user groups included: the analysis of the context in which learning designs were to be applied, the elaboration of sketches of initial design ideas, and schemes for course structures. This finding led to a refinement of pUC1 into UC1, UC2, and UC3 (Table 1), and to consider the integration of conceptualization tools (besides the authoring tools more focused on making learning designs implementation-ready) in ILDE. Moreover, the three user groups showed a high interest in being able to (co-)create learning designs by reusing (duplicating and refining) existing ones (additional use case UC4). This practice was preferred to creating new learning designs from scratch, when possible. Support for the whole life cycle (including instantiation and deployment, pUC3 and pUC4) was expected to simplify processes in the three institutions. Yet, the difference between the terms “instantiation” and “deployment” was unclear to user groups, and therefore both terms were merged and replaced by “implementation” in UC5. Moodle (different versions) was the preferred target VLE for the three user groups, as they were already using it.

TABLE 1

Table 1. List of refined use cases representing stakeholders' goals after iteration A.

We also found that the relevance of the co-production and sharing use cases depends on the collaboration culture of the institution/sector. While collaboration, sharing and reuse turned out to be critical in all three user groups, their applicability seemed only limited to purposes of collective awareness (see reformulations of pUC2 and pUC5 as UC6, UC7 and UC8). Yet, the three institutions indicated the annotation of learning designs by diverse actors as an essential feature (including also students –Adult Education–, and managers –HE, VT–), especially after applying the learning design with students, giving suggestions for (re)design (see UC9 and UC10 as refinements of pU6). Finally, enabling the exploration of learning designs and their comments (about the learning designs and their implementations with students) was regarded as very useful, especially in the cases of Adult Education and Higher Education (refinements of pUC7 as UC11, UC12, and UC13).

The refined use cases cover support for the whole life cycle of learning design, considering practitioners as the users (designers). If situated in overarching processes used in professional instructional design pipelines, such as ADDIE (Branch, 2009), UC1 and UC2 relate to Analysis phases, UC3 and UC7 cover both Design and Development phases (however, note that the aim in ILDE is that development by programmers is not needed), UC5 partly covers the Implementation phase specified by ADDIE but aspects of this phase also happen outside ILDE (e.g., training of teachers and students in the tools considered in the learning design) and, finally, UC9, UC10, and UC4 enable commenting and iterating learning designs in the Evaluation phase.

Broad Interest in Use Cases

To understand to what extent ILDE use cases were relevant to a wide variety of contexts (educational institutions and sector), beyond the user groups participating in the previous phase, a questionnaire was distributed to collect data about the frequency and desirability of the use cases in other institutions (A2 in Figure 1). The questionnaire was distributed, in Italian, German and Spanish, as an online form among contacts of the Metis project partners, aiming to cover the different educational levels and also different roles of respondents. The form was introduced by a text about its aims, and included a short introductory video that showed the first ILDE prototype and explained the use cases. The participants were asked to provide background information and were then presented with descriptions of the 13 use cases, and were asked to provide their opinion about (1) to what degree the use case is happening in their institution (scale from 1 = never to 5 = all the time) and (2) how desirable the use case is for their institution (scale from 1 = very undesirable to 5 = very desirable). For each use case they were offered an (optional) textbox for comments on their rating. The aim of collecting these responses was not to enable rigorous statistical testing, but rather to reveal tendencies and discrepancies of desirability and actual occurrence of the use cases within educational institutions. In total, N = 89 participants from 14 different countries completed the questionnaire. Of those, 49 were from Italy and 18 from Spain. Most of the participants were teachers, researchers and managers in higher or secondary education institutions.

Figure 2 shows that the frequency of appearance of ILDE use cases at participants' institutions ranged between a minimum mean rating of 1.99 for the exploration of duplicates of existing learning designs (UC12), and a maximum mean rating of 3.4 for the conceptualization of learning designs (UC2). This indicates that the use cases ILDE offered were happening rather sparsely. In contrast, they were generally regarded as highly desirable. This finding reinforces the early feedback from user groups and shows an arguably broad high interest in the use cases for ILDE (A1 in Figure 1). Qualitative comments indicate that the use cases are not happening as often as desirable partly because of the current cultures in the organizations (not formally encouraging learning design or collaborative work), and partly because of a lack of satisfactory tool support. The average mean desirability was 4.21 across all use cases, with the most desirable one being the sharing of learning designs for other educators' awareness within the institution (UC6; mean = 4.44).

FIGURE 2

Figure 2. Frequency and desirability of ILDE use cases at questionnaire respondents' institutions (mean ratings, N = 89).

Testing in Practice

The preparation and running of a first round of workshops using the initial prototype version of ILDE (phase B in Figure 1) was not only aimed to test the prototype in practice, but also to further confirm and refine the user groups' requirements. Participants included 13 Higher Education, 16 Vocational Training and 12 Adult Education practitioners, for a total of 41. After the workshops, five volunteers among the workshop participants agreed be monitored and interviewed about their further use of ILDE beyond the workshop and the application of ILDE-created learning designs with their students during daily class activities.

Overall, this evaluation showed that ILDE was positively perceived in all three contexts. Results showed positive trends in the ratings to TAM's quantitative indicators, especially in terms of its usefulness. Remarkably, sharing and exploration functions were one of the most appreciated aspects of ILDE. This confirmed the need to share and exchange ideas with others, suggesting a general willingness of people to be part of a community in order to obtain support and inspiration. Yet, the evaluation also pointed to ease-of-use issues to be solved, and the need for additional features. These aspects were considered in the development of the refined ILDE prototype (section 3, Figure 1).

In particular, ease-of-use issues were mainly focused on navigation and terminology, given that ILDE is an environment that integrates multiple tools that have been developed independently. Decisions for refinements included the addition of filters to assist browsing through shared learning designs, change in menu organization, and a contextual help with explanatory tooltips about terminology and tooling.

No particular need to customize ILDE on the basis of the three different contexts was observed (besides language translation of tools). Only a specific additional conceptualization template (aligned with local regulations) was identified as a requirement in the Vocational Training case. However, interest in availability of more learning design tools, than those integrated in the first prototype, was present in the three contexts. The three institutions expressed the need for exploring alternative authoring experiences and supported pedagogies. Additional observed requirements included the need for licensing learning designs and features for awareness of other community members' activity. All these refined requirements and features led to the iterated ILDE prototype, released for phase C (Figure 1), described below.

ILDE: Iterated Version

This section presents the architecture of ILDE as well as the result from its development to show its feasibility and that the targeted functionality is achieved. A demonstration server and devoted ILDE installation for diverse communities is available at http://ilde.upf.edu/about.

ILDE Architecture

Figure 3 shows ILDE's logical architecture. It is a three-tier architecture, with presentation, services and data layers. The figure shows the subsystems composing the architecture, which articulates an integrative approach that leverages a variety of learning design tools, extends their functions and enables deployment into VLEs—considering also the use of learning tools external to the VLEs. Integration is achieved through: (1) a Web-based portal providing a common user interface, (2) Restful Application Programming Interfaces (API) to support communication between the portal and the tools provided as services, (3) a data layer separated from the services, (4) adapters transforming the generic portal calls to the specific interface of each particular tool, and (5) a devoted subsystem to manage deployment of learning designs into the environments to be accessed by students (VLEs and other tools).

FIGURE 3

Figure 3. Logical architecture for ILDE.

The selection of tools for integration was done considering both available open-source or open-API learning design tools and the requirements identified in the first Design-Based Research phase (section 2.2, Figure 1). Learning design tools integrated should cover the identified use cases for ILDE (e.g., both conceptualization and authoring functions) and VLEs prioritized were those already adopted by the educational centers of the user groups. As for the portal, ILDE extends LdShake (Hernández-Leo et al., 2011, 2014b), a community platform that offer social services, provides an integrated exploration of learning designs and control access to learning design tools.

When the tools are learning design conceptualization templates, they are visualized using a web-based editor as well as by Google Drive applications, thus enabling synchronous co-edition. When they are web editors (e.g., Web Collage; Villasclaras-Fernández et al., 2013), they are visually embedded in the portal web interface. In the case of desktop tools (e.g., OpenGLM; Derntl et al., 2011), learning designs can be either uploaded to ILDE and used (shared, commented, etc.) in the context of the community, both directly in the ILDE user interface or by extending the tool with a feature that connects to ILDE API and enables search in, import from and export of learning designs to ILDE. Both for conceptualization and authoring tasks, ILDE does not impose the usage of one particular tool. Rather, its Restful API allows the integration of new authoring tools also being integrated as of this writing), thus enlarging the set of available choices for practitioners. The enabled integration, through the use of the API and having a data layer separated from the services, avoid the need of double sign-on. It is worth noting that some of the conceptualization templates, the use of Google Drive applications, and the OpenGLM authoring tools were integrated after considering the feedback from the preliminary pilots (phase C in Figure 1, section 2.4).

As a result of this integrative approach, ILDE manages arbitrary file types (formats) for the learning designs, some of which can be used further for implementation—e.g., IMS Learning Design (Koper and Olivier, 2004) compliant packages.

To enable the implementation (instantiation and deployment) of learning designs in mainstream VLEs, ILDE integrates GLUE!-PS and GLUE! Middlewares (Alario-Hoyos et al., 2012; Prieto et al., 2013a). GLUE!-PS uses a set of adapters for translating learning designs, represented with the computational languages of the different authoring tools available in ILDE, into a common internal representation or “lingua franca.” Then, in a second step, GLUE!-PS automatically sets up and configures the target VLE so that it reflects the contents of the implemented learning design. It also manages, by interacting with the GLUE! middleware, all the required third-party learning tools that need to be integrated within the VLE for supporting learning activities. The integration of additional authoring tools and VLEs is also possible, through the development of new GLUE!-PS adapters, without further modifications in the ILDE architecture. Interestingly, along the iterative refinements of ILDE, additional Moodle versions were supported, so as to cover the emerging needs of the user groups; this was achieved through the development of a new GLUE!-PS VLE adapter.

From the perspective of institutional VLEs, the automatic implementation of ILDE-created learning designs in the VLEs may require configuration steps that imply the involvement of VLE administrators. Such configuration steps depend on the technical characteristics and specific configuration of the VLE of interest. The current ILDE version supports the implementation of learning designs into Moodle, MediaWiki and Blogger servers acting as VLEs. Thus, for instance, using the implementation features of ILDE with a Moodle server may require the installation of a plug-in available in the ILDE software distribution. However, to interact with a MediaWiki system, the ILDE simply requires the activation of MediaWiki's built-in web API.

Finally, when the tooling available for integration did not cover aspects of the use cases (e.g., registration and management of VLEs), extensions were developed as new services to be provided through the portal. Moreover, the features of LdShake, of some of the integrated tools, and of GLUE!-PS have been refined to provide an integrated view of the flow of tasks that need to be completed along the learning design life cycle, thus trying to minimize the impact of the transitions among different ILDE components.

The physical architecture used in current deployments of ILDE is shown in Figure 4. The portal and all components in the services layer represent the core of ILDE, which is located in three dedicated servers (databases, implementation services, and portal with the remainder of the services) for performance reasons. Depending on the case, integrated tools can run in external machines (maintained by tool providers) and the VLEs are typically hosted in the educational centers.

FIGURE 4

Figure 4. Physical architecture of various ILDE installations available at http://ilde.upf.edu/about.

Conceptualization and Authoring Support

The implemented architecture offers practitioners access to multiple conceptualization and authoring functions in the context of a single environment. The different artifacts created with any of these functions are equally managed using a generic term that refers to any type of learning design solution (“LdS,” as coined in LdShake) (Hernández-Leo et al., 2011).

Integrated conceptualization templates and tools have different aims (use cases UC1, UC2). Some of them (e.g., Persona Card, Factors, and Concerns) help educators analyze the characteristics of the learning contexts (e.g., audience, constraints of the educational setting, potential obstacles) in which they will apply the learning designs to be created (Cross et al., 2012). For example, Figure 5A shows a screenshot of the Persona Card template integrated in LdShake.

FIGURE 5

Figure 5. (A) Access to conceptualization functions (UC1, UC2) provided by diverse tools, “Persona Card” selected, (B) Access to authoring functions (UC3), Web Collage “selected”.

The Persona Card (Mor et al., 2007; Nielsen, 2013) is a tool for reflecting on the expected audience of the learning design, which includes an understanding of the profile, expectations and potential obstacles for students. Other conceptualization templates and tools integrated allow practitioners to sketch preliminary ideas for their learning designs [e.g., Course Features, Course Map, CompendiumLD (Brasher et al., 2008)]. Besides, users can upload pictures of learning designs created in a board or in paper and edit open (free-form) conceptualizations.

Authoring is the process of producing full-fledged definitions of learning designs so that they are ready to be applied with particular groups of learners (i.e., with descriptions of tasks, supporting resources, etc.; see use case UC3). Authoring functions in ILDE are currently provided by the integrated Web Collage (Villasclaras-Fernández et al., 2013) and OpenGLM (Derntl et al., 2011) editors. If the authored learning designs are represented computationally, the technical setup of the VLE where the learning design is to be implemented can be done automatically (see implementation functions in section Broad Interest in Use Cases). Different authoring tools may use different learning design representations, pedagogical and authoring approaches or may require different levels of expertise. As explained above, Web Collage is specifically devoted to the design of collaborative learning activity flows, using visual representations of pedagogical patterns in a way that guides authoring and hides the technical complexities associated to the computational representations of the created learning designs. Figure 5B illustrates how Web Collage is integrated in the common web portal for the authoring of a “Pyramid pattern” learning design. OpenGLM uses a visual representation based on diagrams of interconnected activities that lends itself to authoring with diverse pedagogies.

Implementation Support

Full-fledged learning designs that use computational representations can be automatically deployed to VLEs thanks to the implementation functions provided by ILDE (UC5). Deployment involves configuring and setting up of learning platform resources according to what is expressed in a learning design (e.g., generating a ready-to-use Moodle course with the activities and groupings specified in the learning design). Practitioners can add and configure their credentials for access to the VLEs where they would like to implement their learning designs (Figure 6A). To implement a learning design, a practitioner needs to first select the learning design (Figure 6B) and then indicate in which of the VLEs configured she would like to do the implementation. As a result of these actions, ILDE knows which are the specific students enrolled in the VLE and which learning tools are available in the VLE and in external third-party web 2.0 tools that are integrated in the VLE using GLUE! (Alario-Hoyos et al., 2012). The next step is to bind the students and tools to the corresponding elements specified in the learning designs (assign students to activities and groups, and indicate which tools will be use to support each activity). This can be done using GLUE!-PS user interface (Figure 6C). Finally, as explained in section Design-Based Research Iterations, GLUE!-PS automatically sets up and configures the target VLE so that it reflects the contents of the implemented learning design (Figure 6D). Once the implementation phase is over, practitioners can optionally use the VLE's own authoring capabilities to fine-tuned the learning design automatically deployed by ILDE.

FIGURE 6

Figure 6. Implementation functions (UC5), in Menu New LdS -> Implement, (A) add/configure your VLE(s), (B) select learning design to implement, (C) edit implementation, (D) outcome of the implementation function (learning situation automatically created in a specific VLE).

Sharing and Exploration Support

ILDE offers sharing and exploration functions to practitioners, adopting and extending the features provided by the LdShake platform for sharing and co-edition (Hernández-Leo et al., 2011, 2014b). Practitioners can share their learning designs created with any of the conceptualization and authoring functions in ILDE with other members of the community (Figure 7). Sharing can be done with either view or edit rights, so that other practitioners can be aware of the learning designs created in the community (UC6), eventually reuse them (UC7) or participate in the co-edition of the learning designs (UC8). By default, learning designs are shared with all of the members in a community with view rights; but this can be changed by the user. The selection of people that have edit rights for a learning design can be done individually or by groups of community members. To facilitate this, LdShake enables the creation of groups by defining a name for a group and indicating the selection of members. Moreover, as requested by user groups in our design-based research process (section 2.4, Figure 1), for each member in the community, it is possible to see the list of learning designs and implementations in which they are involved (UC11, UC13).

FIGURE 7

Figure 7. Sharing with others with editing rights, all members of the community can view the learning design. The list of members in the community and creation of groups is available through the LdShakers menu (UC6, UC7, UC8).

Exploration of learning designs (accessible at least with view rights) can be also done by means of browsing functions. Browsing can make use of filters based on the learning design template or tool used for their creation, as well as of filters based on tags (Figure 8A). Tags are indicated by practitioners when creating the learning designs (see the “Tags” option in Figures 4, 5) and can be organized in different categories (discipline, pedagogical approach, free form tags). The list of tags available for filtering includes numbers indicating how many learning designs are categorized using such tags. Practitioners can add comments to any learning design in the environment (tab “Info and Comments”) in order to, for example, document their reflections after implementing the learning designs with students, or to provide formative comments to learning designs created by others (UC9, U10).

FIGURE 8

Figure 8. (A) Browsing by tool and tags (UC11, UC13) (Menu: Browse LdS). (B) Viewing a learning design (in this case created with CompendiumLD) that can be (co-)edited, duplicated, commented (UC4, UC9, UC10) (list of designs that can be edited accessed through in Menu: My LdS).

Learning designs can be duplicated for refinement and reuse (UC4) (see option “Duplicate” in Figure 8B). Existing duplications of a learning can be also browsed (U12) allowing practitioners to explore how a learning design has been refined for different contexts (Chacón-Pérez et al., 2016). If practitioners have edit access rights to a learning design, they can also participate in their co-edition (“Edit” in Figure 8B). Moreover, practitioners can select a Creative Commons license for their learning designs and “Publish” them to obtain a unique URL that can be used outside ILDE to view the learning designs. ILDE also integrates a contextual help that offers short explanations of the menu options.

Use in Practice

The demonstration of ILDE features in the previous section (and the functional system available online) shows that the development of the envisaged environment is possible. On the other hand, the design of ILDE has been done considering user needs in iterations. Yet, this section (responding to phase C in Figure 1) presents the methodology and results of the ILDE evaluation with users, aimed to understand to what extent the provided system is able to support the whole learning design cycle with community support in a usable and useful way.

Methodology

In analogy with the preliminary evaluation, in order to assess to what extent the ILDE achieved its goals, the TAM and its subsequent evolutions (in particular, TAM2) were chosen (Davis, 1989; Venkatesh and Davis, 2000; Chuttur, 2009; Pozzi et al., 2015). This choice was based on a survey of the existing models to evaluate and/or predict technology acceptance (Persico et al., 2014; Pozzi et al., 2015).

Apart from the users' perceptions, it was also acknowledged that the information provided by users during the application of this model can be complemented with other data, such as data automatically tracked by the system itself. ILDE tracks the total number of times each user activates each function, and collects additional information such as the number of learning designs started by participants, including those that haven't been saved. In the context of the workshops a user could also be a group of users working together with a single user identifier during the workshops. Thus, the “subjective data” about the users' perceptions were then complemented with more “objective data” about what happened when users engaged with ILDE. This information was used mainly to assess the trustworthiness of users' opinions.

Participants were recruited, using a convenience sampling technique (practitioners voluntarily choosing to participate in the trainings), at the five educational institutions described in section 2.1 (Figure 1). Participants' distribution per sectors was 34 Higher Education, 31 Vocational Training, 34 Adult Education and 8 others. The backgrounds they declared included the educational science field (58%), educational technology (18%), and computer science (9%). Their expertise in the learning design field was rather basic. Most of our respondents declared they are beginners (66%), intermediate (20%), experts (4%) and the rest did not respond. Most of them (98.3%) did not know any of the LD tools integrated in ILDE. As far as motivation is concerned, respondents had to rate their agreement about a set of statements related to motivation. Rating was given using a Likert scale (from 1 = min to 5 = max). Respondents declared they attended the workshop out of interest in learning design at a personal level (overall mean = 4.1; st. dev. = 0.79), or because they think their professionalism benefits from that (overall mean = 4.2 st. dev. = 0.92); they are also rather confident that their institutions can benefit from their participation at the workshop (overall mean = 4.1 st. dev. = 0.91). They seem to care less about the fact that people they work with might consider it important (overall mean = 3.7 st. dev. = 1.0). Enactors were 4 vocational teachers, 5 adult educators, 4 higher educators.

The data about the users' perceptions were collected through questionnaires (after the workshops) and interviews (after the enactment): in total we had 107 questionnaires filled in by workshop participants, and 13 interviews from learning design enactors (that is, workshop participants who deployed their learning designs into VLEs and use them in real contexts).

Figure 9 shows the overall evaluation methodology adopted at this stage. The achievement of the main research objective addressed by this paper can be evaluated by answering the two evaluation questions depicted in Figure 9 and already mentioned above (i.e., ILDE support for the whole learning design cycle, and ILDE support for communities of practitioners). In order to answer those two questions, it is necessary to obtain a set of high-level indicators, clustered in three categories: “perceived easy-of-use” and “perceived usefulness” (derived from the TAM/TAM2 model), and the “actual use” category (based on tracked data). The high-level indicators are obtained by aggregating a set of low-level indicators referred to the different ILDE functions, contained in the responses to questionnaires, interview transcripts and tracked data from ILDE logs.

FIGURE 9

Figure 9. Schema of the overall evaluation approach.

The 13 interviews with enactors (that is, workshop participants who deployed their learning designs into VLEs and use them in real contexts) were aimed at collecting information about the impact of our workshops according to Guskey's model (Guskey, 2002), which encompasses 5 levels of evaluation for a training event for teachers: (1) Participants' reaction, (2) Participants' learning, (3) Organizational support and change, (4) Participants' Use of New Knowledge and Skills, (5) Student learning outcomes (students here are those who will finally benefit from the participants' training). A detailed analysis of the particular context of the face-to-face university is described in Asensio-Pérez et al. (2017). Enactors interview data in the five contexts, together with other data collected from workshop participants who did not take part to enactment, allowed us to complement and consolidate the data obtained from the TAM, thus hopefully reducing their possible bias.

Results

Similarly to the previous phase (testing in practice, see section 2.4, Figure 1), participants who filled in the questionnaires and interviews were asked to rate each different items about ease-of-use and usefulness of the ILDE's functions (using a 1–5 Likert scale, from 1 = very low to 5 = very high) and then also give qualitative opinions about them. As already mentioned, these data were complemented with the actions tracked in ILDE log files, both during the workshops and during the enactment of ILDE-created learning designs. In order to investigate the ability of the ILDE to support the whole learning design life cycle, we explored separately each of the ILDE functions related to such life cycle, namely: the Conceptualization, the Authoring and the Implementation of learning designs.

As far as Conceptualization functions are concerned, most of the tools were extensively explored by the workshop participants (data from the log files shows that participants started a total of 131 learning designs with Persona Card, 116 with Design Narrative, and more than 30 learning designs were started with Design Patterns, Factors and Concerns and Heuristic Evaluation each). This means that most of the ILDE Conceptualization functions were extensively used and this allowed us to rely on the feedback obtained as trustworthy (assuming that feedback on functions that were used too little is less trustworthy than feedback on functions that were extensively used). In an attempt to check whether there were differences in the participants' opinions coming from the three educational sectors, we also conducted statistical analyses (as a general rule, we applied the ANOVA test, but when the numerousness of one or two groups was <10, we applied a non-parametric test, i.e., Kruskal-Wallis and the Exact test). As shown in Figure 10, all the functions obtained satisfying ratings (overall mean for ease-of-use: >3.8; overall mean for usefulness: >4.0). If we look at the ratings of the most explored functions only (Course Map, Design Patterns, Design Narratives, Persona Card, Image upload and the custom Vocational Training Template in Greek, see Figure 10), overall average for ease-of-use was >3.9, and >4.2 for usefulness. This makes the evaluation results of the Conceptualization functions very satisfactory.

FIGURE 10

Figure 10. Participants' opinions about the conceptualization functions.

Besides, looking at the statistical analyses (see collection of Table A1 in Supplementary Material), even when tests could be applied, no significant differences among the contexts emerged, suggesting that the Conceptualize functions were equally accepted in all three sectors/user groups. It is interesting to note that two of the tools were explored almost exclusively in only one of the contexts (the Image upload and the custom Vocational Training Template in Greek, in the vocational training sector).

In the case of the Authoring functions (see collection of Table A2 in Supplementary Material), Web Collage was more extensively explored (66 answers and log files reports 141 learning designs started during the workshop), while OpenGLM collected far less responses (7 answers and 1 designed created by a team of users according to the log files). Hence, even if overall averages in both cases were remarkably high, we concentrate here mainly on the quantitative data about Web Collage. In particular, mean rating for ease-of-use was 3.9, while for usefulness was 4.3 (Figure 11). Besides, no statistical differences emerged across the three contexts [ease of use: F_{(2, 38.896)} = 2.948 p = 0.064; usefulness: F_{(2, 63)} = 0.690, p = 0.505], confirming that the tool was equally well accepted in all the sectors. Regarding OpenGLM, the qualitative comments indicated a positive attitude toward the tool by those participants that explored it.

FIGURE 11

Figure 11. Participants' opinions about the authoring functions.

The three Implementation functions were extensively explored: data from the logs reported 66 “Selections of learning designs for implementation,” 90 “Add VLE,” and 30 “Configure VLE” (see collection of Table A3 in Supplementary Material). Looking at Figure 12, it appears that these functions obtained very high ratings, both in terms of ease-of-use (overall mean > 4.0) and usefulness (overall mean > 4.3). Statistical tests did not provide any evidence of differences across educational sectors.

FIGURE 12

Figure 12. Participants' opinions about the implementation functions.

The ability of the ILDE to provide an adequate community support was investigated mainly through the evaluation of the Sharing and Browsing functions. Sharing functions were extensively explored (for example, data from the log files tracked 655 actions as “View someone else's LdS” and 150 as “Edit someone else's LdS), and (almost) all of them got a high number of responses (>10 responses, see collection of Table A4 in the Supplementary Material). The only exceptions were the functions “Exchange messages with other LdShakers” (actions tracked: 7) and “View duplicated LdS” (actions tracked: 0), which both got 9 responses. It is noteworthy that the “Create a LdShakers' group” were used primarily in the adult education context. All the items got very high ratings (see Figure 13); the overall mean obtained by the “Share a LdS with others” function had the lowest score (overall mean of 3.7 for ease-of-use, and 4.3 for usefulness). From the statistical tests performed, no significant differences emerged across the sectors in any of the functions.

FIGURE 13

Figure 13. Participants' opinions about the sharing functions.

Concerning the Browsing functions offered by ILDE, they were sufficiently explored to allow statistical tests, the “Browse by tags” being the function most extensively explored (log files tracked 187 actions). The “Free search” was mostly explored in the higher education sector, the “Browse by discipline” and “Browse by pedagogical approach” functions were explored primarily in the vocational training sector. Also in this case, ratings are really positive, regarding ease-of-use (overall means > 3.8) and usefulness (overall means > 4.1), as shown in Figure 14. No significant differences among educational sectors were detected by the statistical tests conducted (see collection of Table A5 in Supplementary Material).

FIGURE 14

Figure 14. Participants' opinions about the browsing functions.

From the qualitative data, we detected a general satisfaction about ILDE that reflects the good ratings presented so far. The most used words to describe the system included: “useful,” “effective,” and “potential.” One of the most appreciated elements was the ability of the system to support pedagogical reflection and planning, as well as its ability to share ideas with others and to find useful design ideas created by others. Some respondents also highlighted the added value of ILDE to allow the Implementation phase in Moodle (or other VLEs). This is a very important element from the perspective of the research conducted, as it confirms from the practitioners' perspective this distinctive feature of ILDE (the covering of the whole learning design life cycle), which represents an important advance in the field of learning design.

Overall, the quantitative data showed that ILDE was generally considered easy-to-use, but respondents mentioned several aspects as potentially improvable. This had already emerged from the first round of evaluation and the ratings regarding usability were higher at the second round, but from the analysis of the qualitative answers we understand that there is a need for further improvement; especially as far the last steps of implementation and guidance across functions. With this regards, one interesting suggestion mentions the possibility of implementing an ILDE wizard or “assistant,” to provide on demand support to the ILDE users.

As mentioned in the previous section, the data collected about the enactments referred to Guskey's model indicators and for this reason they are not directly comparable with the data collected from workshop participants. However, these data confirm a positive attitude of enactors toward the ILDE and, more generally, toward what they had learnt during the workshops. On the negative side, some enactors with low initial ICT skills pointed out that the learning curve was very steep at the beginning, then becoming more “feasible,” and expressed the intention to enact further learning designs and the belief that further enactments could be much easier. Another emerging aspect to improve had to do with language: especially in the Greek and Spanish workshops, some participants noted the existence of anglicisms in ILDE terminology (e.g., Course Map, Persona Card) that partly hindered understanding.

Finally, data collected from workshop participants who chose not to run any enactment support the hypothesis that the main reasons for this choice did not lie with ILDE usability, but rather with time (i.e., they were not able to enact due to the very short timeframe of the project).

Discussion and ILDE Refinements

Results indicate that ILDE is able to support practitioners in covering the whole learning design life cycle, providing flexibility by offering multiple approaches to conceptualization and authoring, and the possibility of deploying the learning designs in diverse VLEs. The usefulness and usability of ILDE community support was highly valued by the participants (with higher ratings for usefulness), both in terms of the sharing functions that enable co-edition, commenting, or duplication of learning designs for their reuse, and of the browsing functions that facilitate the exploration of the learning designs in the community.

The lack of significant differences in the ratings collected across the three educational sectors shows that ILDE equally fits the needs of the three contexts. This may be due to the flexibility provided by the multiple options for conceptualization, authoring and implementation and by the different levels of collaboration supported by the community functions.

The feedback emerged from this evaluation has also led to additional refinements to ILDE. Browsing functions were extended allowing users to sort learning designs by title (alphabetically) or by last edition date. Contextual help explaining the specific terminology has also been improved. ILDE is a complex tool that integrates a wide array of existing learning design tools, and its use requires a certain degree of familiarization. This was observed with the enactors and in the use of ILDE by a third-party initiative, where ILDE was used to support learning design activities in Massive Open Online Courses (MOOCs) for teachers (Garreta-Domingo et al., 2015). Evaluation data obtained during this MOOC indicate that participants' level of comfort using ILDE increases substantially once they become more familiar the system. To support these MOOCs, additional conceptualization templates were integrated (e.g., Dream, Heuristic evaluation) (Garreta-Domingo et al., 2018).

Besides, to provide more options for authoring, several steps have been already carried out for the integration of additional authoring tools. These include SyncMetaLD, a real-time collaborative learning design authoring tool (Nicolaescu et al., 2013), PyramidApp (Manathunga and Hernández-Leo, 2018), and the Design Problem Generator (Hernández-Leo et al., 2017). Further research will be needed to study whether the use of authoring tools that include also conceptualization scaffolding (e.g., Web Collage) or an ILDE feature to support the management of multi-artifact learning design projects (Chacón-Pérez et al., 2016), allow for smoother transition between phases. To enhance the support for implementation, ILDE has been refined to support duplication of implementations so that they can be adapted for different cohorts of students and the communication between GLUE!PS and LdShake has been improved in terms of performance.

Conclusions

The main contribution of this paper has been to articulate an integrative approach to flexible learning design, and to enable it by means of an extensible architecture. The developed ILDE integrates and extends a collection of learning design tools to enable collaboration between practitioners for sharing and co-editing both conceptualizations and fully-fledged authored learning designs of different kinds (mostly in the context of a single web user interface), and allowing the automatic implementation of authored learning designs into VLEs. This is a significant contribution to the learning design technologies field, since previous existing tools only covered one phase of the cycle, offer one specific approach to learning design or provide limited support for community collaboration.

ILDE is functional, several installations and its source code are open and available. Both the achieved product and its use by practitioners in training workshops and actual educational contexts (diverse educational sectors) answer our research question, by showing that it is possible to provide a community system for practitioners that flexibly supports the full learning design life cycle. Results after practical use highlight the usefulness of ILDE functions and show that the selection of the integrated design options accommodates the needs of the participants and their institutions. These results indicate that ILDE overcomes some of the challenges in learning design identified in the literature.

This work also opens directions for future research and product development. We expect that the integrative approach proposed, and the results obtained after using the developed environment by practitioners in diverse contexts, help sharpen insights into support for flexible learning design cycles. The architecture and environment provided may inspire other researchers, for example, in investigating the role of particular learning design techniques framed in the context of more holistic design processes and in combination with other design strategies and tools. This paper may also inspire other educational technology researchers and developers, helping them understand the complexity of learning design and propose better tooling. Implications include the need of guidance across learning design tools or of interoperability to support flows of learning design artifacts across multiple conceptualization and authoring tools. There are also implications for the training of practitioners; ILDE can help investigations around practice-based teaching training and delve into additional challenges in learning design. Further research should also consider the emerging body of research that focuses on the alignment of learning design elements with their effects in supporting students' learning (Rienties and Toetenel, 2016; Hernández-Leo et al., unpublished).

Ethics Statement

The present paper describes work that was conducted as part of the METIS project (http://websites.cardet.org/metis/index.php), which was funded by the EU's Lifelong Learning Programme. An ethics approval for this research was not required at the time the research was conducted as per the involved Institutions' guidelines and national regulations and written informed consent was obtained from all research participants.

Author Contributions

DH-L led the conception of the environment, the user-centered design methodology, and drafted the manuscript; JA-P contributed to the conception of the environment, the collection of data, and edited the manuscript; MD contributed to the conception of the environment, analyzed data and edited the manuscript; FP and DP conceived the evaluation methodology based on TAM, analyzed data and edited the manuscript; JC contributed to the conception of the environment and edited the manuscript; LP contributed to the conception of the environment, designed figures and edited the manuscript.

Funding

This work has been partially funded by the EC, EACEA, METIS Project 531262-LLP-2012-ES-KA3-KA3MP, with additional support by TIN2014-53199-C3-3-R, TIN2017-85179-C3-3-R, MDM-2015-0502, RecerCaixa CoT, TIN2014-53199-C3-2-R, TIN2017-85179-C3-2-R, VA082U16 and a Marie Curie Fellowship (MIOCTI, FP7-PEOPLE-2012-IEF project no. 327384).

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors acknowledge the contributions from other project members, including the developers: P. Abenia, J. Hoyos and R. Uppal.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fict.2018.00009/full#supplementary-material

References

Agostinho, S. (2011). The use of a visual learning design representation to support the design process of teaching in higher education. Aust. J. Educ. Technol. 27, 961–978. doi: 10.14742/ajet.923

CrossRef Full Text | Google Scholar

Alario-Hoyos, C., Bote-Lorenzo, M. L., Gómez-Sánchez, E., Asensio-Pérez, J. I., Vega-Gorgojo, G., and Ruiz-Calleja, A. (2012). GLUE!: an architecture for the integration of external tools in virtual learning environments. Comput. Educ. 60, 122–137. doi: 10.1016/j.compedu.2012.08.010

CrossRef Full Text | Google Scholar

Amiel, T., and Reeves, T. C. (2008). Design-based research and educational technology: rethinking technology and the research agenda. Educ. Technol. Soc. 11, 29–40.

Google Scholar

Antón, A. I. (1996). “Goal-based requirements analysis,” in Proceedings of the Second International Conference on Requirements Engineering (Colorado Springs, CO), 136–144.

Asensio-Pérez, J. I., Dimitriadis, Y., Pozzi, F., Hernández-Leo, D., Prieto, L. P., Persico, D., et al. (2017). Towards teaching as design: exploring the interplay between full-lifecycle learning design tooling and Teacher Professional Development. Comp. Educ. 114, 92–116. doi: 10.1016/j.compedu.2017.06.011

CrossRef Full Text | Google Scholar

Bennett, S., Agostinho, S., and Lockyer, L. (2017). The process of designing for learning: understanding university teachers' design work. Educ. Technol. Res. Dev. 65, 125–145. doi: 10.1007/s11423-016-9469-y

CrossRef Full Text | Google Scholar

Bottino, R. M., Ott, M., and Tavella, M. (2010). Empowering the design and the sharing of learning plans by means of net technologies: the IAMEL system. Knowl. Manage. Inf. Syst. E-Learn. Sustainability Res. 111, 336–342. doi: 10.1007/978-3-642-16318-0_38

CrossRef Full Text | Google Scholar

Branch, R. M. (2009). Instructional Design: The ADDIE Approach, Vol. 722. New York, NY: Springer Science & Business Media.

Google Scholar

Brasher, A., Conole, G., Cross, S., Weller, M., Clark, P., and White, J. (2008). CompendiumLD – A Tool for Effective, Efficient and Creative Learning Design. Available online at: http://oro.open.ac.uk/15783/2/F16FE490.pdf

Caputi, V., and Garrido, A. (2015). Student-oriented planning of e-learning contents for Moodle. J. Netw. Comp. Appl. 53, 115–127. doi: 10.1016/j.jnca.2015.04.001

CrossRef Full Text | Google Scholar

Celik, D., and Magoulas, G. D. (2016). “A review, timeline, and categorization of learning design tools,” in International Conference on Web-Based Learning (Rome: Springer International Publishing), 3–13.

Google Scholar

Chacón-Pérez, J., Hernández-Leo, D., Mor, Y., and Asensio-Pérez, J. I. (2016). “User-centered design: supporting learning designs' versioning in a community platform,” in The Future of Ubiquitous Learning, eds B. Gros, Kinshuk and M. Maina (Berlin; Heidelberg: Springer), 153–170.

Chuttur, M. Y. (2009). Overview of the Technology Acceptance Model: Origins, Developments and Future Directions. Working Papers on Information Systems, Indiana University, 9–37.

Conole, G. (2012). Designing for Learning in an Open World. Berlin; Heidelberg: Springer.

Google Scholar

Conole, G., and Culver, J. (2010). The design of Cloudworks: applying social networking practice to foster the exchange of learning and teaching ideas and designs. Comp. Educ. 54, 679–692. doi: 10.1016/j.compedu.2009.09.013

CrossRef Full Text | Google Scholar

Cross, S., Galley, R., Brasher, A., and Weller, M. (2012). OULDI-JISC Project Evaluation Report: The Impact of New Curriulum Design Tools and Approaches on Institutional Process and Design Cultures. Available online at: http://oro.open.ac.uk/34140/

Dalziel, J. (2003). “Implementing learning design: the learning activity management system (LAMS),” in ASCILITE: 20th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (Wollongong, NSW: Australasian Society for Computers in Learning in Tertiary Education).

Google Scholar

Dalziel, J. (2015). “Reflections on the Art and Science of Learning Design and the Larnaca Declaration,” in The Art & Science of Learning Design. Technology Enhanced Learning, eds M. Maina, B. Craft, and Y. Mor (Rotterdam: SensePublishers), 3–14.

Google Scholar

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13, 319–340. doi: 10.2307/249008

CrossRef Full Text | Google Scholar

Derntl, M., Neumann, S., and Oberhuemer, P. (2011). “Propelling standards-based sharing and reuse in instructional modeling communities: the Open Graphical Learning Modeler (OpenGLM),” in 11th IEEE International Conference on Advanced Learning Technologies (ICALT), 2011 (Athens, GA: IEEE), 431–435.

Google Scholar

Emin, V. (2008). “ScenEdit: an authoring environment to design learning scenarios,” in 2018 8th IEEE International Conference on Advanced Learning Technologies (ICALT) (Santander: IEEE), 737–738.

Google Scholar

Garreta-Domingo, M., Hernández-Leo, D., Mor, Y., and Sloep, P. (2015). “Teachers' perceptions about the HANDSON MOOC: a learning design studio case,” in Design for Teaching and Learning in a Networked World (Toledo: Springer), 420–427.

Google Scholar

Garreta-Domingo, M., Hernández-Leo, D., and Sloep, P. B. (2018). Evaluation to support learning design: Lessons learned in a teacher training MOOC. Aust. J. Educ. Technol. 34, 56–77. doi: 10.14742/ajet.3768

CrossRef Full Text | Google Scholar

Glinz, M. (2000). “Improving the quality of requirements with scenarios,” in Proceedings of the Second World Congress on Software Quality (Tokyo), 55–60.

Google Scholar

Guskey, T. R. (2002). Professional development and teacher change. Teachers Teach. 8, 381–391. doi: 10.1080/135406002100000512

CrossRef Full Text | Google Scholar

Hernández-Leo, D., Agostinho, S., Beardsley, M., Bennett, S., and Lockyer, L. (2017). “Helping teachers to think about their design problem: a pilot study to stimulate design thinking,” 9th annual International Conference on Education and New Learning Technologies EDULEARN 17 (Barcelona).

Hernández-Leo, D., Asensio-Pérez, J. I., Derntl, M., Prieto, L. P., and Chacón, J. (2014a). “ILDE: community environment for conceptualizing, authoring and deploying learning activities,” in Proceedings of the 9th European Conference on Technology Enhanced Learning Open Learning and Teaching in Educational Communities (Graz: Springer), 490–493.

Hernández-Leo, D., Chacón, J., Prieto, J. P., Asensio-Pérez, J. I., and Derntl, M. (2013). “Towards an integrated learning design environment,” in Proceedings of the 8th European Conference on Technology Enhanced Learning, LNCS (Paphos: Springer), 448–453.

Hernández-Leo, D., Harrer, A., Dodero, J. M., Asensio-Pérez, J. I., and Burgos, D. (2007). A framework for the conceptualization of approaches to “Create-by-Reuse” of learning design solutions. J. Univ. Comp. Sci. 13, 991–1001. doi: 10.3217/jucs-013-07-0991

CrossRef Full Text | Google Scholar

Hernández-Leo, D., Moreno, P., Chacón, J., and Blat, J. (2014b). LdShake support for team-based learning design. Comp. Hum. Behav. 37, 402–412. doi: 10.1016/j.chb.2012.05.029

CrossRef Full Text | Google Scholar

Hernández-Leo, D., Romeo, L., Carralero, M.-A., Chacón, J., Carrió, M., Moreno, P., et al. (2011). LdShake: learning design solutions sharing and co-edition. Comp. Educ. 57, 2249–2260. doi: 10.1016/j.compedu.2011.06.016

CrossRef Full Text | Google Scholar

Kali, Y., Goodyear, P., and Markauskaite, L. (2011). Researching design practices and design cognition: contexts, experiences and pedagogical knowledge-in-pieces. Learn. Media Technol. 36, 129–149. doi: 10.1080/17439884.2011.553621

CrossRef Full Text | Google Scholar

Katsamani, M., and Retalis, S. (2013). Orchestrating learning activities using the CADMOS learning design tool. Res. Learn. Technol. 21:18051. doi: 10.3402/rlt.v21i0.18051

CrossRef Full Text | Google Scholar

Koper, R., and Olivier, B. (2004). Representing the learning design of units of learning. Educ. Technol. Soc. 7, 97–111.

Google Scholar

Laurillard, D. (2012). Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology. New York, NY: Routledge.

Google Scholar

Laurillard, D., Charlton, P., Craft, B., Dimakopoulos, D., Ljubojevic, D., Magoulas, G., et al. (2013). A constructionist learning environment for teachers to model learning designs. J. Comp. Assist. Learn. 29, 15–30. doi: 10.1111/j.1365-2729.2011.00458.x

CrossRef Full Text | Google Scholar

Lockyer, L., Bennett, S., Agostinho, S., and Harper, B. (2008). Handbook of Research on Learning Design and Learning Objects. New York, NY: Information Science Reference.

Google Scholar

Manathunga, K., and Hernández-Leo, D. (2018). Authoring and enactment of mobile pyramid-based collaborative learning activities. Br. J. Educ. Technol. 49, 262–275. doi: 10.1111/bjet.12588

CrossRef Full Text | Google Scholar

Martinez-Maldonado, R., Goodyear, P., Carvalho, L., Thompson, K., Hernandez-Leo, D., Dimitriadis, Y., et al. (2017). Supporting collaborative design activity in a multi-user digital design ecology. Comp. Hum. Behav. 71, 327–342. doi: 10.1016/j.chb.2017.01.055

CrossRef Full Text | Google Scholar

McKenney, S., Kali, Y., Markauskaite, L., and Voogt, J. (2015). Teacher design knowledge for technology enhanced learning: an ecological framework for investigating assets and needs. Instr. Sci. 43, 181–202. doi: 10.1007/s11251-014-9337-2

CrossRef Full Text | Google Scholar

Mor, Y. (2013). “SNaP! Re-using, sharing and communicating designs and design knowledge using scenarios, narratives and patterns,” in Handbook of Design in Educational Technology, eds R. Luckin, S. Puntambekar, P. Goodyear, B. L. Grabowski, J. Underwood, and N. Winters (New York, NY: Routledge), 189–200.

Google Scholar

Mor, Y., Craft, B., and Hernández-Leo, D. (2013). Editorial: the art and science of learning design. Res. Learn. Technol. 21:22513. doi: 10.3402/rlt.v21i0.22513

CrossRef Full Text

Mor, Y., Derntl, M., Hernández-Leo, D., Craft, B., Pozzi, F., Persico, D., Retalis, S., et al. (2007). The Learning Design Grid – Resources. Available online at: http://www.ld-grid.org/

Nicolaescu, P., Derntl, M., and Klamma, R. (2013). “Browser-based collaborative modeling in near real-time,” in 9th International Conference Conference on Collaborative Computing: Networking, Applications and Worksharing (Collaboratecom), 2013 (Austin), 335–344

Google Scholar

Nielsen, L. (2013). Personas. The Encyclopedia of Human-Computer Interaction, 2nd Edn. Aarhus: The Interaction Design Foundation.

Peffers, K., Tuunanen, T., Rothenberger, M. A., and Chatterjee, S. (2007). A design science research methodology for information systems research. J. Manage. Inf. Syst. 24, 45–77. doi: 10.2753/MIS0742-1222240302

CrossRef Full Text | Google Scholar

Persico, D., Manca, S., and Pozzi, F. (2014). Adapting the Technology Acceptance Model to evaluate the innovative potential of e-learning systems. Comp. Hum. Behav. 30, 614–622. doi: 10.1016/j.chb.2013.07.045

CrossRef Full Text | Google Scholar

Persico, D., Milligan, C., and Littlejohn, A. (2015). The interplay between self-regulated professional learning and teachers' work-practice. Procedia-Soc. Behav. Sci. 191, 2481–2486. doi: 10.1016/j.sbspro.2015.04.590

CrossRef Full Text | Google Scholar

Persico, D., and Pozzi, F. (2015). Informing learning design with learning analytics to improve teacher inquiry. Br. J. Educ. Technol. 46, 230–248. doi: 10.1111/bjet.12207

CrossRef Full Text | Google Scholar

Persico, D., Pozzi, F., Anastopoulou, S., Conole, G., Craft, B., Dimitriadis, Y., et al. (2013). Learning design Rashomon I – supporting the design of one lesson through different approaches. J. Res. Learn. Technol. 21:20224. doi: 10.3402/rlt.v21i0.20224

CrossRef Full Text | Google Scholar

Pozzi, F., Persico, D., and Sarti, L. (2015). Evaluating the acceptance of an innovative learning design environment within communities of practitioners. Procedia-Soc. Behav. Sci. 191, 1019–1023. doi: 10.1016/j.sbspro.2015.04.398

CrossRef Full Text | Google Scholar

Prieto, L. P., Asensio-Perez, J. I., Munoz-Cristobal, J. A., Dimitriadis, Y. A., Jorrin-Abellan, I. M., and Gomez-Sanchez, E. (2013a). Enabling teachers to deploy CSCL designs across distributed learning environments. IEEE Trans. Learn. Technol. 6, 324–336. doi: 10.1109/TLT.2013.22

CrossRef Full Text | Google Scholar

Prieto, L. P., Dimitriadis, Y., Craft, B., Derntl, M., Émin, V., Katsamani, M., et al. (2013b). Learning design Rashomon II: exploring one lesson through multiple tools. Res. Learn. Technol. 21:20. doi: 10.3402/rlt.v21i0.20057

CrossRef Full Text | Google Scholar

Rienties, B., Nguyen, Q., Holmes, W., and Reedy, K. (2017). A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University, U. K. Interact. Design Arch. J. 33, 134–154.

Google Scholar

Rienties, B., and Toetenel, L. (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Comp. Hum. Behav. 60, 333–341. doi: 10.1016/j.chb.2016.02.074

CrossRef Full Text | Google Scholar

Svihla, V., Reeve, R., Sagy, O., and Kali, Y. (2015). A fingerprint pattern of supports for teachers' designing of technology-enhanced learning. Instr. Sci. 43, 283–307. doi: 10.1007/s11251-014-9342-5

CrossRef Full Text | Google Scholar

Venkatesh, V., and Davis, F. D. (2000). A theoretical extension of the technology acceptance model: four longitudinal field studies. Manage. Sci. 46, 186–204. doi: 10.1287/mnsc.46.2.186.11926

CrossRef Full Text | Google Scholar

Villasclaras-Fernández, E., HernáNdez-Leo, D., Asensio-PéRez, J. I., and Dimitriadis, Y. (2013). Web Collage: an implementation of support for assessment design in CSCL macro-scripts. Comp. Educ. 67, 79–97. doi: 10.1016/j.compedu.2013.03.002

CrossRef Full Text | Google Scholar

Voogt, J., Westbroek, H., Handelzalts, A., Walraven, A., McKenney, S., Pieters, J., et al. (2011). Teacher learning in collaborative curriculum design. Teach. Teacher Educ. 27, 1235–1244. doi: 10.1016/j.tate.2011.07.003

CrossRef Full Text | Google Scholar