Juxtaposing generic skills development in collaborative knowledge work competences and related pedagogical practices in higher education

This study employs the term knowledge work competence to address generic aspects of higher education graduates’ expected learning outcomes. Twenty-eight higher education courses were investigated: 1069 students responded to the Collaborative Knowledge Practices (CKP) questionnaire to rate their self-evaluated competence development. From the same courses, 56 teachers provided descriptions of the course pedagogical practices. First, students’ self-reported generic collaboration competence gains were analyzed statistically for differences between courses. Second, qualitative categorization of the pedagogical practices based on rich description of pedagogical designs and teachers’ reflective responses was carried out. This offered a categorization with elaborated descriptions and a clustering to three types of enacted pedagogical practices. Finally, the study juxtaposed these previous two results to investigate how the pedagogical features were related to students’ self-evaluations on collaboration competence gains. The findings highlighted one cluster of pedagogical practices, collaborative knowledge creation with systematic support for epistemic and group work, as most beneficial for student competence gains. In it, professional ways of working were explicitly modeled and practiced, teacher support for knowledge creation during contact teaching was available, and time was reserved for reflection with students. Such pedagogical practices are important to ensure graduates’ fluent transition to complex knowledge work.

Conceptual background and research questions

Higher education is expected to prepare future academic experts for the knowledge-driven global world (Barrie, 2012; Karlgren et al., 2020a). Successful learning and working in today’s knowledge-based society demands competence that exceeds individual expertise and engages individuals in joint collaboration and knowledge creation in teams (Binkley et al., 2012). Such competence is embedded in people’s actions, social interaction and the socio-material affordances of their environments as they co-develop knowledge objects (e.g., Damşa and Muukkonen, 2020), engage in epistemic practices (Markauskaite and Goodyear, 2017) and regulate their collaborative learning and working as a team (e.g., Borge et al., 2018; Splichal et al., 2018). To understand how such competence develops in higher education, we need to examine both the experienced competence gains and pedagogical settings in which these competences are nurtured. Although many different factors influence learning, there is a need to further decipher the role pedagogical practices may have on competence development.

Many countries are struggling to keep up with the demands of a highly skilled workforce (OECD, 2018), and the current COVID-19 pandemic has created sudden changes and challenges as team members are forced to work remotely and devise novel practices for collaboration. Professional teamwork has taken a major shift from disciplinary to interdisciplinary teams to respond to the growing complexity and dynamic nature of tasks and to seek better ways to tackle ambiguous challenges (Benoliel and Somech, 2015). However, educational objectives and practices may not be truly aligned with the changes in professional work (Markauskaite, 2020). Particularly in interdisciplinary collaboration, both discipline specific and generic (e.g., critical and analytical thinking, problem-solving, self-management of learning, communication skills, and information and digital literacy, Binkley et al., 2012) competences are needed in an intertwined manner to produce novel ideas, syntheses, designs or practices (Goodyear and Zenios, 2007). How students are directed to engage in learning activities is instrumental for competence development (Goodyear and Zenios, 2007; Puntambekar et al., 2007).

A meso-level investigation of competence development originates theoretically from sociocultural paradigm: Learning is regarded as embedded in social processes, practices, and tool use rather than being an individual venture (Säljö, 2010). This study employs the term knowledge work competence to address the generic aspects of higher education learning. Knowledge work competence (Damşa and Muukkonen, 2020; Karlgren et al., 2020b; Muukkonen et al., 2020) for higher education graduates refers to capacities for advanced knowledge work, i.e., understanding and creating knowledge, orchestrating collaboration, and self and co-regulating performance. As such, defining “work” for higher education graduates and exploring the relationship between work and learning are complex tasks considering interdisciplinarity and dynamics of external environments across professional fields (Jung, 2022). However, graduates from higher education need to be equipped with competences to solve complex authentic problems regardless of field, take part in creating knowledge in real working life settings and promote novel solutions by using the community’s collective, technology-mediated efforts.

This paper carries out a multi-method investigation. First, on a dataset of self-reported student assessments of own generic competence development, more specifically in collaborative knowledge work competence. The self-reports were collected following a specific higher education study unit, referred to as a course. This offers the student perspective on which types of competence development was central in the examined course. Second, the study offers a framework for analyzing pedagogical features in the same courses. The pedagogical features were mapped through a survey to the same courses’ teachers, followed by a categorization of the pedagogical practices based on rich description of pedagogical designs and reflection responses by the teachers. This offers a categorization and elaborated description of the pedagogical practices and their clustering to three types of enacted practices. Third, as the core result, the study juxtaposes these previous two results to investigate how certain types of pedagogical practices may contribute to generic competence development, particularly collaborative knowledge work competences. The courses represented authentic higher education instructional practices: organized as lecture, project, inquiry, and interdisciplinary courses, which all involved some type of collaboration between peers. Hence it was meaningful to examine the variation of competence development in relation to the pedagogical practices.

Previous studies have investigated knowledge work competence by structuring it as object-bound collaboration, integration of personal and collective efforts, development through feedback, persistent development, understanding of different disciplines and related expertise, interdisciplinary collaboration, and using flexible tools and technology (e.g., Karlgren et al., 2020b; Muukkonen et al., 2020; Vesikivi et al., 2020). This builds on the theoretical background of socio-cultural theories of learning and particularly on the knowledge creation metaphor (Paavola and Hakkarainen, 2005). Acquisition and participation as two metaphors of learning were put forward by Sfard (1998). The acquisition metaphor of learning addresses assimilation of knowledge and the individual’s mental models and strategies of learning. The participation metaphor refers to adaptation to the existing cultural and communal practices and the dialogical practices of learning. As a third metaphor, Paavola and Hakkarainen (2005) added the knowledge creation metaphor. It introduced the presence of artifacts, products and practices i.e., objects, and collaboration to advance them as pivotal (Paavola and Hakkarainen, 2005). The objects can for instance be a report or essay co-authored together, or a procedure description, website, or plan co-created in collaboration. Briefly expressed, object-orientedness is a concept formulated in the cultural-historical theory, referring to a shared motive or tangible object for a learning or working community (Miettinen and Virkkunen, 2005). The object mediates knowledge advancement as the participants collaborate to negotiate meanings, extend, and version it. For students, educational activities emphasizing knowledge creation metaphor often involve more open-ended and complex assignments which integrate collective efforts around iterative development of knowledge objects.

Pedagogical design includes many aspects. “Design involves making invitations to other people to act in certain kinds of ways. These invitations can be clear and explicit, but they are sometimes encoded into the affordances of materials. Designers’ knowledge has to include ways of predicting, or at least imagining, how other people will respond to these invitations” (Goodyear, 2015, p. 39). Making these designs involves pre-active aspects of planning but also post-active phases of reflection, evaluation, and assessment. Resulting course documents (e.g., course plans, instructions, assignment descriptions) and digital tool choices can be considered material instantiations of the teachers’ ideas and decisions regarding the organization of tasks, activities, and responsibilities (Goodyear, 2015; Esterhazy et al., 2021).

Next, the introduction of previous research will review object-orientedness, role of integration of efforts in collaborative learning, feedback, cross-fertilization, and digitalization in higher education. Competence development and pedagogical practices are addresses as an intertwined phenomenon, which is further elaborated in the empirical investigation.

Object-orientedness

Evidence in higher education is building on how students engage in meaningful interactions with peers, knowledge resources and objects, and the social and digital-material environment in which such activities take place (Damşa and Muukkonen, 2020). Learning addressed as a process of knowledge creation brings it closer to professional practices, which takes place through interactive practices that contribute to ideas being materialized into (shared) knowledge objects (Paavola et al., 2011). In higher education, such objects may be for instance reports, designs or products ideated and co-created in student collaboration. In a study comparing two anatomical sciences courses with different pedagogical designs, the students reported more competence gains when they had a shared object to prepare, the teaching presentation, compared to just taking part in interaction with peers on an assignment (Laakkonen and Muukkonen, 2019). The shared knowledge object intensified the need to collaborate and learn about planning, coordinating, and sharing responsibility during collaboration as well as the integration of individual and collaborative contributions.

From the pedagogy point of view, the objects are elaborated through intermediate and mediating artifacts and tools, and iterative development of tangible artifacts, such as draft and sketches (Miettinen and Paavola, 2018; Damşa and Muukkonen, 2020). This requires the teacher to make specific choices about the intensity and extend of collaboration, how collaboration is assessed, to plan a process involving iterative cycles of feedback, editing and monitoring the epistemic challenge.

Integration of efforts in collaborative learning

Research emphasizes that engaging in productive co-construction of knowledge does not happen automatically (Baker et al., 2013). Individuals and groups vary in the extent of their competence to collaborate with others and to respond to the situation-specific learning and interaction challenges in authentic educational settings (Näykki et al., 2014). During collaboration, students are expected to negotiate task aims and standards, to act strategically based on monitoring their group activities, to revise processes and outcomes, to select and use suitable digital tools, and to productively deal with any challenges groups face (e.g., Splichal et al., 2018). Through extended practice, successful learners and team members use a repertoire of skills and strategies to regulate their learning processes on cognitive, social, and emotional levels (Baker et al., 2013; Hadwin et al., 2017).

Regarding pedagogy, Vogel et al. (2017) meta-analysis found that computer-supported collaborative learning scripts were particularly effective for domain-specific learning when they prompted transactive activities in which learners build on the contributions of peers and when additional content-specific scaffolding such as worked examples were available. The present study was motivated by the need to better understand content-generic aspects of learning, and, further, aims to examine at an elaborate detail the design of collaboration with peers and scaffolding for generic competence development and its impact on student collaboration competence learning.

Feedback

Student-centered methods in higher education emphasize students’ central role in regulating their own learning. This involves generating and soliciting feedback on their own learning (Boud and Malloy, 2013). Further, orchestration of collaboration extends the competence demands to proactive feedback on both individual and collaborative learning.

Making such a role feasible presumes that teachers need to plan productive feedback opportunities in which students can engage actively with and employ the feedback for future learning. These include dialogical processes and activities which can support and inform the student on the task at hand, while catering for the ability to self-regulate performance on future tasks (Carless et al., 2011; Esterhazy et al., 2021). Similarly, regarding engagement on the shared object, collective feedback is important for co-development and competences for advancing collective outcomes. Further, Esterhazy et al. (2021) showed that productive feedback should not be understood as a prescribed model or solution across all disciplines but contextualized in disciplinary or interdisciplinary objectives and pedagogy.

Cross-fertilization

Modern teamwork is often organized as collaboration in online communities, with heterogeneous and temporary convergence (Faraj et al., 2011). Students entering work life should be ready to act as agentic collaborators who can participate proactively in solving interdisciplinary and ambiguous challenges. Therefore, cultivating competence for working in interdisciplinary teams and creating joint knowledge objects are increasingly considered important objectives in higher education (Cooke and Hilton, 2015).

In pedagogical practice, cross-fertilization refers to interaction between different areas of expertise or organizations, for instance, by collaboration, problem solving or new product development for purposes extending beyond educational institutions (Paavola et al., 2011). Similarly, Cremers et al. (2016) used the term ‘hybrid learning configurations’ to define designs which connect formal learning with workplace experiences by integrating settings for studying and working. Interdisciplinary co-creational activities with ill-defined and authentic tasks are central in such configurations. Project courses are commonly used methods by involving various stakeholders inside and outside of educational institutions. Projects engage students in producing tangible and meaningful results, sometimes in cooperation with professionals, generating, potentially, outcomes for continued use in an organization (e.g., Viswanathan et al., 2012).

Digitalization in higher education

Two important assumptions are in need of consideration in the digitalization of higher education: First, the assumption that technology is an instrumental issue that is neutrally implemented and second, that students became fluent users of technology in a self-directed way (Castañeda and Selwyn, 2018). Both of these assumptions are tightly tied to generic competences. The way that digital tools are integrated to pedagogical design can have a considerable influence on the kinds of practices that can be designed for and enacted in collaboration. Also, students do not necessarily have the required competence to engage in technologically mediated knowledge work, collaboration, or expert-like practices of writing and co-creation, without instructed and guided engagement.

Pedagogy is inherently part of any educational technology use in higher education (Castañeda and Selwyn, 2018) although this is often reduced to learning management systems serving very basic information distribution and communication needs. Theory and pedagogy informed technology design has had considerable efforts invested through research and development, but the mainstream technology use remains designed for the support of logistical processes rather than for pedagogical change (e.g., Collis and Moonen, 2008).

Research questions

The study investigated how higher education students in twenty-eight courses evaluated their learning and competence gains in the generic competences of collaborative knowledge work practices. Further, the study examined the courses’ pedagogical practices to provide a combined, juxtaposed, understanding of how the pedagogical practices were related to student learning. The following research question were examined:

1. Were there differences in students’ self-assessed competence gains between courses?

2. What kinds of design of collaboration did the courses’ pedagogical design reflect?

3. How the pedagogical features were related to students’ self-evaluations on collaboration competence gains?

Materials and methods

The general investigative approach was an explanatory multiple case study (Yin, 2014) and a multi-methods approach was used in the data collection and analysis (De Laat et al., 2007; Cresswell, 2009). The aim of the approach was to gain a triangulated understanding of course pedagogical design, enacted practices and student learning. Teachers were invited to take part in the study, by answering a questionnaire on course design and reflection responses and by passing forward a link to an e-questionnaire to the students and encouraging their participation. Students were asked to answer the questionnaire at the end of their course. All participants were asked to provide their informed consent electronically, and those responses without a consent were excluded from the study.

Courses and participants

The data included responses from twenty-eight courses in two large Finnish universities. The data was not intended to be representative of specific fields, rather, the aim has been to involve multiple fields and types of collaboration to investigate the variation. The fields of study included education, educational psychology, philosophy, life sciences, law, and economics. The courses, participants, and response rates are detailed in Table 1. Courses typically lasted for one period of c. 8-9 weeks and were obligatory courses in the degree program. Courses were included which received more than 7 responses per course from students and a teacher response was available. The response rate to the CKP questionnaire for students was 55.9% (varied between 25-58%), as a total of 1,912 students completed these investigated courses. In total 56 teachers were included in the data, with mean age 48.5 years and 55% female. Some courses had multiple teachers, especially project-type courses or larger courses and thus we obtained several teacher responses per course. In total 1,069 student responses were included for analysis 19.8% male, 79.3% female and 0.8% reported other or missing. 755 students were enrolled in a first-year course, other courses were in later bachelor or master’s degree studies. Students’ average age was 24.9 (SD = 6.6) reflecting the rather high university starting age in Finland.

TABLE 1

Table 1. Participants.

Data collection

The first data consisted of higher education students’ responses to the Collaborative Knowledge Practices questionnaire (CKP; Muukkonen et al., 2020). The CKP questionnaire has been developed and validated for use as a generic self-evaluation tool for students on course-based learning outcomes on generic collaborative knowledge work competences (Karlgren et al., 2020b; Muukkonen et al., 2020). The CKP does not measure content learning, it thus complements other content-related evaluation measures employed in a given course. The scales of CKP were used to measure course-related learning. The seven scales are: Collaborate on shared objects, Integrating individual and collaborative working, Development through feedback, Persistent development of knowledge-objects, Understanding various disciplines, Interdisciplinary collaboration, and Exploit digital technology. Students were asked to evaluate how each statement (27) corresponded to their competence learning on the seven scales of the CKP. “During the course I have learned …,” e.g., “to develop ideas further together with others,” “to understand the value of commenting on work in progress,” and “to use various digital applications and use them together whenever needed” (please see for details Muukkonen et al., 2020; Karlgren et al., 2020b). The statements were on a five-point Likert-scale (1 = not at all – 5 = very much).

The second data consisted of teachers’ open answers (N = 56) to an online questionnaire about the practices (collaboration design, types of tasks, use of digital tools, guidance, assessment) and reflection of experiences in 28 courses. Additional materials collected from the investigated courses, including course descriptions, task guidelines, teaching materials, digital platform content, lesson observations, or students’ feedback forms, were used as complementary data in the analysis of pedagogical practices.

Data analysis

Collaborative knowledge practices questionnaire

Student responses were screened for outliers and seventeen participants were removed from data due to missing data or unvarying responses. In the first two courses, the data was collected with an option “not applicable” (0). We replaced the ‘not applicable’ responses with “not at all” in the scale in order to aid interpretation of the data (please see Muukkonen et al., 2020 for details). The seven scales of the CKP questionnaire were used to examine course related self-reported learning. The scale reliabilities were good (Cronbach’s alpha 0.73-0.86).

Pedagogical practices of the courses

The pedagogical practices analysis aimed to examine the design features based on identified pertinent pedagogical design elements for collaborative practices. The variation within the pedagogical practices was outlined by describing in detail the pedagogical practices for every course. Initial categories were based on related theories such as knowledge building (Bereiter, 2002), metaphors of learning (Paavola et al., 2011), and authentic learning (McCune, 2009) as well as empirical studies (e.g., Lakkala et al., 2015, 2020; Ilomäki et al., 2017. The unit of analysis was the whole course, and the analysis covered, first, the teachers’ questionnaire answers about the course practices and, second, all other data available from the courses. The categories were created through abductive use of theory-informed and data-grounded data analysis (Timmermans and Tavory, 2012).

The pedagogical design of each course was described in a table under the initial categories. The different ways to design collaboration, use scaffolds and modeling for collaboration were documented. Based on these descriptions, categories and subcategories were further elaborated. For instance, process-like emphasis has been raised in prior literature as an important quality of collaborative learning and knowledge building (Muukkonen and Lakkala, 2009; Scardamalia and Bereiter, 2014) to improve student produced contributions. Three subcategories were identified related to this: Iterativeness describing the longitudinal versioning and improvement of outcomes; availability and designed points of Feedback to support the process; and Reflection of practices explicitly included and modeled as part of the learning process. These sub-categories were named as features of pedagogical practices and positioned in three exclusive levels. Level 1 not involving the described features, level 2 to some extend and level 3 to wide extend. Four researchers created the categories and sub-categories together in several joint analysis sessions and made a preliminary analysis with a sub-set of 16 courses. After the preliminary analysis, one researcher made the analysis of all courses, after which the analysis results were, again, discussed together between the researchers in several sessions, clarifying unclear definitions and making decisions about the final categorization. The discussions were carried out until there was an agreement between two coders for the entire data and four coders for a c. 50% of the units of analysis as it was discussed during the development process. Category and subcategory descriptions were written. Finally, each course was scored with all sub-categories using levels 1-3 to explicate the extent and nature of the pedagogical features in the course practices (see Table 2). All the highest levels (score 3) aim to describe a pedagogical practice where the targeted competences are modeled and supported by various design decisions implemented in the course.

TABLE 2

Table 2. Categorization, level descriptions and examples from the analysis of pedagogical practices.

The main categories are the following (see Figure 1):

• Object-orientedness refers to the degree that the course collaboration is organized around shared knowledge objects, such as a report, website, design or a product. The extensiveness of the shared object influences its role in the collaboration. How the developed shared object is planned to be reused by students or other stakeholders may add re-use value to the knowledge object.

• Epistemic challenge is outlined by the wider or more narrow problem space where the student-centered activities are embedded in. Explicit modeling of professional practices is a way to support students in tackling complex epistemic challenges.

• Process-like emphasis involves a design that includes iterations in the process, allowing for revising and improving outcomes. Offering and receiving feedback during the process is a central for improvement as well as the collective and individual reflection of practices.

• Intensity of collaboration is enhanced by design decisions that emphasize the centrality of collaboration and employ tasks which require the integration of individual and collaborative efforts.

• Cross-fertilization refers to involving multiple disciplinary expertise and professional practices in the course activities or participants. External collaborators may also take varying roles in the course, e.g., by giving assignments or being clients.

• Digitality describes the use level of digital tools in a course, e.g., for collaboration, communication, disciplinary activities, or teaching.

• Assessment foci highlights how the pedagogical design incorporates versatile assessment methods to support competence development and, especially, considers also more generic types of competences, such as knowledge work competences.

FIGURE 1

Figure 1. Overview of categories of pedagogical practices.

A K-means Hierarchical cluster analysis was conducted using the scores of sub-categories given to each course for grouping the courses. One-way ANOVA and Post hoc (Tamphane’s T2) analysis were applied for comparing cluster means.

Finally, an ANOVA analysis of variance was carried out to examine how students’ self-evaluations about learning of knowledge work competence were related to the three clusters of pedagogical practices. For this comparison, we used the clusters found for grouping the courses (RQ2) and examined how students’ self-reported competence learning scale means and standard deviations (RQ1) were distributed across these clusters.

Results

Students’ self-assessed competence gains across courses

To examine whether there were differences between students’ self-reported competence learning between courses, scale means and standard deviations were calculated for each course (Table 3). The evaluations showed statistically significant variation between courses (Table 4). Scales related to Understanding various disciplines and Interdisciplinary collaboration were on average scored the lowest. Scales related to learning to Collaborate on objects and Integrate efforts were scored on average the highest.

TABLE 3

Table 3. Collaborative knowledge practices scale means and standard deviations.

TABLE 4

Table 4. Collaborative knowledge practices scale reliabilities, one-way analyses of variance, and correlations.

Pedagogical features related to the design of collaboration and student activities

The analysis of pedagogical practices through cluster analysis (Table 5) uncovered three prototypical types of enacted practices (a) activating learning promoting content learning, (b) self-directed individual or collaborative knowledge creation, and (c) challenging collaborative knowledge creation with systematic support (Figure 2).

TABLE 5

Table 5. Pedagogical practices cluster analysis descriptors.

FIGURE 2

Figure 2. Three clusters of pedagogical practices.

Courses in Cluster 1 can be illustrated to represent activating learning practices promoting content learning. Courses primarily focused on the acquisition of domain content through activating lectures and/or small-scale individual and collaborative knowledge creation tasks both in contact sessions and as assignments. It is noteworthy, that all ten courses in Cluster 1 were for first year students and there were, on average, 96.2 students in a course (min 60, max 178). The courses were from various subject domains: economics, biosciences, law, agricultural sciences and education. It appeared that the pedagogical practices were designed for large class activities, with primary emphasis on engaging students with activating methods on content learning.

As an example of Cluster 1, one course (ID94) was a 5 ECTS course about legal thinking for the first-year law students including about 180 participants. Students had weekly group tasks to analyze law cases and write up one own law. Groups organized their group work time themselves outside contact sessions. In addition to attending expert lectures, students presented their group outcomes and/or were opponents to some other group in contact sessions. The tasks were applied tasks, and although the tasks were the same for all groups, the problem space was open. Course platform Moodle was used for sharing materials and group discussions. In addition, lecturers demonstrated professional databases for students. Groups had a freedom to use digital applications of their choice in presentations. The course was graded on a scale pass-fail and the evaluation was group-based. If students were absent from the group presentations, they had to make an individual reflection task.

Cluster 2 represents pedagogical practices that can be described as self-directed individual or collaborative knowledge creation practices with content-focused contact teaching. Courses included lectures or hands-on sessions and one major open-ended individual or collaborative knowledge creation task (in addition to smaller ones) elaborated mainly outside course meetings. Individual students or groups received occasional and tailored guidance from the lecturers in the meetings to complete the tasks. Six of the courses in Cluster 2 were master level courses, two were for second- or third-year undergraduate students. The average number of participants in the courses was 41.1 (min 25, max 69). The domains of the courses included education, biosciences and forest sciences.

An example in Cluster 2 is a 5 ECTS course (ID92) about the philosophy of science for master students in education for about 25 participants by two lecturers. In the course, each student prepared an individual theoretical essay on a topic chosen by themselves. Students were organized in thematic peer groups where the members commented on each other’s essays at certain points during the course. Weekly contact sessions included lectures with discussions and sometimes also working in the thematic groups. The essays were elaborated and commented on mainly as homework. A course platform Moodle was used for sharing materials, peer commenting and task submissions, MS Word for essay writing. The assessment was based on grading from 1 to 5 done by the lecturers; both the quality of essays and peer commenting activity were taken into account in the assessment.

Cluster 3 included courses where practices represented challenging collaborative knowledge creation with systematic support for epistemic and group work. Courses were shaped by one major open-ended, authentic and challenging collaborative knowledge creation task (in addition to smaller ones) elaborated both in contact sessions and out-of-class assignments. Professional ways of working were explicitly modeled and practiced with the students. Two courses in this cluster were for first year students, the others were targeted for third year undergraduate students and/or master students. The average number of students in the courses or study groups (one course was divided in three study groups because the total number of participants was 375) was 51.8 (min 8, max 124). The subject domains of the courses included biosciences, agricultural sciences and education; one course was a multidisciplinary project course.

In Cluster 3, one example of a course (ID96) is a 3 ECTS obligatory project management and work life skills course for third year undergraduate students including about 40 students. During the course, students completed various assignments relevant for their future careers. Individually they made a CV, a LinkedIn profile and a portfolio, gave a personal elevator pitch and interviewed a professional in the field. In groups they made a summary of interviews as a blog post and produced a project plan in groups through a longitudinal process including several phases and sub-tasks. The project topic was given by experts from another university unit, based on a real task from an existing project. Groups competed on the best solution to the project assignment, and the winner was chosen based on the group report and pitching of the solution in the last course meeting. The course was graded on a scale pass-fail, but all sub-tasks had to be completed acceptably and many of them were commented on and assessed both by the course lectures and experts from other university units.

To summarize, what differentiated cluster 1 from the remaining two clusters, based on the level analysis, was that there was no emphasis on an artifact development, iterativeness in the process, structured feedback during the process nor explicit assessment of generic competences. Cluster 2 was differentiated from cluster 3 especially by more emphasis on lecturing and less on student-centered activities, less self-reflection on collaboration process and fewer involvement of external collaborators. All clusters had a rather low level of multidisciplinarity, but quite systematic use of group work practices. This was affected by the fact that all courses included in the study had some type of collaboration task included in the course, because otherwise completing the CKP questionnaires would not have been meaningful for the participants.

Relationship of students’ self-evaluations on competence gains and pedagogical practices

We compared the means of the seven scales of CKP against the cluster membership. There were 435, 164, and 467 students in clusters 1, 2, and 3, respectively. A one way ANOVA provided evidence that the clusters had statistically significant differences on each of the CKP scales (F(2,1065) = 18.7 – 79.3, p < 0.001). A Scheffe post hoc test showed that in the scales of Integrate efforts, Persistent development, and Interdisciplinary collaboration, clusters 1 and 2 did not have statistically significant differences, while on the scale Feedback cluster 2 and 3 did not have statistically significant differences. All other comparisons were statistically significantly different (p < 0.05). Figure 3 displays the scale means by pedagogical practices cluster.

FIGURE 3

Figure 3. Learning of knowledge work competences by pedagogical practices.

Overall, cluster 2 self-directed knowledge creation held the lowest average CKP scale scores, with the exception of Feedback. The students’ self-evaluated learning of competence gains in generic competences of collaborative knowledge work was the highest in cluster 3 supported collaborative knowledge creation. As described above, cluster 3 had been qualitatively analyzed as pedagogical practices most specifically designed to support collaborative knowledge creation.

Discussion

The study undertook an examination of generic collaboration competences from three directions. Twenty-eight courses in higher education contributed data to the study, each involving some form of collaborative assignments for the students. Further, each course had some field specific expected learning outcomes regarding the content learning as well as some objectives for gaining knowledge work competences, representing generic competences in collaboration and professional epistemic practices.

First, we investigated how the students evaluated self-assessed competence gains across the courses. We found that there were statistically significant differences between courses in how students rated their learning on the seven scales of the Collaborative Knowledge Practices CKP questionnaire.

Second, we analyzed the pedagogical design decisions made in these courses and developed a categorization of the pedagogical features. This enabled a more detailed examination of how the courses were intended to model and support complex knowledge work competences. This analysis provided three clusters of pedagogical practices. Cluster 1 was considered to represent activating learning practices promoting content learning. They emphasized acquisition of domain content through activating lectures and/or small-scale individual and collaborative knowledge creation tasks. These were most prevalent in first year courses in the data. Cluster 2 was named self-directed individual or collaborative knowledge creation practices with content-focused contact teaching. Courses included lectures or hands-on sessions and one major open-ended individual or collaborative knowledge creation task elaborated mainly outside course meetings. The knowledge creation challenge was clearly present in the assignments, but strong facilitation for how to carry it out was missing. Cluster 3 was named collaborative knowledge creation with systematic support for epistemic and group work. Professional ways of working were explicitly modeled and practiced with the students. This third is argued to be an important addition to current educational practices to ensure graduates’ fluent transition to knowledge work.

Thirdly, we investigated how the pedagogical features were related to students’ self-evaluations on competence gains. This provided evidence that the three clusters of pedagogical practices were related to differing student evaluations on competence development on the seven CKP scales. Pedagogical practices of cluster 3 were systematically evaluated to generate more competence gains. Comparison of clusters 1 and 2 offered interesting evidence that the activating learning practices were considered to generate higher competence gains. Further investigation is needed to fully understand these differences. One explanation might be that the self-directed nature of group tasks and shortcomings in the pedagogical support to complete them in cluster 2 courses did not produce strong learning experiences in collaborative knowledge work competences. A parallel result was found in a study focusing on the pedagogical quality of international summer courses (Lakkala et al., 2018): Courses representing traditional teacher-centered lecturing combined with self-directed academic studying outside contact sessions received, on average, lower scores from students in the course evaluation survey, compared with courses that followed practices of activating learning or shared expertise.

The object-orientedness was raised in the qualitative analysis in cluster 2 and 3, but related competence learning was highlighted in cluster 3, which offers crucial information for scaffolding and designing complex collaboration processes. Engaging in integration of efforts is a means to initiate and practice valuable generic competences for collaboration highlighted here, as also pointed out in many prior studies on co-regulation and object-oriented learning (e.g., Borge et al., 2018; Splichal et al., 2018; Damşa and Muukkonen, 2020). But the finding also suggested that mere setting up of collaboration in a course is not enough without modeling and scaffolding. Feedback and cross-fertilization between fields play a key role in all professional practices, therefore becoming competent in them requires effortful practice. On using digital tools in higher education, the outcomes showed that for the most part they were an integrated part of the pedagogical design of courses, with few exceptions.

Theoretically, emergence of the clusters contrast various pedagogical design prototypes for collaborative learning, where Cluster 1 and 3 can be recommended, but cluster 2 points out further support needs for students. The pedagogical features framework offers a tool to examine the design features of collaboration in a structured fashion. Specific aspects targeted in pedagogical designs were also, on average, rated higher by the students on competence gains namely Collaborate on objects and Integrate efforts. The findings suggested higher education can employ student self-evaluations as measures of generic skills development in knowledge work competences. Furthermore, curriculum design can benefit from a systemic approach to mapping and specifying both the features of pedagogical practices and expected learning outcomes on generic skills (see this issue).

This study was conducted as a multi-methods investigation. Contrasting the outcomes of qualitative findings on pedagogical practices and the scaled responses enabled to juxtapose the experienced learning of generic competences and the analyzed pedagogical practices. The sample size of students was quite large and included students from different study years and ages. A limitation of the study is that the age of participants and other background factors like previous work experience were not included as independent variables in the analysis. Further studies should investigate the influence of age on the outcomes. Preliminary examinations with CKP data have suggested that there is considerable individual variation and that a young first year students might provide very different self-evaluation than first year students with prior work experience, so status as first year student is not enough information. This phenomenon might have a relationship with that the questionnaire asks to evaluate the extend of learning in a certain course, and if student experiences that the learning has taken place earlier, then their evaluation may remain moderate. This is a methodological question pertaining to the nature of self-reports more widely. As suggested also by Vogel et al. (2017), the development of collaboration competence might depend on the amount of practice in the corresponding activities, hence previous work experience in knowledge work might influence self-evaluation of new competence gains.

Limitations regarding self-reports of learning have been discussed repeatedly. Although self-report measures are considered suitable for higher education (Roth et al., 2016), there are concerns over the self-report’s closeness to actual behavior. Benton et al. (2013) pointed out that the validity of student self-evaluation of learning depends on that relevant learning objective have been identified and, further, whether students can offer accurate evaluations of their learning. Earlier development process of the CKP questionnaire identified and removed those items that were ambiguous or uninformative, thus contributing to relevance in term of targeted and validated questionnaire. The relevance in terms of the course specific learning objectives was evaluated by the students, which aligned with the teacher descriptions in a theoretically and pedagogically integrated way. Further research will need to add parallel means to evaluate student learning, e.g., by pre-post or teacher evaluations.

Some courses had multiple teachers and how this teacher-student ratio influences the pedagogical choices and abilities to support students is an important future research question. Contextual issues have an effect on the pedagogical decisions, e.g., in mass courses it is not often possible to implement complex design for collaborative knowledge practices. Nevertheless, among the investigated courses there were also courses with large number of participants (e.g., a course in education, about 100 participants in one teaching group) and the courses were implemented representing collaborative knowledge creation practices with systematic support (cluster 3).

The transformation to fully online and hybrid teaching is a further design challenge for higher education. Previous research has suggested that it is rare in online learning settings that students engage in high-quality activities or knowledge creation spontaneously (Kobbe et al., 2007). The current findings evidenced that for teaching collaboration and generic collaboration competences, there are a multitude of pedagogical design decisions to make. The courses in cluster 3 had a distinct emphasis on modeling, offering teacher support during contact teaching, and reserving time for reflection. This demands significant before class preparation from teachers and using time to introduce various tools and models, e.g., for expected learning outcomes for generic competence or orchestration of multidisciplinary collaboration. It is aligned with the call for design for learning (Goodyear, 2015), which entails investing more heavily in the planning phase and recognition of the primary role of design for student learning. Further, the results point out a need for a well-structured digital environment for sharing materials or links to other services, practical information, and guidance about online learning practices and assessment. These are needed to enable students to concentrate on the learning goals instead of trying to find out what is expected of them. Besides content learning objectives, teachers’ social presence is especially important for modeling demanding generic competences and interaction around knowledge creation.

Data availability statement

The datasets presented in this article are not readily available because not all data can be rendered anonymous. Anonymized raw student responses supporting the conclusions of this article can be made available by the authors. Teacher data cannot be made fully anonymous, metadata therefrom can be made available. Requests to access the datasets should be directed to (HM, Hanni.muukkonen@oulu.fi).

Ethics statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work, and approved it for publication.

Funding

This work was partly supported by the Academy of Finland (grant number: 285806).

Acknowledgments

We are very thankful to the students and teachers participating in the study.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Baker, M. J., Andriessen, J., and Järvelä, S. (2013). Affective Learning Together: Social and Emotional Dimensions of Collaborative Learning. Milton Park: Routledge. doi: 10.4324/9780203069684

CrossRef Full Text | Google Scholar

Barrie, S. C. (2012). A research-based approach to generic graduate attributes policy. High. Educ. Res. Dev. 31, 65–77. doi: 10.1080/07294360.2012.642842

CrossRef Full Text | Google Scholar

Benoliel, P., and Somech, A. (2015). The role of leader boundary activities in enhancing interdisciplinary team effectiveness. Small Group Res. 46, 83–124. doi: 10.1177/1046496414560028

CrossRef Full Text | Google Scholar

Benton, S. L., Duchon, D., and Pallett, W. H. (2013). Validity of student self-reported ratings of learning. Assess. Eval. High. Educ. 38, 377–388. doi: 10.1080/02602938.2011.636799

CrossRef Full Text | Google Scholar

Bereiter, C. (2002). Education and Mind in the Knowledge Age. Hillsdale: Erlbaum.

Google Scholar

Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., et al. (2012). “Defining twenty-first century skills,” in Assessment and Teaching of 21st Century Skills, eds P. Griffin, B. McGaw, and E. Care (New York, NY: Springer), 17–66. doi: 10.1007/978-94-007-2324-5_2

CrossRef Full Text | Google Scholar

Borge, M., Ong, Y. S., and Penstein Rosé, C. (2018). Learning to monitor and regulate collective thinking processes. Int. J. Comput. Support. Collab. Learn. 13, 61–92. doi: 10.1007/s11412-018-9270-5

CrossRef Full Text | Google Scholar

Boud, D., and Malloy, E. (2013). Rethinking models of feedback for learning: the challenge of design. Assess. Eval. High. Educ. 38, 698–712. doi: 10.1080/02602938.2012.691462

CrossRef Full Text | Google Scholar

Carless, D., Salter, D., Yang, M., and Lam, J. (2011). Developing sustainable feedback practices. Stud. High. Educ. 36, 395–407. doi: 10.1080/03075071003642449

CrossRef Full Text | Google Scholar

Castañeda, L., and Selwyn, N. (2018). More than tools? Making sense of the ongoing digitizations of higher education. Int. J. Educ. Technol. High. Educ. 15:22. doi: 10.1186/s41239-018-0109-y

CrossRef Full Text | Google Scholar

Collis, B., and Moonen, J. (2008). Web 2.0 tools and processes in higher education: quality perspectives. Educ. Media Int. 45, 93–106. doi: 10.1080/09523980802107179

CrossRef Full Text | Google Scholar

Cooke, N. J., and Hilton, M. L. (eds) (2015). Enhancing the Effectiveness of Team Science. Washington, DC: NAP.

Google Scholar

Cremers, P. H. M., Wals, A. E. J., Wesselink, R., and Mulder, M. (2016). Design principles for hybrid learning configurations at the interface between school and workplace. Learn. Environ. Res. 19, 309–334. doi: 10.1007/s10984-016-9209-6

CrossRef Full Text | Google Scholar

Cresswell, J. W. (2009). Mapping the field of mixed methods research. J. Mix. Methods Res. 3, 95–108. doi: 10.1177/1558689808330883

CrossRef Full Text | Google Scholar

Damşa, C., and Muukkonen, H. (2020). Conceptualising pedagogical designs for learning through object-oriented collaboration in higher education. Res. Pap. Educ. 35, 82–104. doi: 10.1080/02671522.2019.1677751

CrossRef Full Text | Google Scholar

De Laat, M. F., Lally, V., Lipponen, L., and Simons, P. R. J. (2007). Online teaching in networked learning communities: a multi-method approach to studying the role of the teacher. Instr. Sci. 35, 257–286. doi: 10.1007/s11251-006-9007-0

CrossRef Full Text | Google Scholar

Esterhazy, E., Nerland, M., and Damsşa, C. (2021). Designing for productive feedback: an analysis of two undergraduate courses in biology and engineering. Teach. High. Educ. 26, 806–822. doi: 10.1080/13562517.2019.1686699

CrossRef Full Text | Google Scholar

Faraj, S., Jarvenpaa, S. L., and Majchrzak, A. (2011). Knowledge collaboration in online communities. Organ. Sci. 22, 1224–1239. doi: 10.1287/orsc.1100.0614

PubMed Abstract | CrossRef Full Text | Google Scholar

Goodyear, P. (2015). Teaching as design. Rev. High. Educ. 2, 27–50.

Google Scholar

Goodyear, P., and Zenios, M. (2007). Discussion, collaborative knowledge work and epistemic fluency. Br. J. Educ. Stud. 55, 351–368. doi: 10.1111/j.1467-8527.2007.00383.x

CrossRef Full Text | Google Scholar

Hadwin, A. F., Järvelä, S., and Miller, M. (2017). “Self-regulation, co-regulation and shared regulation in collaborative learning environments,” in Handbook of Self-Regulation of Learning and Performance, eds D. Schunk and J. Greene (Milton Park: Routledge), 65–86. doi: 10.4324/9781315697048-6

CrossRef Full Text | Google Scholar

Ilomäki, L., Lakkala, M., and Toom, A. (2017). “Knowledge work assignments in upper secondary school: Results of 13 cases,” in Proceedings of the 17th Biennial Conference for Research on Learning and Instruction (EARLI), Tampere.

Google Scholar

Jung, J. (2022). Working to learn and learning to work: research on higher education and the world of work. High. Educ. Res. Dev. 41, 92–106. doi: 10.1080/07294360.2021.2002274

CrossRef Full Text | Google Scholar

Karlgren, K., Paavola, S., and Ligorio, B. (2020a). Introduction: what are knowledge work practices in education? How can we study and promote them? Res. Pap. Educ. 35, 1–7. doi: 10.1080/02671522.2019.1677761

CrossRef Full Text | Google Scholar

Karlgren, K., Lakkala, M., Toom, A., Ilomäki, L., Lahti-Nuuttila, P., and Muukkonen, H. (2020b). Assessing the learning of knowledge work competence in higher education – cross-cultural translation and adaptation of the Collaborative Knowledge Practices Questionnaire. Res. Pap. Educ. 35, 8–22. doi: 10.1080/02671522.2019.1677752

CrossRef Full Text | Google Scholar

Kobbe, L., Weinberger, A., Dillenbourg, P., Harrer, A., Hämäläinen, R., Häkkinen, P., et al. (2007). Specifying computer-supported collaboration scripts. Int. J. Comput. Support. Collab. Learn. 2, 211–224. 2-007-9014-4 doi: 10.1007/s11412-007-9014-4

CrossRef Full Text | Google Scholar

Laakkonen, J., and Muukkonen, H. (2019). Fostering students’ collaborative learning competencies and professional conduct in the context of two gross anatomy courses in veterinary medicine. Anat. Sci. Educ. 12, 154–163. doi: 10.1002/ase.1811

PubMed Abstract | CrossRef Full Text | Google Scholar

Lakkala, M., Ilomäki, L., Mikkonen, P., Muukkonen, H., and Toom, A. (2018). Evaluating the pedagogical quality of international summer courses in a university program. Int. J. Res. Stud. Educ. 7, 89–104. doi: 10.5861/ijrse.2017.1781

CrossRef Full Text | Google Scholar

Lakkala, M., Muukkonen, H., Ilomäki, L., and Toom, A. (2020). “Framework for evaluating the pedagogical features of university courses representing collaborative knowledge work practices,” in The Interdisciplinarity of the Learning Sciences, 14th International Conference of the Learning Sciences (ICLS) 2020, eds M. Gresalfi and I. S. Horn (Nashville: International Society of the Learning Sciences), 1757–1758.

Google Scholar

Lakkala, M., Toom, A., Ilomäki, L., and Muukkonen, H. (2015). Re-designing university courses to support collaborative knowledge creation practices. Australas. J. Educ. Technol. 31, 21–536. doi: 10.14742/ajet.2526

CrossRef Full Text | Google Scholar

Markauskaite, L. (2020). Commentary: learning for knowledge work practices in the wild. Res. Pap. Educ. 35, 105–115. doi: 10.1080/02671522.2019.1677762

CrossRef Full Text | Google Scholar

Markauskaite, L., and Goodyear, P. (2017). Epistemic Fluency and Professional Education. New York, NY: Springer. doi: 10.1007/978-94-007-4369-4

CrossRef Full Text | Google Scholar

McCune, V. (2009). Final year biosciences students’ willingness to engage: teaching–learning environments, authentic learning experiences and identities. Stud. High. Educ. 34, 347–361. doi: 10.1080/03075070802597127

CrossRef Full Text | Google Scholar

Miettinen, R., and Paavola, S. (2018). “Beyond the distinction between tool and sign: objects and artefacts in human activity,” in The Cambridge Handbook of Social–Cultural Psychology. Second Edition, eds A. Rosa and J. Valsiner (Cambridge: Cambridge University Press), 148–162. doi: 10.1017/9781316662229.009

CrossRef Full Text | Google Scholar

Miettinen, R., and Virkkunen, J. (2005). Epistemic objects, artefacts and organizational change. Organization 12, 437–456. doi: 10.1177/1350508405051279

CrossRef Full Text | Google Scholar

Muukkonen, H., and Lakkala, M. (2009). Exploring metaskills of knowledge-creating inquiry in higher education. Int. J. Comput. Support. Collab. Learn. 4, 187–211. doi: 10.1007/s11412-009-9063-y

CrossRef Full Text | Google Scholar

Muukkonen, H., Lakkala, M., Lahti-Nuuttila, P., Ilomäki, L., Karlgren, K., and Toom, A. (2020). Assessing the development of collaborative knowledge work competence: scales for higher education course contexts. Scand. J. Educ. Res. 64, 1071–1089. doi: 10.1080/00313831.2019.1647284

CrossRef Full Text | Google Scholar

Näykki, P., Järvelä, S., Kirschner, P. A., and Järvenoja, H. (2014). Socio-emotional conflict in collaborative learning - A process-oriented case study in a higher education context. Int. J. Educ. Res. 68, 1–14. doi: 10.1016/j.ijer.2014.07.001

CrossRef Full Text | Google Scholar

OECD (2018). Education at a Glance 2018: OECD Indicators. Paris: OECD Publishing. doi: 10.1787/eag-2018-en

CrossRef Full Text | Google Scholar

Paavola, S., and Hakkarainen, K. (2005). The knowledge creation metaphor—An emergent epistemological approach to learning. Sci. Educ. 14, 535–557. doi: 10.1007/s11191-004-5157-0

CrossRef Full Text | Google Scholar

Paavola, S., Lakkala, M., Muukkonen, H., Kosonen, K., and Karlgren, K. (2011). The roles and uses of design principles for developing the trialogical approach on learning. Res. Learn. Technol. 19, 233–246. doi: 10.3402/rlt.v19i3.17112

CrossRef Full Text | Google Scholar

Puntambekar, S., Stylianou, A., and Goldstein, J. (2007). Comparing classroom enactments of an inquiry curriculum: lessons learned from two teachers. J. Learn. Sci. 16, 81–130. doi: 10.1080/10508400709336943

CrossRef Full Text | Google Scholar

Roth, A., Ogrin, S., and Schmitz, B. (2016). Assessing self-regulated learning in higher education: a systematic literature review of self-report instruments. Educ. Assess. Eval. Account. 28, 225–250. doi: 10.1007/s11092-015-9229-2

CrossRef Full Text | Google Scholar

Säljö, R. (2010). Digital tools and challenges to institutional traditions of learning: Technologies, social memory and the performative nature of learning. J. Comput. Assist. Learn. 26, 53–64. doi: 10.1111/j.1365-2729.2009.00341.x

CrossRef Full Text | Google Scholar

Scardamalia, M., and Bereiter, C. (2014). “Knowledge building and knowledge creation: Theory, pedagogy, and technology,” in Cambridge Handbook of the Learning Sciences, ed K. R. Sawyer (Cambridge: Cambridge University Press), 397–417. doi: 10.1017/CBO9781139519526.025

CrossRef Full Text | Google Scholar

Sfard, A. (1998). On two metaphors for learning and the dangers of choosing just one. Educ. Res. 27, 4–13. doi: 10.3102/0013189X027002004

CrossRef Full Text | Google Scholar

Splichal, J. M., Oshima, J., and Oshima, R. (2018). Regulation of collaboration in project-based learning mediated by CSCL scripting reflection. Comput. Educ. 125, 132–145. doi: 10.1016/j.compedu.2018.06.003

CrossRef Full Text | Google Scholar

Timmermans, S., and Tavory, I. (2012). Theory construction in qualitative research: from grounded theory to abductive analysis. Sociol. Theory 30, 167–186. doi: 10.1177/0735275112457914

CrossRef Full Text | Google Scholar

Vesikivi, P., Lakkala, M., Holvikivi, J., and Muukkonen, H. (2020). The impact of a project-based learning curriculum on first-year retention, study experiences, and knowledge work competence. Res. Pap. Educ. 35, 64–81. doi: 10.1080/02671522.2019.1677755

CrossRef Full Text | Google Scholar

Viswanathan, L., Whitelaw, G. S., and Meligrana, J. (2012). Evaluating the role of the project course in professional planning education and its influence on planning policy and practice. Plan. Pract. Res. 27, 387–403. doi: 10.1080/02697459.2012.673329

CrossRef Full Text | Google Scholar

Vogel, F., Wecker, C., Kollar, I., and Fischer, F. (2017). Socio-cognitive scaffolding with computer-supported collaboration scripts: a meta-analysis. Educ. Psychol. Rev. 29, 477–511. doi: 10.1007/s10648-016-9361-7

CrossRef Full Text | Google Scholar

Yin, R. K. (2014). Case Study Research. Design and Methods (5th ed). Thousand Oaks: Sage Publications.

Google Scholar