University Curricula in Evidence-Informed Decision Making and Knowledge Translation: Integrating Best Practice, Innovation, and Experience for Effective Teaching and Learning

As attention to Evidence Informed Decision Making (EIDM) and Knowledge Translation (KT) in research, policy and practice grows, so does a need for capacity enhancement in amongst evidence producers and evidence users. Recognizing that most researchers enter the professional sphere with little or no appreciation of the importance and power of EIDM, the Centre for Evidence-based Health Care at Stellenbosch University, South Africa, spearheaded the development and accreditation of a foundational course titled Evidence-Informed Decision making: The Art, Science and Complexity of knowledge translation. The curriculum draws on the principles of adult learning and effective teaching that includes integrating seven key aspects: (1) extraction of intuitive and tacit knowledge (2) autonomous knowledge generation (3) diverse perspectives (4) learning by doing (5) peer-support and critique (6) facilitator coaching and (7) constant reflection. In this paper, we reflect on these techniques in enhancing understanding and utilization of KT in advancing EIDM. The in-person short course has been offered 5 times since its launch in September 2017 with attendance by ~85 senior researchers and government officials—each of whom left the workshop with three completed outputs: a stakeholder matrix, an engagement strategy for their chosen stakeholder and a plan for evaluating the impact of their KT strategy. Interest in the course has grown considerably: (a) Requests from local institutes of research for dedicated training to their staff; (b) Incorporation into international program partner capacity enhancing strategies; (c) Publication of a book chapter designed using course content; (d) Adaptation and utilization of the templates and tools as teaching resources (e) Informing organizational stakeholder engagement strategies (f) Adaptation of the modules for conference capacity building workshops. In summary, designing courses that take into consideration adult principles of learning is not a new concept. However, effective delivery of such courses is still nascent. We found that integrating the seven aspects mentioned above, including researchers together with decision-makers in the workshops, and having an experienced facilitator is critical for effective learning. Enhancing knowledge and skills “just in time” rather than “just in case” has demonstrated increased potential for immediate relevance, uptake and sustainability.

Introduction

For countries to achieve universal coverage and equitable access to health care for their populations, difficult decisions regarding the most effective use of limited resources, against the backdrop of competing political priorities, need to be made. Given the inherent complexity of such decision-making, health policy and practice decisions can be greatly facilitated if (a) rigorous, robust, relevant, and reliable evidence is available (1–4), (b) decision-makers value and consider evidence in their policy and practice deliberations (5–9) (c) a trusting, respectful and enduring engagement between evidence producers and decision-makers exists (6, 9, 10).

While there are individuals and institutions that are valiantly addressing the first need above, the greater challenge lies in the latter two points giving rise to the field of Knowledge Translation (KT). While the definition of KT has evolved over time, we refer to the prevailing definition that “Knowledge Translation is defined as a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically-sound application of knowledge to improve the health of [the population], provide more effective health services and products and strengthen the health care system.” (11).

KT approaches have been described oftentimes by the actor initiating engagement and the direction of the engagements (12) and comprise 4 models: “producer-push” whereby researchers disseminate results of their studies to perceived and actual users; “user-pull” describes instances when decision-makers actively seek evidence leading to demand-led research; “exchange” efforts recognize collaborative linkages and interactions between producers as well as users; and “integrated” models refer to the added role of a knowledge translation platform or intermediary.

Evidence-informed Decision Making (EIDM) is therefore closely linked to the efforts of KT in that EIDM is “the process of distilling and disseminating the best available evidence from research, practice and experience and using that evidence to inform and improve public health policy and practice” (13).

The Impetus for a Course in EIDM and KT

As attention to EIDM and KT in research, policy and practice has grown, so has the need for capacity enhancement in relevant knowledge, skills and activities amongst evidence producers (e.g., researchers) and evidence users (policymakers, practitioners, media, non-governmental organizations, Civil society organizations, the public etc.). Various scoping studies have revealed an encouraging response to offering KT specific curricula. For instance, a global scoping study of KT curricula commissioned by International Development Research Centre (IDRC) Canada, in 2008 (14) identified 123 certificate, diploma and post-graduate level KT offerings. However, the differences in the distribution between Low and Middle-Income countries (LMICs) and High-Income Countries (HICs) was marked: only 6 (5%) KT curricula were created by LMICs for LMICs and 11 KT curricula were produced for/conducted in LMICs in partnership with HICs. The rest (86%) were all focused on HICs. An internal review of “courses” identified through internet web searches in 2014 by the Centre for Evidence-based Health Care (CEBHC) at Stellenbosch University in South Africa (15) identified 17 offerings that fit the search criteria of Knowledge Translation courses globally. Fourteen of the 17 courses were offered in HICs: Canada (6), UK (3), Australia (2), US (1), Netherlands (1), and Kosovo (1). EVIPNet Europe initiated a mapping of university-embedded KT curricula in 2015 (personal communication), with a focus on HICs (Europe and the Americas). The results of the mapping study are not yet available, however. While the two scoping studies mentioned above employed different search and identification criteria, courses aimed at researchers often aligned with the objectives of KT in general to:

• Enrich understanding and appreciation of the user context

• Encourage trusting relationships between researchers and research-users

• Enhance capacity of researchers to package, share, and communicate research plans as well as results in a variety of ways relevant to research-users.

While such courses were offered by institutions in a variety of formats, the IDRC study demonstrated that of the 75 offerings by universities at the time—none were African (14). To fill this gap, the IDRC supported the creation of a KT course at the Kenya Methodist University in 2011. The growing interest in KT in LMICs since then has been demonstrated by the plethora of one-off workshops through donor-funded initiatives or conferences in several countries—oftentimes with differing KT foci. Unfortunately, we were unable to find public evaluations of any of the listed courses that might have further informed our planned course in this area. The concerns about sustainability, integration and mainstreaming of KT-relevant skills and capacities amongst the next generation of researchers remain important, topical issues that urgently need to be addressed.

In response to the above, the Centre for Evidence-based Health Care (CEBHC) at Stellenbosch University, South Africa spearheaded the development and accreditation of a foundational course titled Evidence-Informed Decision making: The Art, Science and Complexity of knowledge translation, followed by modules designed for engagement with stakeholders.

Given that KT is a large and evolving field, the required skills set is equally large in breadth and depth. The course is, therefore, designed to permit participants to leave the course with not only a better understanding of the theory and science of KT but also practical tools to navigate the complexity of its implementation in different contexts.

Pedagogical Framework(s), Principles, and Competencies/Standards Underlying the Educational Activity

“Tell me, and I will forget. Show me, and I may remember. Involve me, and I will understand.” Confucius

We believe that educators need to be facilitators of learning (16)—“guides on the side”—rather than teachers or “sages on the stage” in the traditional definition. Skills in facilitation of such learning have been shown to be important elements of the learning experience (17, 18). To maximally inspire the learning experience of the course content we employed principles of Piaget's constructivism, Papert's constructionism (19) and Vygotsky's socio-constructivism (20) in the design and delivery of the course. As such the overarching pedagogical principles upon which the design and delivery of the course was based, were that learning is incremental, “developmental, experiential, and interactionist” (21) and that participants should “own their own learning” (22).

Mezirow's (23) perspective of Transformative Learning Theory (TLT) posits that adults learn through “aha moments.” As such, “Discovery” sessions or “a-ha moment activators” were applied as initiating activities for each topic in order to facilitate the learning process and to explore participants' tacit knowledge and perspectives of key concepts before formally presenting core course materials in each session. We also included deliberate opportunities for facilitated dialogue with peers in order to foster deeper reflection, critical thinking and reasoning (23, 24) among course participants.

Recognizing that researchers, policy makers, and practitioners are not tabula rasas, or blank slates, we expected each participant to approach the course with their own mental models for KT and their own expectations of the value, content, and strategies for effective outputs and outcomes. This understanding, together with Heifetz et al.'s (25) “balcony and dance floor” analogy for adaptive leadership, inspired a course design that encouraged experiential sharing and discussion, that embedded opportunities for zooming in and out of individual projects.

The CEBHC course was therefore designed in a way to dovetail the science and art of KT by drawing on principles of adult learning and capacity strengthening that integrates 7 important aspects: (1) extraction of intuitive and tacit knowledge (2) autonomous knowledge generation (3) diverse perspectives (4) learning by doing (5) peer-support and critique (6) facilitator coaching and (7) constant reflection.

Hence, the course draws upon tools that encourage participants to “own their own learning” by actively being involved in a process of meaning and knowledge construction as opposed to passively receiving information. In Stenhouse's (22) definition, this would reflect “curriculum as process,” while still encompassing elements of knowledge transmission as well as output generation.

Aim of the Paper

In this paper we describe our experience with the design and delivery of a short course in KT, aimed at LMIC participants that incorporated foundational techniques of constructivism and adult pedagogy to enhance the understanding and utilization of KT in advancing EIDM. In particular we:

• Describe the approach and content of the course curriculum

• Provide details on how we integrated key pedagogical principles from across history and disciplines as a means to deliver the course material.

• Share our reflections and that of course participants, on the implementation process and the short-term impact of the course.

Learning Environment (Setting, Students, Faculty); Learning Objectives; Pedagogical Format

Learning Environment

Setting

The course was offered over 2 days using a face-to-face approach. An outline of the course agenda is provided in Annex I. The interactionist principle of learning was engendered by limiting the course to 20 participants and seating participants in groups of 4 to 6 around circular tables. This ensured an intimate and interactive setting.

Students

There was no restriction on participants in terms of previous qualifications or seniority. The short course was open to anyone keen to learn more and/or contribute to the learning on this topic and therefore junior researchers, tenured professors, practitioners, decision-makers etc. were all welcome. Each offering of the short course was advertised across the country through social media, emails, networks, and course alumni.

Faculty

Faculty members at Stellenbosch University developed the course content, the design as well as the delivery of the course. They also served as facilitators for the various offerings.

Learning Objectives

The aim of this course was to deliver KT-specific information, tailored to participants from LMICs, using foundational techniques of constructivism and adult pedagogy to enhance the understanding and utilization of the KT information. Specific learning objectives of the course are outlined below, in Box 1.

Box 1. Specific learning objectives of the course.

At the end of the foundational course, participants are expected to be able to:

1. Appreciate opportunities, challenges and nuances around the science, art, and context complexity of KT

2. Translate appreciation for the various actors, stakeholders and their roles in the KT continuum into rigorous stakeholder analyses

3. Select and apply various KT strategies and tools appropriately

4. Consider and incorporate time, budget, HR and skills into KT plans

5. Incorporate deliberate measurement and metrics of research uptake and use

6. Design a draft KT and stakeholder engagement strategy

Pedagogical Format

As mentioned earlier, the overarching pedagogical principle upon which the design and delivery of the course was based, was that learning is incremental, “developmental, experiential and interactionist” (21) and that participants should “own their own learning.”

Each day began with energizing ice-breakers intended to minimize hierarchical beliefs and behaviors (between junior researchers and professors or researchers and clinicians/policy makers) and encourage participation, discussion, networking, and interactionist learning.

The course was divided into five key topics in line with the steps required to design a comprehensive KT and engagement strategy. These include: an overview of key concepts relevant to KT and EIDM, people and power, tools and strategies, resource considerations, and monitoring and evaluation. Each session comprised of three main components: Discovery (the “aha” moment), Concept Introduction (didatic) and Concept Application (practical).

Discovery

Interactive polls were used to understand participant values, perspectives, tacit knowledge, and assumptions. For the interactive polls we used Poll Everywhere (26), a web-based program that can be integrated directly into presentation software in order to provide real-time poll results. Poll questions included voting on various EIDM beliefs and myths, interpretation (and consequent reporting of data) etc. This posed some challenges in situations when there was unreliable internet or electricity supply. Flipcharts for the polls and voting were used as optional backups in such cases.

The simulated role-play was designed for the session on stakeholder engagement was a first step in achieving objective 2 (see Box 1). We adapted an existing case study to replicate a community meeting in a hypothetical district suffering from maldistributed health services (27). This exercise encouraged course participants to step out of their comfort zones and embody various personas (for example midwives, the district chief, patients) from the district, each with a different perspective of the health care disparities experienced, who were likely to be impacted by research and/or policy decisions proposed in the case-study. Observers and role-play participants reflected on and critiqued the simulation leading to questions about power, coalitions, roles of researchers, roles of decision-makers, values, modes of engagement, and conflict management. Participants therefore began to demonstrate appreciation of the various actors, stakeholders and their roles in the KT continuum.

During the “engagement proposal,” course participants were presented with a matching game which aimed to elicit participants' understanding of modes and methods of engaging with stakeholders.

Participants were each assigned a specific target audience (i.e., researchers, media stakeholders, policymakers, practitioners, or members of the public). They then matched the most appropriate engagement tools and strategies, from the list of 29 provided to them, with their assigned audience. This was first done independently and then with other participants assigned to the same audience in a think-pair-share exercise. The objective of the former strategy was to permit participants to judge for themselves what made sense; the objective of the latter step was to expose participants to potentially different choices by peers that may have been influenced by personal mental models, assumptions, experiences and contexts. Where there were differences in choices, participants were challenged to convince the other of their chosen tools and strategies. During the post-discovery discussion participants reflected on questions such as “What did I discover?” and “What affected the differences in my opinions with my peers?” This exercise intended to introduce participants to a variety of KT strategies, tools and their appropriate uses as well as enhance appreciation of tailoring strategies to a specific audience. They were key aspects related to objective 3 and 4 (see Box 1).

The “gallery walk” consisted of five poster size scenarios that outlined an engagement strategy; each for a different audience. Participants had 3 min to discuss how they would evaluate the impact of the engagement and knowledge translation strategy outlined in each poster before moving onto the next one in the gallery. This was the beginning of introducing concepts relevant to objective 5 (see Box 1). After a complete rotation, a group representative reported back to all participants on the poster they were currently stationed at, prompting discussion between all participants for all scenarios. Using their tacit knowledge of indicators and their new learnings of KT, participants began to connect the constructs and deliberate appropriate choices for indicators on research uptake and use. Questions such as “which indicators are critical?” and “how do you attribute change to your activities?” were important to deliberate on.

Each discovery session was followed by a reflection activity, during which the course participants explored diverse perspectives on the learnings experienced during the discovery activity. This assisted with engendering the principles of autonomous knowledge generation and experiential learning.

Concept Introduction

The discovery sessions led to enhanced interest about the topics that we complemented with formal, structured presentations on the concepts that emerged in the discussions and deliberations prior. We had a total of 5 presentations to introduce the concepts, each focusing on a specific aspect of the course but building upon the previous materials. We interwove didactic approaches throughout the short course to concretize the value, utility, scope, and structure of the concepts across objectives 1 to 6 (see Box 1). This approach was used in parallel with more interactive engagement of participants to enhance a learning-by-doing philosophy and engender more developmental learning. These were, once again, speckled with opportunities for discussion, debate and reflection.

Concept Application

The first step in “owning one's own learning” is to incorporate learned concepts into practical activities that are meaningful to each individual. To facilitate this, participants were requested to bring to the course, a specific research result or policy-relevant issue they wished to share with a wider audience. Consequently, each participant designed, planned and constructed three outputs by the end of the course: a stakeholder map for their selected projects or research studies (linked to objective 2, see Box 1), a draft engagement strategy for one particular stakeholder considered a priority (linked to objectives 3, 4, and 6, see Box 1), and a plan for evaluating the impact of their KT strategy including a variety of indicators for each of the activities outlined in the KT strategy (Objective 5).

Using a template created by the lead author, the engagement strategy for each stakeholder followed the steps from the KT engagement framework (28): the purpose for engagement, the message, the medium/forum, the messenger, the timing, the resources required and indicators to measure the effectiveness of the planned KT and engagement strategy. Detailed steps and references for the content of each have been documented elsewhere (29).

By encouraging participants to create products based on pre-identified topics relevant to their work place, we employed the principle of “just in time” learning (JIT). JIT learning creates immediate value to participants in contrast to learning topics “just in case” (JIC) that may (or may not be) relevant or useful in the future. With JIT learning, participants are able to strategically align their new knowledge to their work, implement it immediately, engage with facilitators and peers to reinforce it, and increase their chances of retaining it.

Throughout the course, deliberate opportunities for reflection, feedback and support from facilitators as well as peers were carved into the agenda to engender the importance of learning by interacting (21).

Box 2 shows the link between the various Discovery, Concept Introduction, and Concept Application sessions and the specific learning objectives of the course.

Box 2. Pedagogical approaches for delivering on the KT course objectives.

Course Evaluation

The combination of presentations, case examples, expert engagement, peer discussions, and the provision of templates for course deliverables embody the philosophy behind the training with emphasis on completed products for use “just in time” as opposed to “just in case.”

Given that this was a short course offered for professionals outside of any degree requirement, there was no requirement to offer evaluations in the forms of examinations to test knowledge or analyze student learning. However, in the spirit of ensuring the path to mastery we used four methods (outlined below in Box 3) to evaluate the extent to which we achieved the specific learning objectives of the course (see Box 1). Two evaluation methods were embedded within the course with one administered at the end. This is similar to methods used in approaches to Problem-Based Learning (PBL) (30, 31).

Box 3. Methods for evaluating participant learning and course utility.

1. Ensuring that the participants actually produced several outputs as a result of the training (stakeholder analyses for objective 2, create a fully resourced stakeholder engagement and KT strategy for Objectives 3 and 4 and 6, and develop indicators to measure the various activities outlined in the KT strategy for Objective 5) throughout the course. Evaluation of the outputs through peer as well as facilitator feedback contributed to enhanced learning and mastery

2. Embedding structured time for reflections and discussions throughout the course on all issues relevant to objective 1–6 so as to provide facilitators with real-time feedback on participant understanding, challenges and perspectives. This allowed for any course correction, immediate clarifications, and peer-peer learning

3. Participant evaluations of the course that asked specific questions with respect to achievement of the specific learning objectives as well as the utilization of the key pedagogical principles that we outline. At the end of each course, participants completed an online, anonymized evaluation form comprised of Likert scale and open-ended questions (see Annex II). The Likert scale questions assessed participants' satisfaction with the content of the presentations, the facilitators and the organization of the training as well as the effectiveness of the practical, peer, and discovery sessions. Feedback from the evaluation forms led to improvements and adaptations to subsequent offerings of the course

4. Informal feedback and reflections over time with previous participants about the value of the course provided anecdotal evidence of course relevance and utility

Facilitators debriefed after each course offering to reflect on new ideas, challenges to previous ideas, innovative examples, and participant feedback to refine and enhance the content as well as delivery of each subsequent version of the course. The course continues to evolve as a result.

Results to Date

The in-person short course has been offered 5 times since its launch in September 2017. Majority of the participants were academic researchers from universities or research councils (55%) (Table 1). Most participants (n = 71) were from South Africa with nine delegates from other countries (Table 2).

TABLE 1

Table 1. Distribution of participants by organization.

TABLE 2

Table 2. Distribution of participants by Country.

Course Outputs

Each participant left the course with three completed outputs: a stakeholder matrix, an engagement strategy for their chosen stakeholder and a plan for evaluating the impact of their KT strategy relative to their chosen stakeholders. Specific templates for designing the KT strategy were developed by the principal author (NJ).In this way the specific learning objective 6 and the pedagogical framework around Concept application were met. The design of the course provided opportunity for constant reflection through group discussion, facilitator input and peer feedback contributing to iterative refinement of each of these course outputs.

Reflections on the Learning Style/Pedagogy

Participants were satisfied with the organization, structure, and facilitation method of the courses.

Figure 1 provides an overview of respondent reflections on the various aspects of the course. None rated any of the aspects as “poor.”

FIGURE 1

Figure 1. Participant evaluation of the pedagogical approaches to the course (2017–2019).

The discovery sessions elicited the most energy and excitement as they sparked the desired “a-ha” moments for participants. They enjoyed the challenge of testing their assumptions and preconceived notions to interrogate a concept before diving into the theory of it. These sessions allowed the facilitators to discern tacit knowledge and assumptions from which to springboard discussions and learning so as to equalize the knowledge base. Participant reflections are noted in the quotes below:

“[The stakeholder simulation] was a great, creative and exciting session - very different to my usual work.” (November 2017).

“[The most enjoyable/useful part of the workshop was] the [gallery walk]discovery session on metrics.” (March 2018).

The majority of new content was delivered through presentations, complemented by discussions and deliberation as mentioned earlier. Different aspects resonated with participants, depending on their previous knowledge and experience of a topic. However, new ways about thinking of some of the topics was also highlighted:

“The session on stakeholder mapping was very useful as it provides important frameworks for conducting this highly important step in knowledge translation.” (September 2017).

Putting learnings into immediate practice allowed participants to reflect on the relevance and realities of some of the theoretical concepts and it also allowed them to leave the sessions with outputs that were immediately useful. Participant evaluations indicate a 97% satisfaction score (Figure 1) for this aspect of the course with reflections on the immediate utility of the products:

“[The most enjoyable/useful part of the workshop was] the practical session, where you were challenged to apply this to your current work. It was great it was not a theoretical course only.” (November 2017).

Reflections on Peer Learning

The power of peer engagement and contributions was evident in the energy that was created when such opportunities arose. In Figure 1, 95% of participants evaluated the peer sessions as either “excellent” or “good.” Participants often got excited about the contributions they were able to make in their peers' outputs and immensely grateful for their peers' contributions to their own work:

My favorite aspect of the course were the peer sessions. They enabled me to get other people's practical insights on the research project I'd like to undertake. The fact that peers' experiences were so diverse and different from mine created an opportunity to consider other approaches I might not have thought about. (November 2017).

Reflections on Facilitation and Content

Figure 2 indicates participant appraisal of facilitation and content of the course.

FIGURE 2

Figure 2. Participant evaluation of the facilitation and content of the course.

Facilitation using the key pedagogical approaches was an important element of the course. We used participant feedback to continue to improve the delivery and content of the course as well as the time allocated to the various sessions. We also ensured that the content of the course was contextually relevant, something that participants contributed to as well as appreciated:

“I thought the facilitation was grounded and realistic - often at workshops like this, I find people running through their material and formulaic responses without the relevant nuance and flexibility to audience etc. but I felt it was well handled, insightful, and useful.” (September 2017).

The various forms of our evaluations provided us with positive feedback from participants on the value of incorporating their learnings from the course into their own work environments. Informal conversations with, and impromptu emails from, several participants indicate that they have actively integrated the principles, learnings, and tools into their programs and projects in order to enhance relationships and engagement with their identified stakeholders. The courses have also led to an informal network of colleagues across disciplines sharing their experiences of KT strategy implementation and use.

“It is almost disturbing to think that this kind of training is not a more standardized tool being used in research settings, because it would save some time in the end. However, I'm glad that I was able to attend and learn skills that would enable me to refocus my workload in a more constructive way.” (September 2017).

“This was a great course and needs to be incorporated into academic curricula for all researchers!” (November 2017).

“It was interesting to note that the course could be applied to projects at conception stage and completed projects requiring implementation.” (March 2018).

Constructive Feedback

The inaugural course was 3 days long. Following constructive feedback from the evaluation forms, to reduce the time dedicated to practical sessions, we scaled the course back initially to 2 and a half days and finally to 2 days. Overall, the value for such an offering was evidenced by interest in the course (or modules within it) growing considerably as shown in Box 4:

Box 4. Evidence of increased demand for a course on EIDM and KT (2017–2019).

a) Requests from local institutes of research for dedicated training for their staff (South African Medical Research Council, Health Systems Trust)

b) Incorporation into international program partner capacity enhancing strategies (Future Health Systems, 2019, Collaboration for Evidence-Based Healthcare in Africa (CEBHA+), 2018, Bill and Melinda Gates Institute for Population and Reproductive Health, 2019)

c) Publication of a book chapter formulated using course content (29)

d) Adaptation and utilization of the templates and tools as teaching resources (Stellenbosch University Centre for Rehabilitation Studies, Stellenbosch University Centre for Rehabilitation Studies Department of Global Health)

e) Informing organizational stakeholder engagement strategies (Cochrane South Africa)

f) Adaptation of the modules for conference capacity building workshops (Cochrane South Africa, 2019)

Informal conversations with, and impromptu emails from, several participants indicate that many have actively integrated the principles, learnings and tools into their programs and projects in order to enhance relationships and engagement with their identified stakeholders. The courses have also led to an informal network of colleagues across disciplines sharing their experiences of KT strategy implementation and use.

Discussion on the Practical Implications, Objectives and Lessons Learned

Designing courses that take into consideration adult principles of learning is not a new concept. However, effective delivery of such courses is still nascent. The strength of this paper is the description of the experience of incorporating key principles of adult pedagogy to design and deliver a short course on Evidence-Informed Decision making and Knowledge Translation, in South Africa, where documented evidence is limited. Results from 5 iterative versions of the course demonstrated that integrating seven key pedagogical strategies (1) extraction of intuitive and tacit knowledge (2) autonomous knowledge generation (3) diverse perspectives (4) learning by doing (5) peer-support and critique (6) facilitator coaching and (7) constant reflection, including researchers as well as decision-makers in the short courses, and having an experienced facilitator are valued elements of effective learning.

Demand for KT knowledge, skills and activities is growing as demonstrated by the requests received as well as by the plethora of KT courses on offer (14, 15, 32). If the next generation of researchers need to be better prepared to engage with a variety of actors and stakeholders (33–35), it is imperative that these skills be embedded into Master and Doctoral training programs which would enhance ownership as well as sustainability of the courses (36). Skills in EIDM KT have been recognized as part of the competencies needed for health professionals and researchers (37–39) but have yet to be designed and integrated into training programs. We therefore promote the enhancement of graduate curricula to match industry needs and equip graduate with the relevant skills and competencies. One may argue that a standalone course such as this one could be one option for embedding such skills. However, students with little understanding of the importance of KT skills may not necessarily opt to take such a course unless required (40). This was clear in our discussions with participants who were seasoned researchers that indicated that they hadn't realized the importance of these skills; suggesting that learning about the utilization aspect of research—and not just the generation of it—should be a fundamental part of research training.

We posit that the skills embodied in this course should be embedded across a variety of graduate programs so that students learn to employ relevant techniques for stakeholder engagement and integrated knowledge translation in whatever research they pursue. We also recommend that the programs should consider employing or adapting the experience of our pedagogical approach in order to deliver knowledge and practical application of these skills. However, this would require architectural accommodations for classes that do not conform to auditorium style lectures (41); it would require all faculty to understand the principles of KT and also embody a “learning paradigm” in contrast to an “instruction paradigm” (42, 43); it would require students to be comfortable with owning their learning (44); and it would require an empathetic atmosphere for students with more introverted dispositions.

While most of the KT courses on offer use a face-to-face approach, there are a few that are delivered online and others that utilize a blended learning approach, incorporating both face-to-face and online learning (14, 15, 32). With constant innovation in teaching and learning, it is possible that courses such as these could evolve into a more blended approach (45) over time or provided entirely online similar to a course on Evidence-Based Health Care (46). However, experience with the intensity of this course as well as the value for the peer interactions and practical aspects, indicate that a blended approach may compromise many of the seven key principles that, together, contributed to the innovation in this approach. While each of the seven principles have already been demonstrated, individually, as effective, we combined them in order to capitalize on their individual effectiveness for enhanced learning. This is the first instance, that we are aware of, where such an innovative approach has been perused and we argue for continued experimentation in this way.

Limitations and Constraints

The reflections in this paper result from the facilitators' own assessment that was informed by their experiences, their reflections, and feedback from participants. To truly understand the long-term impact of courses such as these, it would be important to design pre-post studies that are able to capture the true learning and retention of skills and knowledge (44). One may argue that other study designs could consider two versions of the course offered simultaneously (47, 48)—one more traditional, the other incorporating these seven key elements—to distinguish empirically whether one is more effective than the other. However, given the fact that the curriculum was designed with evidence of effective strategies, it would seem counter-intuitive and perhaps even wasteful to test whether proven techniques are indeed best-practice. What would be important is to see how best to export these techniques and experiences into other educational arenas. An independent external evaluation of the course over time may be able to determine the combined effectiveness but that was beyond the scope of this paper.

Conclusion

In conclusion, enhancing knowledge and skills “just in time” rather than “just in case” has demonstrated increased potential for immediate relevance, uptake and sustainability. Therefore, integrating best practice, innovation and experience can greatly enhance effective teaching and learning in the field of KT, and perhaps more broadly.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Author Contributions

NJ and TY conceived the course. NJ and LN designed the curriculum, the tools, and the templates. NJ delivered all the short courses and workshops to date and led the drafting and writing of the manuscript. LH and LN co-facilitated some of the workshops. All authors reviewed and approved the final version.

Funding

The offering of the course was supported by the SUCCEED project; a grant from the President's Emergency Plan for AIDS relief (PEPFAR) through Centres for Disease Control (CDC) under the terms of GH00153; without restrictions on the curriculum content or approach. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funder.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

1. Harris JK, Allen P, Jacob RR, Elliott L, Brownson RC. Information-seeking among chronic disease prevention staff in state health departments: use of academic journals. Prev Chronic Dis. (2014) 11:e138. doi: 10.5888/pcd11.140201

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Dobbins M, Thomas H, O'brien MA, Duggan M. Use of systematic reviews in the development of new provincial public health policies in Ontario. Int J Technol Assess Health Care. (2004) 20:399–404. doi: 10.1017/S0266462304001278

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Yousefi-Nooraie R, Dobbins M, Brouwers M, Wakefield P. Information seeking for making evidence-informed decisions: a social network analysis on the staff of a public health department in Canada. BMC Health Serv Res. (2012) 12:118. doi: 10.1186/1472-6963-12-118

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Dobbins M, Jack S, Thomas H, Kothari A. Public health decision-makers' informational needs and preferences for receiving research evidence. Worldviews Evid Based Nurs. (2007) 4:156–63. doi: 10.1111/j.1741-6787.2007.00089.x

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Innvær S, Vist G, Trommald M, Oxman A. Health policy-makers' perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. (2002) 7:239–44. doi: 10.1258/135581902320432778

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Oliver K, Innvær S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Services Res. (2014) 14:2. doi: 10.1186/1472-6963-14-2

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Rosenbaum SE, Glenton C, Wiysonge CS, Abalos E, Mignini L, Young T, et al. Evidence summaries tailored to health policy-makers in low- and middle-income countries. Bull World Health Organ. (2011) 89:54–61. doi: 10.2471/BLT.10.075481

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Jessani NS, Siddiqi S, Babcock C, Davey-Rothwell M, Ho S, Holtgrave DR. Factors affecting engagement between academic faculty and decision-makers: learnings and priorities for a school of public health. Health Res Policy Syst. (2018) 16:1–15. doi: 10.1186/s12961-018-0342-9

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Jessani NS, Babcock C, Siddiqi S, Davey-Rothwell M, Ho S, Holtgrave DR. Relationships between public health faculty and decision-makers at four governmental levels: a social network analysis. Evid Policy J Res Debate Pract. (2018) 14:499–522. doi: 10.1332/174426418X15230282334424

CrossRef Full Text | Google Scholar

10. Ross S, Lavis J, Rodriguez C, Woodside J, Denis JL. Partnership experiences: involving decision-makers in the research process. J Health Serv Res Policy. (2003) 8(Suppl 2):26–34. doi: 10.1258/135581903322405144

PubMed Abstract | CrossRef Full Text | Google Scholar

11. CIHR. About Knowledge Translation. (2016). Available online at: http://www.cihr-irsc.gc.ca/e/29418.html#2 (accessed October 07, 2017).

Google Scholar

12. Lavis JN, Lomas J, Hamid M, Sewankambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ. (2006) 84:620–8. doi: 10.2471/BLT.06.030312

PubMed Abstract | CrossRef Full Text | Google Scholar

13. National Collaborating Centre for Methods and Tools. A Model for Evidence Informed Decision-Making in Public Health. (2012).

Google Scholar

14. Best A, Bitz J, Larsen C, Clelford S, Watters N. Knowledge Translation Training Opportunities: Global Scoping Study Insights. (2010).

Google Scholar

15. Zunza M, Wiysonge CS. Knowledge Translation Online Inventory Report (2014).

Google Scholar

16. Saban A, Kocbeker BN, Saban A. Prospective teachers' conceptions of teaching and learning revealed through metaphor analysis. Learn Instruct. (2007) 17:123–39. doi: 10.1016/j.learninstruc.2007.01.003

CrossRef Full Text | Google Scholar

17. Young T, Rohwer A, van Schalkwyk S, Volmink J, Clarke M. Patience, persistence and pragmatism: experiences and lessons learnt from the implementation of clinically integrated teaching and learning of evidence-based health care – a qualitative study. PLoS ONE. (2015) 10:e0131121. doi: 10.1371/journal.pone.0131121

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Schwartz RM. The Skilled Facilitator: Practical Wisdom for Developing Effective Groups, 1st ed. A Joint Publication in the Jossey-Bass Management, Public Administration, and Nonprofit Sector Series. San Francisco, CA: Jossey-Bass, Inc. (1994).

Google Scholar

19. Harel I, Papert S (editors). Constructionism. Westport, CT: Ablex Publishing Co. (1991).

Google Scholar

20. Vygotsky LS. Mind in Society: The Development of Higher Psychological Processes. Cambridge, MA: Harvard University Press (1978).

Google Scholar

21. Constructivism (s): Shared roots crossed paths multiple legacies. Constructionism (2010).

Google Scholar

22. Stenhouse L. An Introduction to Curriculum Research and Development. London: Heinemann (1975).

Google Scholar

23. Mezirow J. Fostering Critical Reflection in Adulthood: A Guide to Transformative and Emancipatory Learning. San Francisco, CA: Jossey-Bass (1990).

Google Scholar

24. Mazur E. Peer Instruction: A User's Manual, 1st ed. Pearson Education Limited (1996).

Google Scholar

25. Heifetz RA, Grashow A, Linsky M. The Practice of Adaptive Leadership: Tools and Tactics for Changing Your Organization and the World. Boston, MA: Harvard Business Press (2009).

Google Scholar

26. PollEverywhere. PollEverywhere. (2019). Available online at: www.polleverywhere.com (2019).

Google Scholar

27. McIntyre D. Case Study: Promoting Equitable Access to Health Care for Households. (2011).

Google Scholar

28. Bennett G, Jessani N. The Knowledge Translation Toolkit: Bridging the Know-Do Gap: A Resource for Researchers. New Delhi1: SAGE (2011).

Google Scholar

29. Mash B, Jessani N, Nicol L. Reaching decision-makers and achieving social impact with your research. In: Goodyear-Smith F, Mash R, editors. How To Do Primary Care Research, 1st ed. Boca Raton, FL: CRC Press (2018). p. 277–81.

Google Scholar

30. Cheong F. Using a problem-based learning approach to teach an intelligent systems course. J Inform Technol Educ. (2008) 7:47–60. doi: 10.28945/178

CrossRef Full Text | Google Scholar

31. Delisle R. How to Use Problem-Based Learning in the Classroom. Alexandria, VA: Association for Supervision and Curriculum Development (1997).

Google Scholar

32. Public Health Agency of Canada. Evidence-Informed Decision Making: Guide to Learning and Training Opportunities. (2011) Available online at: https://www.canada.ca/en/public-health/services/chronic-diseases/reports-publications/diabetes/evidence-informed-decision-making-guide-learning-training-opportunities.html (accessed June 2019).

Google Scholar

33. Brownson RC, Fielding JE, Green LW. Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research. Annual Rev Public Health. (2018) 39:27–53. doi: 10.1146/annurev-publhealth-040617-014746

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Stamatakis KA, Norton WE, Stirman SW, Melvin C, Brownson RC. Developing the next generation of dissemination and implementation researchers: insights from initial trainees. Implement Sci. (2013) 8:29–5908. doi: 10.1186/1748-5908-8-29

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Kothari A, MacLean L, Edwards N. Increasing capacity for knowledge translation: understanding how some researchers engage policy makers. Evid Policy J Res Debate Pract. (2009) 8:1–21. doi: 10.1332/174426409X395402

CrossRef Full Text | Google Scholar

36. Centre for Higher Education Transformation (CHET). Capacity Building Initiatives in Higher Education. (2002).

PubMed Abstract | Google Scholar

37. Volmink JA, Bruce J, de Holanda Campos H, de Maeseneer J, Essack S, Green-Thompson L, et al. Reconceptualising Health Professions Education in South Africa. (2018). doi: 10.17159/sajs.2018/a0281

CrossRef Full Text | Google Scholar

38. Council on Education for Public Health. Accreditation Criteria: Schools of Public Health and Public Health Programs. (2016).

PubMed Abstract | Google Scholar

39. HEFCE. Research Excellence Framework: Second Consultation on the Assessment and Funding of Research. (2009).

Google Scholar

40. Celletti F, Reynolds TA, Wright A, Stoertz A, Dayrit M. Educating a new generation of doctors to improve the health of populations in low- and middle-income countries. PLoS Med. (2011) 8:e1001108. doi: 10.1371/journal.pmed.1001108

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Lambert C. Twilight of the lecture: the trend toward “active learning” may overthrow the style of teaching that has ruled universities for 600 years. Harvard Magazine. (2012).

Google Scholar

42. Chickering AW, Gamson ZF. Seven principles for good practice in undergraduate education. AAHE Bull. (1987) 39:3–7.

Google Scholar

43. Barr RB, Tagg J. From teaching to learning - a new paradigm for undergraduate education. Change Mag High Learn. (1995) 27:12–26. doi: 10.1080/00091383.1995.10544672

CrossRef Full Text | Google Scholar

44. Fagen AP, Crouch CH, Mazur E. Peer instruction: results from a range of classrooms. Phys Teach. (2002) 40:206. doi: 10.1119/1.1474140

CrossRef Full Text | Google Scholar

45. Garrison DR, Vaughan ND. Blended Learning in Higher Education: Framework, Principles, and Guidelines. San Francisco, CA: Jossey-Bass (2008).

Google Scholar

46. Rohwer A, Motaze NV, Rehfuess E, Young T. E-learning of evidence-based health care (EBHC) in healthcare professionals: a systematic review. Campbell Syst Rev. (2017) 13:1–147. doi: 10.4073/csr.2017.4

CrossRef Full Text | Google Scholar

47. Turnip B, Wahyuni I, Tanjung Y. The effect of inquiry training learning model based on just in time teaching for problem solving skill. J Educ Pract. (2016) 7:177–81.

Google Scholar

48. Thacker B, Kim E, Trefz K, Lea SM. Comparing problem solving performance of physics students in inquiry-based and traditional introductory physics courses. Am J Phys. (1994) 62:627–33. doi: 10.1119/1.17480

CrossRef Full Text | Google Scholar

ANNEX I

Annex I. Course agenda.

ANNEX II

Annex II. Participant evaluation form for a course on EIDM and KT (2017–2019).