Assessing the Quality of Knowledge for Adaptation–Experiences From Co-designing Climate Services in Sweden

Adaptation to climate change is becoming more urgent, but the wealth of knowledge that informs adaptation planning and decision-making is not used to its full potential. Top-down approaches to knowledge production are identified as one important reason for the gap between science and practice and are criticized for not meeting the needs of intended users. In response to this challenge, there is a growing interest in the creation of user-oriented and actionable climate services to support adaptation. At the same time, recent research suggests that greater efforts are needed to evaluate the effectiveness of knowledge co-production processes and the best criteria by which to gauge the quality of knowledge outcomes, while also considering different stakeholder perspectives. This paper explores these issues through a critical assessment of the quality of knowledge for adaptation generated from a climate services co-design process in two case studies in Sweden. The study draws on experiences from a 5-year research collaboration in which natural and social science researchers, together with local stakeholders, co-designed climate services to support climate adaptation planning and decision-making. The well-established knowledge quality criteria of credibility, legitimacy, saliency, usability, and usefulness remain relevant, but are not sufficient to capture factors relating to whether and how the knowledge actually is applied by climate change adaptation planners and decision-makers. We observe that case-specific circumstances beyond the scope of the co-design process, including the decision-making context as well as non-tangible outcomes, also play crucial roles that should be accounted for in the knowledge assessment processes.

Introduction

Despite a strong increase in climate change impact and adaptation research and practice over the past decade, the wealth of knowledge and experience is seldom used to its full potential in climate change adaptation planning and decision-making. This gap between research and action (Klein and Juhola, 2014; Palutikof et al., 2019) signals that there is a lack of actionable knowledge to support adaptation decision-making (Ernst et al., 2019; Mach et al., 2020). Climate services have emerged as a response to the urgent need for more context-specific, user-driven and decision-oriented climate information that can better support decision-making and action on climate change (Vaughan and Dessai, 2014; Daniels et al., 2020). In an ambition to close the science-policy-action gap that appears to hinder climate-resilient development, a growing number of social and behavioral science studies have identified barriers to an effective uptake of climate information in stakeholder assessments and in policy- and decision-making (e.g., Vulturius et al., 2020a,b).

One key barrier identified is the conventional top-down approach to adaptation (e.g., Dessai and Hulme, 2004), or what is commonly framed in terms of supply-oriented or supply-driven climate services (Lourenço et al., 2016; Daniels et al., 2020). Climate information providers have been shown to have incomplete understanding of decision contexts (McNie, 2007) and narrow perceptions of user types (Porter and Dessai, 2017). There is also inadequate attention to the wider decision-making context (Vincent et al., 2018), which involves many pressing concerns beyond future climate impacts, and is shaped by competing interests, decision-making cultures and legitimacy claims (Dilling and Lemos, 2011). Moreover, there is empirical evidence that a scientific approach to, and differential understanding of, uncertainty and technical information may confuse rather than help decision-makers (Patt and Dessai, 2005; Porter and Dessai, 2017; Christel et al., 2018).

Other scholars highlight that relationships between providers and users are often weak or ad hoc (Lemos and Morehouse, 2005; Lowrey et al., 2009; Brasseur and Gallardo, 2016). A related concern is that scientific information and its providers lack credibility, legitimacy and trust in the eyes of users (Cash D. W. et al., 2003; Moser and Ekstrom, 2010); decision-makers also underestimate the importance and value of climate information (Cortekar et al., 2017). Other barriers to user-oriented, decision-driven climate services are inflexible institutional rules (Dilling and Lemos, 2011); a mismatch in spatial, institutional and temporal scales of research vis-à-vis decision-making and policy timescales (Bruno Soares and Dessai, 2016; Vincent et al., 2018); as well as underestimation of the value of integrating different knowledge types, from scientific to indigenous (Lemos et al., 2012).

To overcome the challenges of climate information for policy-making and action, recent studies increasingly advocate a transdisciplinary knowledge integration approach (Daniels et al., 2020) where “.researchers and knowledge users meaningfully interact to co-create knowledge that is actionable in decision-making” (Mach et al., 2020, 30). Such an approach has been shown to be useful not only for adaptation decision-making (Vaughan and Dessai, 2014), but for fostering mutual understanding and learning, enhancing the perceived saliency, credibility, and legitimacy of research outcomes; empowering users, motivating them, and increasing their sense of ownership; building trust, creating networks, and boosting institutional capacity (Bremer et al., 2019; Cvitanovic et al., 2019; Gerger Swartling et al., 2019; Schneider et al., 2019; Daniels et al., 2020)

However, it has been challenging to scale up knowledge co-production, learn from practice, and improve approaches because of a lack of reflection and clarity on how the concept is interpreted and applied (Norström et al., 2020); even the terminology is inconsistent. As a first step, there is a need for increased reflexivity and transparency among scholars adopting co-production approaches about how and when they should be used (Bremer and Meisch, 2017; Jagannathan et al., 2020); as well as how to move beyond learning within projects to capture lessons learned across contexts (Lang et al., 2012).

There is a growing literature on achieving high-quality knowledge for adaptation, and this has highlighted the need to better understand how to evaluate the effectiveness of knowledge co-production processes, and what criteria are best used to gauge the quality of outcomes. In this paper we start from the notion of adaptation as a process of continuous learning to build our understanding of a changing climate and adapt accordingly. That, in turn, requires high-quality knowledge to guide effective adaptation action. The aim of this study is to critically assess the perceived quality of knowledge generated from a climate services co-design process to inform adaptation planning and decision-making (i.e., adaptation knowledge), based on how well it meets currently accepted principles of adaptation knowledge quality. The research questions are:

1. To what extent does the adaptation knowledge meet different quality criteria?

2. What factors in the co-design process contributed to the resulting adaptation knowledge meeting those criteria?

We adopt three quality criteria—credibility, legitimacy, and saliency—developed by Cash D. et al. (2003) in studies to identify enabling conditions for high-quality knowledge generation. Credibility refers to the trustworthiness of the knowledge as well as its “scientific plausibility and technical adequacy” (Cash D. W. et al., 2003, 4). As noted by Lemos and Morehouse (2005), the process by which the knowledge is produced is important, as practitioners rarely are able to assess the quality of the information per se. Legitimacy denotes the fairness of the process from a “political and procedural” perspective—that is, that all relevant stakeholders were consulted and that the knowledge is perceived as unbiased (Cash D. et al., 2003; Norström et al., 2020). Saliency generally denotes the relevance to user needs (Clifford et al., 2020; Norström et al., 2020). Two other terms describe closely related criteria: usefulness–whether the knowledge and information are provided at temporal and spatial scales that match users' practices and needs–and usability–whether users can actually access and use the information as it was provided (e.g., online or on paper, in English or in the local language, in complex scientific terms or simple wording) (Lemos and Morehouse, 2005).

To structure our analysis, we also apply an evaluative framework for the co-production of usable climate science, developed by Wall et al. (2017). It is useful for our study because it captures the contributions of a set of indicators associated with the quality of knowledge co-production, covering different components of the process. The indicators include context-related factors (i.e., input and external factors) that capture “preexisting conditions that may influence researchers' and stakeholders' ability to engage in the co-production of science and ultimately use the information” (Wall et al., 2017, 100). In summary, inputs refer to project setup and the various skills, resources and capacities that both researchers and stakeholders bring into the process, whereas external factors refer to circumstances outside the process, including aspects such as staff turnover, political will, and financial resources. Other factors in the framework relate to the process, including timing and level of stakeholder engagement, frequency of meetings, etc. Finally, there are three factors—outputs, outcomes and impacts—that gauge different aspects of the results of the process. Outputs denote the concrete products of the process (e.g., peer-reviewed articles or technical reports) and their delivery and dissemination. Outcomes, the main focus of this paper, involve the actual knowledge produced, evaluated by its perceived credibility, legitimacy, saliency and usability. Impacts refers to the actual use of the results, which includes inter alia contributing to problem understanding, instrumental use, confirmational use, motivational use, and factual use, and how it eventually informs adaptation planning and action.

In this paper we start from perceived outcomes of the co-design process and draw on the framework by Wall et al. (2017) to identify components that have been critical to the achievement of the adaptation knowledge quality criteria.

Materials and Methods

Description of Case Studies

This paper builds on work carried out in the project HazardSupport, which ran from 2015 to 2020. The project aimed to develop a new, collaborative method for tailoring information about how climate change affects natural hazards, in order to inform adaptation decisions while also generating new scientific knowledge for adaptation for broader dissemination. The study involved providers, intermediaries and users of climate services from the Swedish Meteorological and Hydrological Institute (SMHI) and Stockholm Environment Institute (SEI), as well as municipal officers from Karlstad and the City of Stockholm. HazardSupport employed a co-design process that included focus groups meetings, workshops, interviews, and meetings (Figure 1).

FIGURE 1

Figure 1. Overview of HazardSupport co-design process.

The number of case study participants varied over the course of the co-design process (Table 1). In total nine municipal officers were engaged in Karlstad Municipality and seven officers were engaged in City of Stockholm. Initially a larger group of participants representing different areas of work were involved, including: technical services and property management, urban planning and building, and fire and rescue services (Karlstad) and; city development, urban development and management, property management, and environmental administration (Stockholm). The rationale was to include a range of perspectives and experiences in the discussions and thereby increase the potential for learning within the group, as well as to ensure the robustness of the final results (e.g., Bremer and Meisch, 2017). However, as the project evolved, a smaller group of key individuals were actively engaged in the project.

TABLE 1

Table 1. Overview timing of meetings and number of participants.

In Sweden, municipalities play an important role in planning and implementing adaptation measures. Since 2018 they have been obliged to consider climate risks, and how to minimize or eliminate these risks in the built environment (Government bill, 2017/18:163). Consequently, Swedish municipalities have advanced their adaptation work in recent years (Matschke Ekholm and Nilsson, 2019), yet, municipalities also report that they lack planning and/or decision-support (Sjöberg et al., 2019). The City of Stockholm and Karlstad Municipality are among the forerunners in Sweden, and due to their exposure and vulnerability to water-related hazards, they have comparatively long experience with adaptation.

City of Stockholm Case Study

The adaptation challenge in the Stockholm case study related to the city's rapid growth and urgent development needs. The official target of building 140,000 homes by 2030 (City of Stockholm 2018) is forcing the city to densify and expand. As identified by municipal officers during the initial phase of HazardSupport, this development goal might have implications for the vulnerability of the city to climate change. The city was particularly interested in the use of green infrastructure as a climate adaptation measure. While green infrastructure can be used to address several climate hazards, its role in heat stress mitigation was the focus in this project.

A key question for Stockholm has been to further understand how the city can develop while ensuring that it adapts to current and future climate risks. The main objective of the case study was to investigate (i) how the urban climate will be affected by the expansion and densification of the city, and (ii) the potential of green infrastructure to reduce heat stress during warm summer days.

For this purpose, a dynamic downscaling weather forecast process (Amorim et al., 2020; Gidhagen et al., 2020) was applied to estimate future summer temperatures over Stockholm, based on the development plans for 2030 and 2050. These two scenarios were defined by SMHI in cooperation with representatives from the Municipality.

Model results revealed a warming effect occurring mostly over the urbanized area, closely connected to the expansion/densification pattern, and not widespread over the entire city. In fact, the already dense city center did not show significant changes in air temperature. In terms of magnitude, our results showed a local maximum warming of 1.35°C as an average for the entire summer due to urbanization alone (before accounting for climate change), and the number of hot days per year could increase by 10 up to in 2050. For further details, see Amorim et al. (2020).

Parks showed a cooling effect of a magnitude equivalent to the city's urban heat island. This means that vegetation can help counteract, locally, the human-induced warming of the urban atmosphere, with benefits to the thermal comfort of city dwellers. The case study concluded that in conjunction with urbanization, city planning should prioritize measures that increase access to nature areas, including the connection of public urban green spaces through green corridors. This is particularly relevant for vulnerable groups, such as elderly or people who are ill. Lastly, in Nordic cities, such climate-sensitive planning should account not only for the warmest days of the year, but also for the cold and dark season, when sunlight is most desired.

Karlstad Municipality Case Study

In the Karlstad case study, the adaptation challenge in focus was flood protection for the Skåre area, in northern Karlstad. Skåre is an attractive residential area with plans for densification. Situated in the river delta of the large snow-fed Klarälven river, which connects to the nearby lake Vänern, and with the tributary Skårenoret running through the area, Skåre is today subject to multiple flood hazards, including the spring flood of Klarälven and cloudbursts. The issue of flood protection in Skåre reached a critical point in 2016, when flood risks led the County Administrative Board to reject a densification plan for Skåre proposed by the Municipality. In order to continue development in the area, Karlstad initiated a flood defense program for Skåre to investigate and implement a comprehensive flood protection solution, primarily a flood barrier.

To mitigate the threat of flood hazards in the Skåre area, Karlstad Municipality has been investigating the potential for building a flood barrier along the western shore of the Klarälven. The measure calls for studies of potential adverse effects of the flood barrier, such as changes in flood hazards from the Skårenoret inside the planned flood barrier, including cloudbursts evacuating through the Skårenoret.

To meet this need, SMHI used information about current local cloudburst intensity and frequency to gauge the magnitude of the cloudburst hazard for identified durations of one and six (based on Olsson et al., 2017, 2019) and created model simulations of peak flood levels. The analysis concluded that the flood barrier would not increase flood risk from cloudbursts, as any additional water in the Skårenoret could be evacuated with pumping stations near the flood barrier.

For single hazards, the flood barrier would thus be mainly beneficial. However, the multi-hazard nature of Skåre prompted further investigations. The focus was on co-occurrence of extreme floods in river Skårenoret together with a cloud burst. The topic was addressed by looking at the seasonality of the extremes, as well as the meteorological conditions where the extremes occur. The records were limited to about 20 years, but the available data did not show any significant co-occuring events, since the main cloud burst season peaks in July–August, while peak floods in Skårenoret occur in late autumn. The preliminary conclusion was thus that the co-occurrence of extremes is unlikely. However, the analysis calls for longer time series to set appropriate return levels for such events.

Method (Co-design Process)

The co-design process was facilitated by three members of the research team with experience of knowledge co-production processes, who had the dedicated role to act as intermediaries between case study stakeholders and climate researchers.

The first phase of the project aimed at engaging relevant stakeholders within the case studies (see section Description of Case Studies) and scoping their current use of and need for tailored climate change impacts information to guide future adaptation planning and decision-making. This included exploring adaptation challenges and the institutional and decision-context as a basis for co-defining questions to address over the course of the project. This was achieved through initial meetings with key contact persons and two sequent focus group meetings with a larger group of stakeholders in each case study. Both focus group meetings were open in character and applied different participatory techniques such as brainstorming exercise and maps to structure the discussions. A survey with free text answers was also conducted with the participants and researchers to gain deeper insight into their expectations of the project and as well as its final outputs and results. Further, semi-structured interviews were completed via phone with a majority of the participants and researchers.

The second project phase focused on refining problem definitions by co-exploring case studies' specific information needs, for example, regarding spatial and temporal scales and other parameters such as assumptions of future scenarios. During an interactive workshop preliminary findings were shared with the case studies and used to stimulate the discussions on the direction for developing final results in the form of new scientific knowledge and tailored climate change impact information. In addition, there were a number of email exchanges and phone meetings to follow up and fine-tune the climate information, and to exchange data between stakeholders and researchers.

In the third phase of each case study (communication of results and monitoring of the process and outcomes), the final results were shared and discussed. Participants' perceptions of the co-design process were also explored at a focus group meeting. This meeting centered around a few key questions covering the following aspects: the interaction and dialogue between case studies and researchers; challenges to the process, as well as how the results related to ongoing activities and adaptation plans. A timeline was also drawn where important milestones to the co-design process were highlighted.

The results were presented at a meeting and summarized in a written report. Stakeholders were given the opportunity to review a draft version, ask questions, and provide feedback. A few months after the final results were shared, stakeholders were interviewed to capture their perceptions of the information in relation to the adaptation knowledge quality criteria. The interviews were semi-structured and, questions included different dimensions of the criteria such as whether and how the results had been used, their relevance, presentation and communication, and perceived (scientific) quality. The co-design process ended with a final stakeholder workshop in which participants from both case studies met virtually with a wider group of stakeholders to share the case study results and reflect on key insights and lessons learned from the project. During a break-out session, stakeholder discussed across case studies, their experiences of the project and the collaboration process that contributed to or hindered the uselessness of the results.

Our analysis builds in particular on detailed notes from the meetings, interviews and final stakeholder workshop carried out in phase 3. First, we deductively applied (Shaw and Holland, 2014) the knowledge quality criteria (credibility, legitimacy, saliency, usability, and usefulness) to the material to assess the extent to which they were perceived to be met by the case study representatives. Secondly, we analyzed the entire co-design process and identified inductively factors that had contributed positively or negatively to addressing the criteria. In this part of the analysis, we also included research teams' observations and reflections. The results were synthesized and related to the evaluation framework by Wall et al. (2017) to highlight the most critical factors in this particular co-design process.

Results

Results are presented in two parts. First, we describe how each of the adaptation knowledge quality criteria were achieved in the case study of City of Stockholm (section Perceptions of adaptation knowledge quality: City of Stockholm) followed by Karlstad Municipality (section Perceptions of adaptation knowledge quality: Karlstad Municipality) and summarized in Table 2. Second, most critical factors–representing different elements of the co-design process–that contributed positively or negatively to addressing the criteria are outlined (section Elements of the Co-design process that contributed to addressing the adaptation knowledge quality criteria). See Table 3 for an overview.

TABLE 2

Table 2. Results overview: achievement of adaptation knowledge quality criteria in case studies.

TABLE 3

Table 3. Important factors in HazardSupport co-design process, based on components in the Wall et al. (2017) evaluation framework of knowledge co-production processes.

Perceptions of Adaptation Knowledge Quality: City of Stockholm

Results from the modeling of how ongoing urbanization will affect the urban climate of Stockholm were expected to feed into municipal planning processes by providing improved information about the role of climate-sensitive planning, and particularly urban green infrastructure, for heat stress mitigation. Stakeholders stressed that the results from the two scenarios would be particularly useful to enhance awareness and create discussions about the future development of the city.

Credibility

Participants expressed very high trust in both the model results and in SMHI as a state agency and research institution; therefore, the scientific accuracy was taken for granted. For example, one municipal officer noted that “in the context of climate change, there is a high level of confidence in the scientific quality of SMHI and that it is substantiated.” This also meant that, given that the municipal staff now have numbers and statistics confirming their previous assumptions, the results added value to their internal planning processes and increased the credibility of the municipal staff in their communication with other actors (see also Saliency).

Moreover, participants said the results were clearly communicated, and they appreciated that the researchers were not overly cautious in their communication of the results, but instead proposed clear recommendations. They liked that the researchers did not adhere so strictly to standards for scientific credibility during the co-design process that they could not provide information credible enough to use in a planning context.

Legitimacy

Legitimacy was not perceived as a major concern in the City of Stockholm case study. One participant noted though that the group of involved stakeholders had been limited in number, and that it would have been useful to engage stakeholders from different departments within the City, to increase the chances of wider uptake and use of the information. This relates (indirectly) to legitimacy of the process since representation of stakeholders' is one factor that is known to characterize this criterion.

Saliency

Research results were overall perceived as relevant to provide further guidance and to inform practitioners' work on mitigating heat stress through green infrastructure. So far, the results have both inspired and strengthened the content of the local Environment Programme adopted by the City in May 2020 (City of Stockholm, 2020). Participants said the results provided valuable planning support and gave them better arguments for further highlighting the importance of green infrastructure to mitigate heat stress by providing them with a better understanding of the issue. The relevance of the results was also mirrored in their demand for SMHI to disseminate and discuss the results with a wider audience, including other departments within the City administration. At the same time, the results were mainly perceived as planning support, not decision support, which would inform more concrete measures.

The saliency of the results was ensured through the co-design process, in which representatives of the City of Stockholm and SMHI worked together to both define the problem and choose the scenarios. Data were also shared to make sure that SMHI scenarios corresponded to the City of Stockholm's planning scenarios as expressed in strategic documents such as the current master plan.

Usefulness and Usability

Considering the spatial scale of the two scenarios for Stockholm's future urbanization, the results were perceived primarily to be useful for planning and communication. To develop action plans, participants thought more detailed information at the district level would be needed, for example, to be able to estimate the effects of specific measures. However, participants recognized the limitations of the current status of scientific knowledge, and that they received the best available information.

The usability of the results was perceived as positive—for example, the use of GIS maps that showed areas at risk of heating. Participants also said that the results had been communicated clearly and concisely in the report, which made it easier for lay persons to understand them and interpret their implications. In addition to the report, participants also mentioned that the ongoing oral communication via seminars and meetings throughout the project had been helpful in their understanding of the results.

Perceptions of Adaptation Knowledge Quality: Karlstad Municipality

The information developed within HazardSupport was intended to feed into the Municipality's appraisal of a potential flood defense wall in advance of budget and implementation decisions.

Credibility

A main rationale for Karlstad Municipality in participating in the project was to “have some weight going into the discussions with the County Administrative Board. Working with a Swedish administrative authority like SMHI gives that weight.” The collaboration with SMHI, which was perceived as a credible science provider, was thus expected to allow municipal officials to strengthen their arguments on flood protection for the Skåre area in relation particularly to the County Administrative Board.

The knowledge generated through the project was perceived as highly credible by all practitioners from the Municipality, and like the Stockholm case study participants, they saw SMHI research as credible: “We have full confidence in what they have delivered—it is a reliable and trustworthy Swedish administrative authority.”

For the data on multiple extremes, the results were based on a short time series, which the SMHI researcher said gave it low scientific credibility (Berg et al., 2019). While the Karlstad Municipality stakeholders had originally hoped to be able to get more solid information on multiple extremes, they appreciated the limitations in terms of what is scientifically possible, and felt they could still use the information provided, even if it was relatively uncertain. Municipal officers noted that “no one else than SMHI can provide this information [on multiple extremes],” and they appreciated getting any information that was “good enough” for their decision-making.

SMHI and Karlstad Municipality agreed at the handover meeting that if the Municipality wanted access to the project data, for instance, to hand it over to consultants for additional analyses, SMHI would provide a contract for the data specifying a best-before-date for its validity. This is because the output data are based on modeling and may lose credibility as modeling techniques improve over time. This solution was suggested by SMHI researchers to maintain their credibility and ensure that no outdated information would be used and further circulated.

Legitimacy

There are no indications that legitimacy has been a concern in the Karlstad Municipality case study.

Saliency

The knowledge provided by SMHI relates directly to an ongoing planning process in Skåre, where the Municipality needs to demonstrate that building the flood barrier would not increase the flood risk within the area it is supposed to protect. SMHI provided Karlstad Municipality with information on return times and probabilities of cloudbursts, and municipal officials said they now have more certainty about the risks associated with cloudbursts. However, the information was not new per se, but rather confirmed previous assumptions. The knowledge on multiple extremes was new to Karlstad Municipality, and can be interpreted as a factual (vs. conformational) use of information.

Usefulness and Usability

From a usefulness perspective, the timing of the delivery of results was important for Karlstad Municipality, as the information feeds into an ongoing planning process. There were continuous discussions between SMHI and municipal officers about the time plan for delivering the final results, and the timing of the delivery corresponded to the needs expressed by the Municipality.

With regard to usability, the SMHI researcher gave a presentation and was available for questions in conjunction with the delivery of the final report, at the request of the municipal officers. As the information concerned a relatively narrow and technical topic, some of the stakeholders afterwards said they found it challenging to interpret the information. Results presented in the form of maps of flooding consequences of cloudbursts were perceived as easier to interpret, as municipal officers are used to working with this format. The maps were also useful to address uncertainty, as the Municipality could easily observe that the water levels were far from reaching critical infrastructure. Stakeholders also noted that the conclusions were summarized in bullet points, which was perceived as making the results more accessible.

A Hindsight Perspective on Knowledge Quality Criteria in Karlstad Municipality

At the conclusion of the project, and after the final information had been delivered and followed up, some potential challenges surfaced with regard to the knowledge quality criteria. While it remains uncertain, at the time of writing this article, exactly how the process unfolded, we address some potential, though speculative, explanations below.

After receiving the final results [see Method (Co-design Process)], Karlstad Municipality handed over the data to a consultant for further computations to define the need for installed pump capacity to evacuate excessive water across the flood barrier. At this stage, questions arose relating to the appropriateness of the delimitations in the definition of the catchment area (a usefulness concern). In hindsight, there also appear to have been at least partially divergent understandings of what would be delivered within the project, in particular with regard to how detailed the data related to the needed pump capacity would be (a saliency concern). This divergence may be related to insufficient articulation of needs at an early stage in the process. High staff turnover may also have been a factor, as new staff members may have entered the project with different interpretations of what had been previously decided, or started at a point in a municipal planning process when new, more specific information needs had emerged. It also appears that the technical nature of the final results, and the fact that not all relevant stakeholders were able to interpret the information at the time of the handover (a usability concern), may have contributed to the situation, as some of the issues that later arose might have been possible to solve or mitigate.

Elements of the Co-design Process That Contributed to Addressing the Adaptation Knowledge Quality Criteria

Case Study Information Needs and Path to Use

An important prerequisite in any climate service co-design process is a jointly defined question that is interesting both from a scientific and practical perspective. In HazardSupport, one critical factor as regards the perceived saliency of the results is that both municipalities had an interest and an evident need for improved planning and decision support to advance their adaptation work (IN2).

During the first phase of the project, several meetings were facilitated to identify and define case study needs (including decision contexts and current uses of climate information), and how climate researchers could meet those needs given the current status of scientific knowledge and expertise. However, despite stakeholders' good level of understanding of flood-related risks in Karlstad Municipality, it took several iterations and multiple meetings before a decision could be made on which parameters and climatic factors to analyze, which were then refined throughout the entire process.

The presentation of emerging results at a stakeholder workshop in phase two (Figure 1) seemed to be instrumental to further define and jointly agree on the next steps. However, as concluded in section A hindsight perspective on knowledge quality criteria in Karlstad Municipality, some questions regarding the scope of the study surfaced after the final results had been shared with the stakeholders. These experiences show that it may be difficult for users to develop an understanding of their needs and priorities as regards climate information. This is a particular concern if users have limited experience with the issue, but can also happen with more experienced users. The co-design process therefore plays an important role to support stakeholders to articulate their needs and to ensure that there is mutual understanding between stakeholders and researchers. Also, circumstances outside the scope of the project (e.g., as internal planning processes progress and needs change, or when the information is used by external actors, in this case a consultant) may change stakeholder perceptions of the saliency of the results.

The path to using the information has been different in the two case studies (IN4). Both internal processes and external events contributed to an increased sense of urgency and relevance of the adaptation challenges (EF2). In Karlstad Municipality, the issue of building a flood defense wall has become more and more concrete as the internal planning process evolved, with increasingly specific information needs as a result. In Stockholm, a severe heat wave that hit Sweden in 2018 functioned as a wake-up call and facilitated internal discussions of the need to consider heat stress in municipal planning. While it is yet an emerging topic in the City's adaptation planning, in comparison with water-related hazards such as heavy rainfalls or flooding, heat stress has gained traction over the course of the HazardSupport project. It was addressed in the new local Environmental Programme, for instance, which has increased the salience of knowledge on heat stress.

In both case studies it was clear that the information developed within the project constituted one piece of a much larger information puzzle for the municipalities. The results gained from the project were combined with other types of information, and connected to a much wider decision-making context in which the climate adaptation aspect was weighted against other concerns, such as budgetary consequences related to the flood defense wall in Karlstad, and the need for housing in Stockholm.

Path to Use Affects Knowledge Quality Requirements

The two case studies represent different uses of climate information (IM1), with implications for how the knowledge quality criteria were perceived. In the Stockholm case study, the results were mainly used to get a better understanding of the adaptation challenge [i.e., “problem understanding use” (sensu Wall et al., 2017)] as well as to motivate the search for more information (i.e., “motivational use”). The new, quantified results from SMHI largely confirmed the users' expectations about the correlation between green infrastructure and heat stress in the city, which reinforced their argument in relation to other parts of the Municipality (i.e., “confirmational use”). The information can be used in a variety of contexts to make the case for addressing heat stress in planning. In Karlstad, on the other hand, the climate information concerned a much more specific, technical question related to the localized flooding implications of building a flood defense wall. This piece of information is needed in a specific planning process and is of limited general interest beyond the specific location. Instead, it is used in an “instrumental” and “factual” way in the sense that stakeholders were provided with more precise data and numbers. Further, though to a minor extent, knowledge about multiple extremes was used to improve their understanding of the problem.

While the direct connection to a specific planning process arguably creates good conditions for generating tailor-made information, it also makes the co-production process and achievement of the criteria more sensitive. For example, in relation to saliency, there is on the one hand a clearly defined need for climate information. On the other hand, this puts higher requirements on the user to clearly articulate the need and for the provider to understand the need and the decision-making context. It is also sensitive to changes in need over time, for instance, as a planning process progresses and the need becomes clearer, or as new colleagues get involved who may perceive the need differently. This puts high requirements on communication over time, to ensure that saliency is maintained.

Similarly, from a usefulness point of view, the timing of the delivery of results (O1) is much more critical in a case such as Karlstad Municipality, where results need to be delivered in the right phase of the planning process, as opposed to the City of Stockholm case, where results can likely be used over years to come in a range of contexts, such as seminars, presentations, and dialogues. From a usability point of view, the Karlstad Municipality case required quite detailed and specific technical competence (EF4) from the user to be able to interpret and use the information in the planning process, ask for clarifications and adjustments in the data, etc. This is a capacity that only a small number of staff in the Karlstad Municipality had. In the City of Stockholm case, on the other hand, as the results are more generic, they are accessible to a wider group of civil servants or even the wider public.

Continuity, Documentation, and Communication Key in Long-Term Projects With High Staff Turnover

The HazardSupport project spanned a relatively long time period (5 years), which was a prerequisite for doing both research and developing the specific climate information. It also allowed substantial time for running an iterative co-production process. From a scientific point of view, it also allowed for the exploration of new methods and synchronization with other activities that provided necessary information. However, as mentioned in section A hindsight perspective on knowledge quality criteria in Karlstad Municipality the long time span also meant a high staff turnover in the project (EF1), in both case studies and on both the researcher and practitioner side which proved to be a challenge. Especially in Karlstad case study where this seems to have affected the perceived saliency of the information, as well as the usability, as new project members were not aware of how they could access additional information from SMHI. The associated risks related to knowledge quality need to be mitigated with consistent documentation and communication throughout the project (P1).

At the same time, building strong relationships between users and providers of climate information takes time, and from this perspective, 5 years can be seen as solid foundation to further build on. This was for example brought up by stakeholders in the City of Stockholm, who referred to an ongoing exchange and dialogue between officers within the City and SMHI over many years. This meant they had both a preexisting relationship (IN3) and a good level of mutual understanding about needs and capacities, which facilitated phase one of the project and possibly contributed to the high level of credibility and perceived saliency of the results.

Cutting-Edge Research vs. Repackaging Existing Knowledge

An important factor for the co-design process and usability of results, and to ensure buy-in from both researchers and users, was the focus on identifying issues that were interesting and relevant from both a scientific and practical point of view (IN1). This meant that developing the knowledge base to inform adaptation planning and decision-making was the purpose, rather repackaging existing scientific knowledge into a practical tool or service. However, as the co-design process evolved, stakeholders expressed a need for specific and tailored data, which were difficult to deliver as the issues in focus were at the cutting-edge of science (EF3) e.g., as regards multiple extremes in Karlstad Municipality. Balancing the demands for achieving scientific and practical outcomes was challenging from the researchers' point of view as it was difficult to predict the exact outcomes of the research process. The capacity to develop information at relevant scales and with very high resolution—required by stakeholders for decision-making—was therefore limited.

Discussion

In this section, we discuss the results with a focus on the relevance and interrelatedness of the criteria, as well as the need to consider context-dependent factors in the assessment. Then we discuss the need to capture additional dimensions of the co-design process that could further our understanding of how to achieve actionable adaptation knowledge as well as associated non-tangible outcomes.

Knowledge Quality Criteria Are Interrelated and Context-Dependent

First, the analysis showed that all criteria were relevant, and were all, except legitimacy, actively addressed in interviews with stakeholders. Saliency seemed to be the most critical criterion to the co-design process and to how stakeholders perceived the quality of the adaptation knowledge. This is likely explained by the characteristics of the project, with its emphasis on developing new knowledge to inform adaptation planning and decision-making, as opposed to new tools or repackaging of existing information. As discussed under section Cutting-Edge Research vs. Repackaging Existing Knowledge, this focus was driven by stakeholder needs, which required new scientific knowledge.

It is also worth noting that the two case studies differed in terms of the specific adaptation challenges and natural hazards at hand, which meant that they needed information at different levels of detail. For the City of Stockholm, the primary use was for awareness-raising, whereas Karlstad Municipality needed specific information to feed into a planning process. This difference related to saliency spills over into the other knowledge criteria, which become more critical as saliency requirements become more specific. Going forward (especially in the City of Stockholm), tools to assess and implement measures at the district level will be needed, and the usability and usefulness of the adaptation knowledge will likely be more critical. However, if the saliency criterion is not fulfilled, it is unlikely that the other criteria will be met.

We also note that the saliency criterion benefited from the structure of the co-design process (c.f. de Vente et al., 2016) particularly the first phase, which helped to ensure a mutual understanding of needs, capacities, and limitations. The literature also highlights the importance of jointly defining the problem, which is a critical part of setting the right conditions for a successful co-design process (c.f. Hegger et al., 2012; Jagannathan et al., 2020; Norström et al., 2020).

It is also clear from our results that the criteria may be interpreted differently among researchers than among practitioners (c.f. Hegger et al., 2012). Specifically, stakeholders in the City of Stockholm said the results were credible enough for use in planning, even if they did not, at the time, meet standards for scientific credibility. They appreciated that researchers shared results during the co-design process that were salient to the Municipality—a sign that a good balance was in this case struck between credibility and other criteria.

Moreover, the results in this study indicated that context-specific factors can affect the relevance of the quality criteria. As noted in the cases studies, the trust in both the knowledge provided and SMHI as knowledge provider was very high, and in 2020, SMHI was ranked as the national agency in Sweden with the highest reputation (Orbe and Sjörén, 2020). This does not mean that the credibility criterion was irrelevant, but its greatest importance was at the outset, laying a foundation for the collaborative process and making it more attractive for stakeholders to join.

The legitimacy criterion does not appear to have been an issue in either case. It is worth noting that both case studies took place within local government administrations under democratic control, which may explain why legitimacy did not surface as a concern. The subjects of the studies were also relatively uncontroversial, and climate change concerns are now commonly addressed in Swedish municipal planning (Matschke Ekholm and Nilsson, 2019). This may also reduce the importance of both credibility and legitimacy concerns in this context. If the legitimacy and credibility criteria were not met, it would probably negatively affect the perceived saliency. For example, participants clearly saw the high credibility of the adaptation knowledge as particularly useful and something that added value to their own work to further motivate and communicate the need for adaptation measures, especially in dialogue with other actors, such as decision-makers.

Decision-Making Contexts and Non-tangible Outcomes Matter

Our findings showed that all five criteria—credibility, legitimacy, saliency, usefulness, and usability—are relevant to assessing the quality of knowledge for adaptation. Together, they capture a range of interconnected features that are all necessary to the development of scientific knowledge that can support adaptation planning and decision-making. However, the criteria are not sufficient to understand whether the knowledge is indeed perceived as actionable by actors responsible for its implementation. The step of actually putting information into use hinges on other factors beyond the scope of the co-design process. Our results illustrate that the adaptation knowledge needs to be aligned with existing planning tools and processes and combined with other types of non-climate-related information.

The climate information itself is only one of many considerations for practitioners (as also noted by Klein and Juhola, 2014), and understanding whether knowledge is actionable requires an assessment of the broader planning and decision-making contexts, typically looking beyond climate change adaptation issues per se. The fact that the two municipalities' evolving internal processes and external events affected the perceived saliency of the knowledge over time suggests a need to iteratively assess the planning and decision contexts as the co-design process evolves. Furthermore, as suggested by Hansson and Polk (2018) in the context of transdisciplinary research, additional stakeholders may be involved in the actual use of adaptation knowledge, and they may have different perceptions of its quality than those directly involved in the co-design process. Thinking upfront about a potentially broader user base could help project leaders to identify additional participants whom they might want to include in the co-design process. To consider context-based factors in the assessment of knowledge co-production processes is also necessary, as proposed by Norström et al. (2020).

Though it is too early to assess the long-term impacts of the co-design processes in these two case studies, we note the potential for important intangible outcomes as well, such as shared learning, capacity-building, and long-term relationships that then help increase stakeholders' adaptive capacity. Our analysis showed that the knowledge quality criteria did not fully capture these long-term and more intangible effects [what Jagannathan et al. (2020) refer to as community outcomes]. As crucial as it is to meet the knowledge quality criteria in order to bridge the gap between science and practice, other process-oriented values, such as “mutual interest in longer-term collaboration” (Wall et al., 2017), can contribute to that objective in the longer term, and can play a key role in supporting adaptation planning and action beyond an individual project. Further, Reed et al. (2021) note the complexity of measuring and attributing policy impacts of research and point to the need to consider both positive and negative effects as well as tailoring the impact evaluating design with “aims and context of the evaluation.”

Along with meeting the quality criteria, we therefore suggest, similar to Daniels et al. (2020) and Beier et al. (2017), that, to be most effective the co-production of knowledge for adaptation should focus not primarily on standalone projects and knowledge products, but rather on processes in which decision contexts, needs, goals, and capacities are fully appreciated. To consider context-based factors in the design of participatory processes is further acknowledged by de Vente et al. (2016). Such a process-centric, transdisciplinary knowledge integration approach to climate services has shown to support a shared understanding of a problem, build trust, capacity, and confidence to engage in unfamiliar knowledge spaces, and, in turn, strengthened relationships and networks over a longer timeframe (Daniels et al., 2020). The co-design process in this study shows great potential for that approach. The project benefited from building on long-term relationships that underpinned the development of the knowledge base. Going forward, those relationships can provide a strong foundation on which to build strategies and policies and co-produce additional knowledge as needed.

Conclusions

This study shows that the proposed knowledge quality criteria, and in particular the saliency criterion, are relevant yet insufficient to fully capture whether and how adaptation knowledge is perceived as actionable. Our findings suggest that the criteria do not capture the wider decision-making context that in turn affects stakeholders' perceptions of the quality of the knowledge and their ability to apply it to adaptation planning and decision-making. Further, the criteria overlook important long-term and intangible effects of the co-design process, such as strengthened relationships, networks learning and capacity.

These conclusions point to two key ways to improve both the knowledge co-production process, and the criteria used to assess the quality of its outcomes. First, we observe a need to design knowledge co-production processes that consider the wider decision-making context to the greatest extent possible, and this ought to be captured in the adaptation knowledge quality assessment. Second, in the design and consequently the quality assessment, non-tangible outcomes should also be considered and acknowledged. For example, it matters whether the work created conditions for long-term engagement and building relationships, trust, and mutual learning. Such outcomes may be critical to the success of the adaptation process—and in the process, they may improve the quality of the adaptation knowledge itself.

Data Availability Statement

The datasets presented in this article are not readily available because it contains confidential data. Requests to access the datasets should be directed to corresponding author.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author Contributions

KA: conceptualization, methodology, validation, formal analysis, investigation, writing—original draft, writing—review & editing, and funding acquisition. LJ: conceptualization, methodology, validation, formal analysis, investigation, writing—original draft, and writing—review & editing. ÅGS: conceptualization, methodology, validation, investigation, writing—original draft, writing—review & editing, and funding acquisition. PB: validation, writing—original draft, and writing—review & editing. DS and JHA: validation, writing—original draft, writing—review & editing, and funding acquisition. LS: validation, writing—review & editing, and project administration. All authors contributed to the article and approved the submitted version.

Funding

HazardSupport: Risk-based decision support for adaptation to future natural hazards was funded by the Swedish Civil Contingencies Agency (grant number 2015-3631) and has benefited from collaborations with the City of Stockholm, and Karlstad Municipality.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors wish to thank all stakeholders for their participation in the project. We are also thankful to Chantal Donelly (Bureau of Meteorology, previous SMHI) project coordinator during the first phase of the project and Sandra Tenggren (Swedish Transport Workers' Union, previous SEI) for contributions to the co-design process, also during the first phase. Our thanks also to Marion Davis for editing.

References

Amorim, J. H., Segersson, D., Körnich, H., Asker, C., Olsson, E., and Gidhagen, L. (2020). High resolution simulation of Stockholm's air temperature and its interactions with urban development. Urban Climate 32:100632. doi: 10.1016/j.uclim.2020.100632

CrossRef Full Text | Google Scholar

Beier, P., Hansen, L. J., Helbrecht, L., and Behar, D. (2017). A how-to guide for coproduction of actionable science. Conserv. Lett. 10, 288–296. doi: 10.1111/conl.12300

CrossRef Full Text | Google Scholar

Berg, P., Hundecha, Y., Olsson, A., Tofeldt, L., and Yang, W. (2019). Översvämningsrisk i Skåre. HazardSupport deliverable 2.2. Norrköping: SMHI.

Google Scholar

Brasseur, G. P., and Gallardo, L. (2016). Climate services: lessons learned and future prospects. Earth's Future 4, 79–89. doi: 10.1002/2015EF000338

CrossRef Full Text | Google Scholar

Bremer, S., and Meisch, S. (2017). Co-production in climate change research: reviewing different perspectives. WIREs Clim. Change 8:e482. doi: 10.1002/wcc.482

CrossRef Full Text | Google Scholar

Bremer, S., Wardekker, A., Dessai, S., Sobolowski, S., Slaattelid, R., and van der Sluijs, J. (2019). Toward a multi-faceted conception of co-production of climate services. Clim. Serv. 13, 42–50. doi: 10.1016/j.cliser.2019.01.003

CrossRef Full Text | Google Scholar

Bruno Soares, M., and Dessai, S. (2016). Barriers and enablers to the use of seasonal climate forecasts amongst organisations in Europe. Clim. Change 137, 89–103. doi: 10.1007/s10584-016-1671-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Cash, D., Clark, C. W., Alcock, F., Dickson, M. N., Eckley, N., and Jäger, J. (2003). Salience, Credibility, Legitimacy and Boundaries: Linking Research, Assessment and Decision Making. KSG Working Papers Series, 2003. Available online at: http://nrs.harvard.edu/urn-3:HUL.InstRepos:32067415 (accessed February 17, 2021).

Google Scholar

Cash, D. W., Clark, W. C., Alcock, F., Dickson, N. M., Eckley, N., Guston, D. H., et al. (2003). Knowledge systems for sustainable development. Proc. Natl. Acad. Sci. U. S. A. 100, 8086–8091. doi: 10.1073/pnas.1231332100

CrossRef Full Text | Google Scholar

Christel, I., Hemment, D., Bojovic, D., Cucchietti, F., Calvo, L., Stefaner, M., et al. (2018). Introducing design in the development of effective climate services. Clim. Serv. 9, 111–121. doi: 10.1016/j.cliser.2017.06.002

CrossRef Full Text | Google Scholar

City of Stockholm (2020). Environment Programme 2020–2023. Stockholm: City Executive Offce.

Google Scholar

Clifford, K. R., Travis, W. R., and Nordgren, L. T. (2020). A climate knowledges approach to climate services. Clim. Serv. 18:100155. doi: 10.1016/j.cliser.2020.100155

CrossRef Full Text | Google Scholar

Cortekar, J., Lamich, K., Otto, J., and Pawelek, P. (2017). Review and Analysis of CS Market Conditions. EU-MACS European Market for Climate Services. Available online at: http://eu-macs.eu/wp-content/uploads/2017/07/EU-MACS-D11_CLIMATE-SERVICE-MARKET-CONDITIONS.pdf (accessed February 17, 2021).

Google Scholar

Cvitanovic, C., Howden, M., Colvin, R. M., Norström, A., Meadow, A. M., and Addison, P. F. E. (2019). Maximising the benefits of participatory climate adaptation research by understanding and managing the associated challenges and risks. Environ. Sci. Policy 94, 20–31. doi: 10.1016/j.envsci.2018.12.028

CrossRef Full Text | Google Scholar

Daniels, E., Bharwani, S., Gerger Swartling, Å., Vulturius, G., and Brandon, K. (2020). Refocusing the climate services lens: introducing a framework for co-designing “transdisciplinary knowledge integration processes” to build climate resilience. Clim. Serv. 19:100181. doi: 10.1016/j.cliser.2020.100181

CrossRef Full Text | Google Scholar

de Vente, J., Reed, M. S., Stringer, L. C., Valente, S., and Newig, J. (2016). How does the context and design of participatory decision making processes affect their outcomes? Evidence from sustainable land management in global drylands. Ecol. Soc. 21:224. doi: 10.5751/ES-08053-210224

CrossRef Full Text | Google Scholar

Dessai, S., and Hulme, M. (2004). Does climate adaptation policy need probabilities? Climate Pol. 4, 107–128. doi: 10.1080/14693062.2004.9685515

CrossRef Full Text | Google Scholar

Dilling, L., and Lemos, M. C. (2011). Creating usable science: opportunities and constraints for climate knowledge use and their implications for science policy. Glob. Environ. Change A 21, 680–689. doi: 10.1016/j.gloenvcha.2010.11.006

CrossRef Full Text | Google Scholar

Ernst, K. M., Gerger Swartling, Å. G., André, K., Preston, B. L., and Klein, R. J. T. (2019). Identifying climate service production constraints to adaptation decision-making in Sweden. Environ. Sci. Policy 93, 83–91. doi: 10.1016/j.envsci.2018.11.023

CrossRef Full Text | Google Scholar

Gerger Swartling, Å., Tenggren, S., Andre, K., and Olsson, O. (2019). Joint knowledge production for improved climate services: insights from the Swedish forestry sector. Environ. Policy Governance 29, 97–106. doi: 10.1002/eet.1833

CrossRef Full Text | Google Scholar

Gidhagen, L., Olsson, J., Amorim, J. H., Asker, C., Belusic, D., Carvalho, A. C., et al. (2020). Towards climate services for European cities: lessons learnt from the Copernicus project Urban SIS. Urban Climate 31:100549. doi: 10.1016/j.uclim.2019.100549

CrossRef Full Text | Google Scholar

Government bill (2017). Nationell strategi för klimatanpassning [National strategy for climate adaptation]. Stockholm: Ministry for the Environment and Energy.

Google Scholar

Hansson, S., and Polk, M. (2018). Assessing the impact of transdisciplinary research: the usefulness of relevance, credibility, and legitimacy for understanding the link between process and impact. Res. Eval. 27, 132–144. doi: 10.1093/reseval/rvy004

CrossRef Full Text | Google Scholar

Hegger, D., Lamers, M., Zeijl-Rozema, A. V., and Dieperink, C. (2012). Conceptualising joint knowledge production in regional climate change adaptation projects: success conditions and levers for action. Environ. Sci. Policy 18, 52–65. doi: 10.1016/j.envsci.2012.01.002

CrossRef Full Text | Google Scholar

Jagannathan, K., Arnott, J. C., Wyborn, C., Klenk, N., Mach, K. J., Moss, R. H., et al. (2020). Great expectations? Reconciling the aspiration, outcome, and possibility of co-production. Curr. Opin. Environ. Sustain. 42, 22–29. doi: 10.1016/j.cosust.2019.11.010

CrossRef Full Text | Google Scholar

Klein, R. J. T., and Juhola, S. (2014). A framework for Nordic actor-oriented climate adaptation research. Environ. Sci. Policy 40, 101–115. doi: 10.1016/j.envsci.2014.01.011

CrossRef Full Text | Google Scholar

Lang, D. J., Wiek, A., Bergmann, M., Stauffacher, M., Martens, P., Moll, P., et al. (2012). Transdisciplinary research in sustainability science: practice, principles, and challenges. Sustain. Sci. 7:25. doi: 10.1007/s11625-011-0149-x

CrossRef Full Text | Google Scholar

Lemos, M. C., Kirchhoff, C. J., and Ramprasad, V. (2012). Narrowing the climate information usability gap. Nat. Clim. Chang 2, 789–794. doi: 10.1038/nclimate1614

CrossRef Full Text | Google Scholar

Lemos, M. C., and Morehouse, B. J. (2005). The co-production of science and policy in integrated climate assessments. Glob. Environ. Change A 15, 57–68. doi: 10.1016/j.gloenvcha.2004.09.004

CrossRef Full Text | Google Scholar

Lourenço, T. C., Swart, R., Goosen, H., and Street, R. (2016). The rise of demand-driven climate services. Nat. Clim. Chang 6, 13–14. doi: 10.1038/nclimate2836

CrossRef Full Text | Google Scholar

Lowrey, J. L., Ray, A. J., and Webb, R. S. (2009). Factors influencing the use of climate information by Colorado municipal water managers. Clim. Res. 40, 103–119. doi: 10.3354/cr00827

CrossRef Full Text | Google Scholar

Mach, K. J., Lemos, M. C., Meadow, A. M., Wyborn, C., Klenk, N., Arnott, J. C., et al. (2020). Actionable knowledge and the art of engagement. Curr. Opin. Environ. Sustain. 42, 30–37. doi: 10.1016/j.cosust.2020.01.002

CrossRef Full Text | Google Scholar

Matschke Ekholm, H., and Nilsson, Å. (2019). Klimatanpassning 2019 - Så långt har Sveriges kommuner kommit [Climate adaptation 2019 - so far has Swedish municipalities come]. Stockholm: IVL, Swedish Environmental Research Institute.

Google Scholar

McNie, E. C. (2007). Reconciling the supply of scientific information with user demands: an analysis of the problem and review of the literature. Environ. Sci. Policy 10, 17–38. doi: 10.1016/j.envsci.2006.10.004

CrossRef Full Text | Google Scholar

Moser, S. C., and Ekstrom, J. A. (2010). A framework to diagnose barriers to climate change adaptation. Proc. Natl. Acad. Sci. U. S. A. 107, 22026–22031. doi: 10.1073/pnas.1007887107

PubMed Abstract | CrossRef Full Text | Google Scholar

Norström, A. V., Cvitanovic, C., Löf, M. F., West, S., Wyborn, C., Balvanera, P., et al. (2020). Principles for knowledge co-production in sustainability research. Nat. Sustain. 3, 182–190. doi: 10.1038/s41893-019-0448-2

CrossRef Full Text | Google Scholar

Olsson, J., Berg, P., Eronn, A., Simonsson, L., Södling, J., Wern, L., et al (2017). Extremregn i nuvarande och framtida klimat: Analyser av observationer och framtidsscenarier. Norrköping: Swedish Meteorological and Hydrological Institute, SMHI Available online at: https://www.smhi.se/publikationer/publikationer/extremregn-i-nuvarande-och-framtida-klimat-analyser-av-observationer-och-framtidsscenarier-1.129407 (accessed December 4, 2019).

Google Scholar

Olsson, J., Södling, J., Berg, P., Wern, L., and Eronn, A. (2019). Short-duration rainfall extremes in Sweden: a regional analysis. Hydrol. Res. 50, 945–960. doi: 10.2166/nh.2019.073

CrossRef Full Text | Google Scholar

Orbe, J., and Sjörén, T. (2020). Anseendet för Svenska Myndigheter 2020. Kantar Sifo. Available online at: https://www.kantarsifo.se/sites/default/files/reports/documents/kantar_sifos_anseendeindex_myndigheter_2020.pdf (accessed February 17, 2021).

Google Scholar

Palutikof, J. P., Leitch, A. M., Rissik, D., Boulter, S. L., Campbell, M. J., Perez Vidaurre, A. C., et al. (2019). Overcoming knowledge barriers to adaptation using a decision support framework. Clim. Change 153:607. doi: 10.1007/s10584-018-2177-3

CrossRef Full Text | Google Scholar

Patt, A., and Dessai, S. (2005). Communicating uncertainty: lessons learned and suggestions for climate change assessment. Comptes Rendus Geoscience 337, 425–441. doi: 10.1016/j.crte.2004.10.004

CrossRef Full Text | Google Scholar

Porter, J. J., and Dessai, S. (2017). Mini-me: why do climate scientists' misunderstand users and their needs? Environ. Sci. Policy 77, 9–14. doi: 10.1016/j.envsci.2017.07.004

CrossRef Full Text | Google Scholar

Reed, M. S., Ferré, M., Martin-Ortega, J., Blanche, R., Lawford-Rolfe, R., Dallimer, M., et al. (2021). Evaluating impact from research: a methodological framework. Res. Policy 50:104147. doi: 10.1016/j.respol.2020.104147

CrossRef Full Text | Google Scholar

Schneider, F., Giger, M., Harari, N., Moser, S., Oberlack, C., Providoli, I., et al. (2019). Transdisciplinary co-production of knowledge and sustainability transformations: three generic mechanisms of impact generation. Environ. Sci. Policy 102, 26–35. doi: 10.1016/j.envsci.2019.08.017

CrossRef Full Text | Google Scholar

Shaw, I., and Holland, S. (2014). Doing Qualitative Research in Social Work. London: SAGE Publications. doi: 10.4135/9781473906006

CrossRef Full Text | Google Scholar

Sjöberg, T., Hjerpe, K., Lundgren Kownacki, K. L., and Andersson, L. (2019). Kommunernas arbete med klimatanpassning 2019–Analys av statusrapportering till SMHI. Norrköping: SMHI.

Google Scholar

Vaughan, C., and Dessai, S. (2014). Climate services for society: origins, institutional arrangements, and design elements for an evaluation framework. WIREs Clim. Change 5, 587–603. doi: 10.1002/wcc.290

PubMed Abstract | CrossRef Full Text | Google Scholar

Vincent, K., Daly, M., Scannell, C., and Leathes, B. (2018). What can climate services learn from theory and practice of co-production? Clim. Serv. 12, 48–58. doi: 10.1016/j.cliser.2018.11.001

CrossRef Full Text | Google Scholar

Vulturius, G., André, K., Gerger Swartling, Å. G., Brown, C., and Rounsevell, M. (2020a). Does climate change communication matter for individual engagement with adaptation? Insights from Forest Owners in Sweden. Environ. Manage. 65, 190–202. doi: 10.1007/s00267-019-01247-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Vulturius, G., André, K., Gerger Swartling, Å. G., Brown, C., and Rounsevell, M. (2020b). Successes and shortcomings of climate change communication: insights from a longitudinal analysis of Swedish Forest owners. J. Environ. Plann. Manage. 63, 1177–1195. doi: 10.1080/09640568.2019.1646228

CrossRef Full Text | Google Scholar

Wall, T. U., Meadow, A. M., and Horganic, A. (2017). Developing evaluation indicators to improve the process of coproducing usable climate science. Wea. Climate Soc. 9, 95–107. doi: 10.1175/WCAS-D-16-0008.1

CrossRef Full Text | Google Scholar