Enhancing climate services design and implementation through gender-responsive evaluation

Gumucio, Tatiana; Hansen, James; Carr, Edward R.; Huyer, Sophia; Chiputwa, Brian; Simelton, Elisabeth; Partey, Samuel; Schwager, Saroja

doi:10.3389/fclim.2022.908602

PERSPECTIVE article

Front. Clim., 16 September 2022

Sec. Climate Services

Volume 4 - 2022 | https://doi.org/10.3389/fclim.2022.908602

This article is part of the Research Topic Insights in Climate Services: 2021 View all 8 articles

Enhancing climate services design and implementation through gender-responsive evaluation

$\nTatiana Gumucio$ Tatiana Gumucio¹^*

James Hansen²

Edward R. Carr^1,3

Sophia Huyer^4,5,6

Brian Chiputwa⁷

Elisabeth Simelton⁸

Samuel Partey⁹

Saroja Schwager²

¹Humanitarian Response and Development Lab (HURDL), George Perkins Marsh Institute, Clark University, Worcester, MA, United States
²International Research Institute for Climate and Society (IRI), Columbia University, New York, NY, United States
³International Development, Community, and Environment Department, Clark University, Worcester, MA, United States
⁴CGIAR Accelerating the Impact of CGIAR Climate Research (AICCRA) Program, Brighton, ON, Canada
⁵Women in Global Science and Technology (WISAT), Brighton, ON, Canada
⁶International Livestock Research Institute (ILRI), Nairobi, Kenya
⁷World Agroforestry (ICRAF), Nairobi, Kenya
⁸World Agroforestry (ICRAF), Stockholm, Sweden
⁹International Crops Research Institute for the Semi-Arid Tropics (ICRISAT), Bamako, Mali

Assessing and responding to gender inequalities, and promoting women's empowerment, can be critical to achieving the goals of climate services, such as improved climate resilience, productivity, food security and livelihoods. To this end, our paper seeks to provide guidance to rural climate service researchers, implementing organizations, and funders on gender-responsive evaluation of climate services, including key questions to be asked and appropriate methodology. We draw on case studies of rural climate services in Mali, Rwanda and Southeast Asia to illustrate how gender-responsive evaluations have framed and attempted to answer questions about climate information needs, access to information and support through group processes, and contribution of climate services to empowerment. Evaluation of how group participatory processes can enable women's and men's demand for weather and climate information can help close knowledge gaps on gender equity in access to climate services. Quantitative methods can rigorously identify changes in demand associated with varying interventions, but qualitative approaches may be necessary to help assess the nuances of participatory communication processes. Furthermore, evaluation of how women's and men's information needs differ according to their roles and responsibilities in distinct climate-sensitive decisions can help assess gender inequities in climate services use. Evaluation that critically considers the local normative and institutional environment influencing empowerment can help identify pathways for climate services to contribute to women's empowerment. Qualitative and mixed method methodologies can be helpful for assessing the normative and institutional changes upon which empowerment depends. Although evaluations are often conducted too late to inform the design of time-bound projects, they can contribute to improvements to climate services if results are shared widely, if implementers and funders consistently factor evidence and insights from prior evaluations into the design of new initiatives, and if ongoing climate service initiatives conduct preliminary evaluations regularly to support mid-course adjustments.

Introduction

Climate services involve the production, translation, transfer, and use of climate knowledge and information in relevant decision-making, policy and planning¹. In the context of agriculture, climate services encompass a range of products and services such as seasonal climate outlooks, historical climate analyses, routine weather forecasts, agrometeorological advisories, agricultural production forecasts, early warning systems and climate-informed agricultural decision support. As with other development interventions, robust evaluation is crucial for improving the design, delivery and impact of rural climate services (Carr et al., 2020), and for supporting appropriate investment by generating evidence of the value of climate services to particular types of users (Tall et al., 2018; Vaughan et al., 2019).

The field of climate services is coming into alignment with long-standing concerns in development and climate change adaptation (e.g., World Bank, 2009; FAO et al., 2019), by recognizing that achieving development goals requires addressing gender inequalities and promoting women's empowerment (Chi et al., 2015; Kantor et al., 2015; Farnworth et al., 2018; Gumucio et al., 2019). The socio-culturally defined roles, responsibilities, and daily activities that women and men carry out shape how they perceive, respond to, and manage the various shocks and stressors in their lives (Carr and Thompson, 2014; Bee, 2016; Rengalakshmi et al., 2018). Consequently, gender and other facets of social identity (e.g., ethnicity, race, caste, age) influence the interests, needs, and opportunities that individuals face in accessing, using, and benefiting from climate services (Carr and Owusu-Daaku, 2016; Jost et al., 2016; Carr and Onzere, 2017; Kristjanson et al., 2017; Gumucio et al., 2020).

Although there is a growing knowledge base on gender-related challenges to access, use and benefit from climate services, there is a need for more gender-responsive evaluation to better understand the nuances of how gender conditions challenges in particular contexts, and to guide efforts to develop or improve services to address these challenges (Bamberger, 2013; Tall et al., 2018; Gumucio et al., 2020). Drawing on three case studies, our paper provides guidance for rural climate service researchers, implementing organizations, and funders on key questions to consider and methods to employ in gender-responsive evaluation of rural climate services, and insights into using evaluation results to improve the design and implementation of those services.

Approach

Case studies of three contrasting rural climate service initiatives (Table 1) provide context for exploring different facets of gender-responsive climate service evaluation, and for proposing good practice guidelines. We selected these because they have been the subjects of gendered evaluations, and because co-authors' involvement in each enabled us to draw on first-hand knowledge of project interventions and evaluation methods. These initiatives varied in their gender-related ambitions, the interventions that they employed and their evaluation approach.

TABLE 1

Table 1. Case studies.

In their review of the gender-related rural climate service literature, Gumucio et al. (2020: 241) noted that, “Climate services can be a promising means of empowerment and resilience-building for rural women … however, they risk reinforcing the gender-based inequalities that are prevalent in other institutional structures if they fail to understand and effectively target the needs of both women and men.” The review highlighted gender norms that affect the decisions under women's and men's control and hence climate information needs, and differing access to group and ICT communication channels, as key drivers of gender-based inequalities in rural climate services. Given the importance of gender-differentiated climate information needs, participatory group processes, and empowerment for gender-responsive rural climate services, our discussion of each case study focuses on evaluating one of these issues. The discussion of each case study includes the gender-related issue, summary of project interventions, key gender-related evaluation results, their relevance to the design and implementation of services, and evaluation methods employed.

Gender-responsive evaluation case studies

To what extent do women's and men's climate information needs differ?

Socio-cultural norms concerning livelihood roles that influence the resources and decisions under women's and men's control can affect the types of weather and climate information that are relevant to their livelihood decision-making (Tall et al., 2014; Carr and Owusu-Daaku, 2016; Carr et al., 2016a,b). Climate information needs can vary with gender and its interaction with other locally relevant socio-economic characteristics (Carr, 2014; Coulibaly et al., 2015; Poulsen et al., 2015; Tall et al., 2015; Carr et al., 2016b). By shaping demand, context-specific norms that influence the ability of women and men to act on particular types of climate information can be manifest as differing rates of information access. A farmer will invest effort to access available information only if it meets a perceived need.

Results from an evaluation of the Agrometeorological Advisory Program (AAP) in Mali highlight the importance of understanding how social norms can impact the usefulness of climate services and contribute to inequities in their use. The AAP was initiated in Mali's Southern Region in the early 1980s in response to frequent drought-influenced food insecurity. Its goal was to increase production of key staple and cash crops (millet, sorghum, maize, peanuts, and cotton) through agrometeorological advisories (Carr and Onzere, 2017). A multidisciplinary working group for agro-meteorological assistance (Groupe de Travail Pluridisciplinaire d'Assistance Agrométéorologique), comprised of Malian government offices and directorates, was formed to produce and disseminate advisory bulletins to smallholder farmers using radio, television, and extension services. AAP could be described as “gender-blind” in the sense that its goal was expressed in terms of aggregate crop productivity and did not consider equitable farmer well-being.

The evaluation found that, although some senior men made use of the advisories for several agricultural decisions, junior men and most women rarely used them (Carr and Onzere, 2017). Lack of control over land, lack of ownership of draft animals and plows, and limited decision-making power over the program's target crops generally restricted women from acting on advisories. Social norms in the region led to greater ownership of draft animals and plows among men and among more senior community members. Farmers who owned these assets were able to respond to weather-based plowing and planting date advisories, whereas farmers who lacked them needed to wait to borrow or rent the equipment and animals before they could plant. Senior women had somewhat greater opportunity to cultivate the program's target crops on their own fields than junior women, but only when they had time available after contributing to the family's male-controlled farmland. The resulting constraints to following the advisories in a timely manner made them largely irrelevant to their agricultural practice.

The gender-sensitive evaluation of AAP was conducted nearly three decades after the program's inception, too late to influence its design. USAID commissioned the evaluation to obtain evidence and insights to inform its own climate service investment strategy in Africa. The publications that came out of the evaluation contributed to awareness within the climate services community of how context-specific social norms can influence who can act on weather and climate information, and what types of information they may find relevant.

Qualitative methodological approaches can distinguish distinct user needs that exist among women and men and correspondingly, help accurately evaluate gender inequities in use. The Mali evaluation used the Livelihoods as Intimate Government (LIG) approach (Carr, 2013) to assess inequities in use of advisories. LIG is a qualitative, ethnographic method that analyzes smallholders' vulnerability context, as conditioned by their livelihood activities. From this understanding, the LIG then assesses the key agricultural practices and decisions for which farmers are responsible and determines behavior changes and decisions that may have been influenced by the advisories (Carr, 2014; Carr and Onzere, 2017). LIG employs focus group discussions, structured and semi-structured interviews, and participant observation. This and similar approaches are useful for producing contextualized, ethnographic information that can help identify gendered vulnerability contexts, critical for assessing women's and men's integration of weather and climate information into their adaptive strategies (Roncoli et al., 2009).

How do group participatory processes enable or constrain women and men?

Climate services use a range of communication channels, including broadcast media, ICT (e.g., mobile phones, internet) and group processes, to reach their intended users. Gender influences access to communication channels through the roles and responsibilities that women and men carry out in the household and community (Archer, 2003; Venkatasubramanian et al., 2014; Tall et al., 2015), differences in ownership of communication assets (Hampson et al., 2014; Owusu et al., 2017; Partey et al., 2018), and normative structures and institutions that work to favor men's participation over women's in extra-communal activities (Roncoli et al., 2003, 2009; CICERO, 2017). Whether a farmer invests in the effort to access available information depends, among other factors, on how well it fits her needs. Group participatory communication processes aim to go beyond dissemination of information, to build users' capacity to understand complex climate information and integrate it into their decision-making. Well-designed group communication and planning processes have proven effective in part because they provide opportunity for formal training and social learning, and they help overcome cognitive challenges to processing and using probabilistic information (Hansen et al., 2019). However, the nature of the participation determines whether they reduce or reinforce gender inequality in climate services. “Participatory” group activities sometimes disregard local power relations and effectively marginalize women or rather, certain groups of women or men, even when they might be facilitated access to the meeting itself (Roncoli et al., 2009; Elias et al., 2017).

Results from the evaluation of the Rwanda Climate Services for Agriculture (RCSA) project (2016-2020) show how group participatory processes can overcome barriers that women face with other communication channels. The project, led by the CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) and funded by USAID, sought to strengthen the capacity of the country's meteorological service, Meteo-Rwanda, to provide climate information tailored to the needs of the agriculture sector, and the capacity of its agricultural extension service, Twigire Muhinzi, to deliver climate services to women and men farmers, and support them to effectively use the information to manage their risk. The project adopted Participatory Integrated Climate Services for Agriculture (PICSA) to build farmers' risk management capacity, and trained Farmer Promoters–lead farmers who are part of the Twigire Muhinzi extension service–to facilitate the process with their existing mixed-gender farmer groups. PICSA is a group training and planning process that integrates local historical and forecast climate information with participatory decision-making activities to build farmers' capacities to make climate-informed decisions (Dorward et al., 2015). The project also piloted mixed-gender Radio Listeners' Clubs (RLCs) who meet weekly to listen to and participate in interactive climate services radio programs, and share plans to act on the information.

The project's endline evaluation showed that inclusion in participatory communication processes (i.e., PICSA and RLCs) improved women's and men's awareness and access to climate information (Birachi et al., 2020). Although gender inequities persisted for weather information accessed via radio and mobile phones, these participatory processes largely eliminated a gender gap in the use of weather and climate information that is apparent in the control sample. They also contributed to diminished gender inequities in investment in their farms, perceived coping capacity, and confidence in planning. A follow-up qualitative study, focused on the RLCs, confirmed that participation in RLCs eliminated the disparities in awareness, access and use of climate information that otherwise exist between women and men smallholder farmers; and that female RLC participants access climate information through a greater diversity of channels than non-participants or male RLC participants (Ingabire, 2021). RLC participation appeared to prompt women to seek out climate information, despite homecare responsibilities during climate services radio broadcasts and other gender-related access challenges.

The results of the RCSA evaluation studies largely support the project's strategy of investing in agricultural extension capacity to facilitate participatory communication strategies as the main mechanism to build farmers' capacity to access and act on climate information. The results also provide a caution against treating mobile phone or broadcast media dissemination channels as suitable substitutes for participatory communication processes. Among African countries, the government of Rwanda is known for its proactive approach to gender equity. Evaluation results suggest that future development of climate services could ensure that rural women benefit by strategically combining these complementary communication channels.

The RCSA's quantitative evaluation was able to assess the influence of PICSA and RLCs, alone and in combination, on awareness, access, use and benefit from climate information by sampling sectors (i.e., the third administrative level) where they had and had not been implemented within each sampled district, and by comparing with a non-participant control sample (Birachi et al., 2020). Disaggregating analysis by sex provided information about differences in information access and use among different groups within the population, with and without the influence of project interventions. Although quantitative methods identified associations between interventions and outcomes, they cannot necessarily confirm that these associations represent causal relationships. Qualitative methods, such as semi-structured interviews and focus group discussions, can help critically assess how interventions influence outcomes and impacts, identifying key factors and clarifying causal pathways to gender equity in access to climate services (Tall et al., 2018; Carr et al., 2019). New innovative methods may be necessary to assess what types of participatory processes (e.g., mixed gender vs. single gender) are appropriate for local gender dynamics and why. For example, Roncoli et al. (2011) used socio-linguistic analysis, in-depth interviews, and ethnographic observation to assess styles of participation in group discussions of seasonal climate forecasts and generate insights into how social processes influenced participants' perception of the new information and trust in its sources. Similar qualitative approaches can also help identify how group processes used for climate services can contribute to women's and men's trust and interest in seeking out weather and climate information.

How can climate service interventions empower women?

Gender-responsive evaluation can help identify potential pathways for climate services to contribute to women's empowerment. Our understanding of empowerment is informed by Naila Kabeer's conceptualization: “the expansion of the capacity to make strategic and meaningful choices by those who have previously been denied this capacity, but in ways that do not merely reproduce, and may indeed actively challenge, the structures of inequality in their society” (Kabeer, 2017: 651). Empowerment entails shifts in individual-level consciousness, as well as institutional-level processes that engage with deep-rooted normative beliefs about gender and power relations (Cornwall, 2016). While it may be challenging for climate services alone to effect changes in women's empowerment, there is some evidence that climate services may help local actors challenge restrictive gender roles (Mittal, 2016; Rengalakshmi et al., 2018). For example, climate services tailored to women's needs and interests have helped them to make informed agricultural decisions in cases in India; and their increased role in decision-making has influenced a shift in gender roles, wherein men are no longer the sole decision-makers and women are seen as more than farm laborers (Rengalakshmi et al., 2018).

The project, “Enhancing adaptive capacity of women and ethnic minority smallholder farmers through improved agro-climate information in Southeast Asia” (ACIS), demonstrates how evaluation can inform efforts to foster empowerment through climate services by considering the local normative and institutional environment. Led by the World Agroforestry (ICRAF) and co-implemented with CARE's country offices, the project was implemented in Vietnam, Laos and Cambodia over two phases from 2015 to 2021. It aimed to improve the adaptive capacity of women and ethnic minorities by enhancing their access to and use of agro-climate services (Simelton et al., 2018a). The project sought to build capacity through Participatory Scenario Planning (PSP) workshops (Ambani et al., 2018; Le et al., 2018), through which women and men farmers, including those from ethnic minorities, engaged in dialogue with scientists and service providers and provided feedback on the actionability of weather forecasts and accompanying agricultural information. The project partner CARE also provided gender awareness trainings. The group-based climate service communication and capacity development approaches that the project implemented were adapted to local gender dynamics and the project context (Huyer et al., 2021). For example, the project worked with women's Village Savings and Loan Associations (VSLAs) in sites in Cambodia, considering the limited voice that ethnic minority women have in public spaces and their constrained role in agricultural decision-making relative to their male counterparts in these contexts (Young Park and Maffii, 2017). In contrast, in a site in Vietnam the project approach relied upon mixed-gender groups established through a climate-smart village (CSV) (Aggarwal et al., 2018), recognizing the substantial, often interchangeable, roles that both women and men played in household farming decisions in the locality.

At project locations in Vietnam, local social norms already encouraged collaborative farm labor and joint decision making among spouses (Huyer et al., 2021). In this context, results from a preliminary evaluation indicated that participants perceived improvements in intra-household communication and increased participation of wives in farm decision making as a result of project interventions. Shared responsibility to participate in mixed-gender workshops appeared to facilitate these changes. Although increased participation in production-related decision-making is an indicator of agency (Alkire et al., 2013), additional data and analyses would be needed to assess whether the enhanced communication and participation in farm decision making significantly empowered women or challenged constraining gender roles.

Although the ACIS project did not complete the intended rigorous endline evaluation, lessons and guidelines for implementing gender-responsive rural climate services, based on the preliminary evaluation, were shared with relevant stakeholders in the region in the form of a policy brief (Simelton et al., 2019).

Qualitative and mixed-method evaluation approaches can help explain pathways to empowerment, highlighting context-specific assumptions important to the empowerment process. This enables better understanding of how climate services can influence the normative environment to create empowerment impacts. The ACIS project drew from prior years of rural development work addressing gender relations to develop a rapid assessment approach for its evaluation, based on single-gender focus group discussions, carried out with women and men PSP participants. Each respondent completed a set of 27 short answer and Likert-scale questions, and then discussed his or her responses in the focus group. The evaluation recorded the group discussions and collected the individual questionnaire responses, in this way permitting qualitative and quantitative data collection. Correspondingly, information was collected concerning changes in effective use of the agro-advisories for farm planning, outcomes from their use, and major benefits perceived. The project team applied their knowledge of local gender relations to analyze the data (Simelton et al., 2018b). The qualitative results from the ACIS assessment are useful for explaining how the gender normative environment influences changes in household decision-making processes. Qualitative research can help capture the nuances of gender relations in local rural livelihoods and clarify the normative and institutional changes upon which empowerment processes depend (Tavenner and Crane, 2022). Recently, Tavenner and Crane (2022) recommended that qualitative, participatory methods be used as a first step in evaluation methodologies, to analyze local gendered relationships in rural livelihoods. This can allow for defining contextually validated indicators of empowerment, which can then be incorporated into surveys to assess broader patterns. Such approaches can be applied in rural climate services evaluation, to accurately understand and identify pathways to empowerment, per varying local contexts.

Discussion and conclusions

The questions that are relevant for gender responsive evaluation can be informed by existing knowledge of how inequities can arise, and by context-specific, gender-related climate service goals and challenges. Informed by a previous literature review, our three case studies illustrate how questions about climate information needs, access to information and support through group processes, and contribution of climate services to empowerment were framed and answered. The results of these evaluations, which have been reported in greater detail elsewhere, contribute to the body of knowledge about interactions between gender and climate service interventions.

Using evaluation to learn about the extent that user needs are met by a climate service is needed to understand gender inequities in use. Qualitative methodologies can identify and assess differentiated user needs; furthermore, they provide needed explanation of how the gendered vulnerability context interacts with locally specific identities, roles, and responsibilities to shape opportunities to use climate information in decision-making. Evaluation that provides evidence for how group processes can contribute to women's and men's demand for climate and capacity to act on it can guide investment in communication channels that promote equitable access. Quantitative methodologies and sampling methods that appropriately differentiate potential intervention effects can help assess changes in demand; however, qualitative methodologies may be needed to understand the nuances of how participatory communication processes contribute to demand. Evaluation that critically considers the local normative and institutional environment surrounding gender can help address significant knowledge gaps concerning climate services' potential to contribute to women's empowerment. Qualitative and mixed methods can be designed to assess the context-specific normative and institutional changes necessary for empowerment processes and understand how climate services can contribute to empowerment impacts in a given context. The targeting of these knowledge gaps through gender-responsive evaluation facilitated through rural climate service researchers, implementing organizations, and funders can help climate services achieve enhanced outcomes and impacts; it also contributes to greater resilience and sustainable development for rural populations.

One of the goals of evaluation is to provide insights and evidence to guide improvements in a program or service. Empirical evaluations of time-bound projects typically conducted near the end of the project can be too late to influence the project's design and implementation. In the case of Mali's Agrometeorological Advisory Program, a gender-sensitive evaluation wasn't conducted until nearly three decades after the program's inception. Despite the timing challenge, project endline evaluations can contribute to improvements to climate services if: (a) evaluation results are published and shared widely within the climate services community; (b) climate service implementers and funders consistently factor general and context-specific knowledge and evidence, available from prior evaluations, into the design of new or improved services; and (c) ongoing climate service initiatives conduct preliminary evaluations early and regularly, to support mid-course adjustments. Empirical evaluations, such as those that this paper discusses, are necessary but not sufficient to ensure that climate services equitably meet the needs of their intended beneficiaries and contribute to development goals. To achieve these goals, periodic evaluation studies should complement ongoing engagement of user communities in iterative co-production, feedback and learning.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

TG led paper conceptualization and writing. JH, SH, ES, and SP contributed to initial idea for the article, drawn from a working paper. SS contributed to a literature review for the paper. JH, EC, and SH contributed to conceptualization of the present version of the paper. JH led revisions, and EC, BC, and SH contributed to revisions and editing. ES contributed inputs for the ACIS case referred to in the paper. All authors contributed to the article and approved the submitted version.

Funding

This work was supported by the CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS), which was carried out with support from the CGIAR Trust Fund and through bilateral funding agreements (for details please visit https://ccafs.cgiar.org/donors).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^http://www.climate-services.org/about-us/what-are-climate-services

References

Aggarwal, P. K., Jarvis, A., Campbell, B. M., Zougmoré, R. B., Khatri-Chhetri, A., Vermeulen, S. J., et al. (2018). The climate-smart village approach: framework of an integrative strategy for scaling up adaptation options in agriculture. The climate-smart village approach: framework of an integrative strategy for scaling up adaptation options in agriculture. Ecol. Soc. 23, 14. doi: 10.5751/ES-09844-230114