Equal Acess International
In 2020, Equal Access International (EAI) began implementing the Securing Nigerian Communities program, a four-and-a-half-year initiative that aims to improve civilian security across four states of North West and North Central Nigeria. The program adopts an integrated, norms-change approach, combining multi-sectoral efforts to establish and/or strengthen existing community security mechanisms, while reducing acceptance of violence and encouraging communities to work toward peaceful coexistence. While not explicitly focused on gender-based violence (GBV) prevention, the program seeks to increase women and youth’s safe participation in civilian security efforts, fostering wider community acceptance of the positive roles that they can play in building peace.
The program was initially designed to include Outcome Harvesting as a core monitoring and evaluation approach. Considering the strong focus on radio drama and community dialogue, EAI had recognized the qualitative nature of the programming and advocated for the use of Outcome Harvesting as an effective method for assessing progress toward the complex social behavior changes that the program was seeking to achieve. The donor was also interested in hearing stories from community participants and believed that Outcome Harvesting could give them access to local voices and illustrative examples. Despite this strong institutional buy-in, the EAI Nigeria team did not have previous experience designing and implementing an Outcome Harvest and sought support to translate their plans into action.
- Outcome Harvesting
Outcome Harvesting is a monitoring and evaluation technique designed to capture clear examples of change in the community, and then work backward to assess the contributions of a project or program toward that change.
Operationalizing Outcome Harvesting
As EAI began formulating plans to integrate Outcome Harvesting into the program, they identified InterAction’s Gender-Based Violence Prevention Evaluation Framework (GBV PEF) as a valuable resource for supporting organizations to measure and evaluate their prevention outcomes in innovative ways. The Technical Advisor participated in associated activities facilitated by InterAction, including a webinar on Outcome Harvesting aimed at deepening practitioner competencies and regular community of practice sessions that gave space for exchange on challenges and good practices that organizations were experiencing in attempting to pilot the outcome-oriented methods described in the GBV PEF.
Although the team had already developed and begun populating an Outcome Harvesting database, the insights gleaned from the GBV PEF enabled the Technical Advisor to further tailor the tools, guide the Securing Nigerian Communities program, and train the field team. The training focused on empowering team members to identify and capture various types of social behavior change outcomes and shifts in social norms. The training relied heavily on practical exercises and examples to strengthen the capacity and confidence of the team to generate quality content for each component of the Outcome Harvesting database, including outcome statements, contribution statements, and significance statements. It also sought to nurture a spirit of collective ownership of the process to ensure that both M&E and program team members recognized their roles in proactively observing and documenting complex changes, so that the program could more readily and holistically tell the stories of its impact.
- Contribution statements
Contribution statements present evidence about how the program may have influenced an observed outcome. These statements help answer questions such as: How (if at all) did the program influence this change? What did the program do that helped to bring this change about? Answering these questions enables programs to make clear linkages between their activities and observed results.
To do so, the program established a process of gathering outcomes through monthly activity reports, quarterly progress reports, and focus group discussions that were conducted and compiled by program officers and field monitors. The Senior M&E Officer, Technical Officer, and Technical Manager then extracted potential outcome statements from the reports to populate the Outcome Harvesting database. The team developed a color-coding system to identify preliminary entries that were incomplete or self-reported, which were referred back to the field team for further substantiation. This validation process entailed direct engagement with program participants to verify and add details to the initial outcome statement, and to further unpack the program contribution and significance of the change from the perspective of local stakeholders. This information was subsequently added to the Outcome Harvesting database. To date, the program has documented more than 90 distinct outcomes.
The process of gathering and substantiating outcomes was reinforced through consistent coaching by senior staff. This included regular calls to provide feedback on the content and phrasing of outcome statements and to guide the field team on how to look for relevant information in order to develop stronger statements. Senior staff also continuously emphasized the importance of adopting a facilitative style for the follow-up conversations to substantiate emerging outcomes with participants. This approach helped the team to avoid extracting information about preconceived outcomes and, instead, to encourage reflection, even if it contradicted team assumptions. This intensive accompaniment approach was incredibly time- and resource-intensive but ultimately resulted in high-quality data
Finally, the team analyzed the outcomes in the database by identifying recurring themes and comparing them to the expected outcomes that were initially outlined in the program log frame. In some cases, activities and results diverged from the proposal. For example, in line with their desired outcome that underrepresented women and youth lead localized prevention and response efforts, the EAI team documented clear examples of how their activities had contributed to the improved involvement of women in local civilian security and peacebuilding initiatives through their increased representation in community structures and their active role in sensitization and advocacy activities.
However, the Outcome Harvesting process also helped to unearth unexpected outcomes, such as a spontaneous initiative by women leaders to lobby government officials to develop local action plans, including in areas beyond the program target localities. This process of “backward mapping” to explore the relationship between program implementation and the observed results enabled the team to not only demonstrate their direct contributions to outcome-level changes, but also to refine their causal logic in a way that informed the design of a new program.
For traditional approaches, you have your baseline survey with your KIIs and FGDs, but a lot of that subtle detail still gets missed out on. We really saw the value of the Outcome Harvesting approach to identify community motivations and how things have changed.
Arti Lad, former Technical Officer, EAI
REFLECTIONS ON THE PROCESS
Many of EAI’s approaches for operationalizing Outcome Harvesting worked well, with its various training and accompaniment techniques supporting genuine ownership and expertise by the field team in a way that led to robust results. Despite these efforts, EAI faced consistent challenges related to the time and motivation of the field team to collect qualitative data. Although the program had always envisioned including Outcome Harvesting as one of its primary M&E methods, it also continued to monitor standard indicators that required regular collection of quantitative data. Since these indicators were more familiar to the field team and more explicitly linked to the program log frame and reporting requirements, Outcome Harvesting activities were consistently deprioritized and delayed. Consequently, the team struggled to identify opportunities within the program cycle to reflect on the results emerging from the Outcome Harvesting and to collectively translate them into learning and program adaptation.
The team also recognized limitations in the extent to which their Outcome Harvesting process was participatory and inclusive of community participants. While the field team directly engaged local stakeholders to substantiate their outcome statements, the analysis and formulation of conclusions were still conducted solely by the technical team. This meant that the team ultimately decided on which outcomes and themes to focus on based on the significance to the program, rather than on the perspectives and perceived significance to target communities.
“Adopting the Outcome Harvesting methodology provided a new learning opportunity to the Nigerian MERL team to really strengthen our qualitative data collection skills, and also opened up the team to a totally new way of viewing program impacts…Using [outcome-oriented] approaches also deepened our understanding of unexpected outcomes, which allowed us to be more intentional about looking out for them during data collection processes.”
Fatima Turaki Ibrahim, Senior Monitoring and Evaluation Officer, EAI
RECOMMENDATIONS FOR PRACTITIONERS
· Emphasize collaboration at all levels – Robust Outcome Harvesting demands consistent coaching to ingrain new skills and ways of thinking for teams to produce quality data. Programs should promote a spirit of collaboration between the senior technical and field teams in order to create space for effective accompaniment. This same value of collaboration is needed between field teams and program participants to ensure effective substantiation and outcomes in a way that centers local voices in the process. There may also be potential for collaboration with local stakeholders in the analysis of outcomes.
· Allocate sufficient resources – This level of collaboration requires extensive time commitment from teams over an extended duration. Programs should account for this investment during the design phase by allocating sufficient funding in the budget and time in work plans for teams to contribute to effective Outcome Harvesting.
· Start the process early – Ideally, Outcome Harvesting should commence at the outset of the program cycle and be embedded into every stage of program implementation. This deep integration can ensure that the program and M&E teams as well as program participants are actively involved in the process of identifying and defining outcomes. Starting early also gives more time for intangible behavioral changes to be observed over a longer period, so that patterns can be more easily identified in the data.
· Identify opportunities for consistent uptake – While Outcome Harvesting may not appear to align naturally with the program cycle, programs should proactively schedule regular sessions for teams to reflect collectively on emergent learning, identifying short-term opportunities for adaptation as well as long-term opportunities to inform future, evidence-based program design.