The Norwegian Refugee Council’s Protection Prevention M&E, Analysis, and Learning Plan

The Norwegian Refugee Council (NRC) developed its Civilian Self-Protection (CSP) program to support civilians to disrupt patterns of violence, coercion, and deliberate deprivation. The program builds on civilian agencies, strengthening their existing or developing self-protection capacities to proactively reduce protection risks and complementing those efforts through relevant external interventions. Rather than offering a set menu of activities linked to a predefined logframe, the CSP program advocates deep relationship building and collaboration with communities to jointly identify existing self-protection strategies and reinforce local capacities related to conflict analysis and protection, informing NRC’s design of complementary activities.

This type of adaptive and prevention-focused programming can be difficult to measure. The challenge of monitoring when a behavior or incident doesn’t happen—let alone assessing the specific contributions of an individual program or organization—is compounded by a frequent lack of baseline data, short timeframes of humanitarian programs, and dynamic contexts. As a result, NRC rethought its monitoring and evaluation (M&E) approach to assess its CSP programming.

In the development of their own integrative learning approaches, NRC identified InterAction’s Gender-Based Violence Prevention Evaluation Framework (GBV PEF) as one of few available resources that offers innovative ways to monitor and evaluate prevention outcomes. NRC’s resulting Protection Prevention M&E, Analysis, and Learning Plan aims to strengthen protection and conflict analysis by teams and communities, inform locally led programming, and assess how CSP programs contribute to changes in local perceptions of safety and the prevention of violence. The plan focuses on outcome-level results with an emphasis on qualitative data, deepening community participation, and adopting alternative M&E methods. While quantitative data is not ignored, qualitative data takes center stage, strengthening the understanding not only of what has happened or what changes took place, but also how and why such changes occurred. The Protection Prevention M&E, Analysis and Learning Plan is currently in being piloted in Colombia, Afghanistan, and Palestine.

The Protection Prevention M&E, Analysis, and Learning Plan was designed to be an integrated and inseparable component of NRC’s broader CSP program, whereby protection analysis activities—such as community mapping and perceptions of insecurity exercises—that are traditionally viewed as serving only programmatic functions are also used for M&E, including measurement of progress against program indicators. If conducted using the same tools at the beginning and end of a program, teams can compare the results to understand prevention outcomes related to community capacities, perceptions, and behaviors.

Similarly, methods that are often perceived as only for measurement—such as theories of change and observation forms—are built directly into the program implementation to inform continuous context-specific protection risk analysis, activity design, and adaptation. Recognizing the persistent human resource constraints that most teams face, NRC adopted this approach of blending program and M&E activities to ensure efficiency while generating rich, qualitative data that could be leveraged for multiple purposes, including protection advocacy. This integration also helps to mitigate assessment fatigue and overreliance on quantitative data. Moreover, given the sensitivities of discussing certain protection issues with communities, this approach enables team members to invest in more consistent engagement, relationship building, and data collection, as opposed to one-off visits to request information about local experiences of violence and conflict.

In addition to these qualitative approaches, NRC invested in developing performance indicators that would both fulfill standard protection requirements from donors and enable contextualization by program teams. In particular, the indicator definitions were written broadly enough so that teams would not be anchored to implementing specific activities, but rather have space to engage communities in determining their own priorities and actions. Teams were encouraged to include these indicators into their programs in order to mitigate potential reluctance from donors to support the inclusion of outcome-oriented monitoring approaches and to create space for locally led approaches. Consultations with CSP project participants in Afghanistan suggested that there is indeed greater need for teams to engage communities in defining desired protection outcomes and understanding how identified self-protection measures can contribute to those outcomes. NRC has responded to this feedback by investing in more systematic and frequent monitoring activities to facilitate this joint understanding.

The development of the M&E toolkit required creative, multi-sectoral thinking and a substantial investment of resources. While none of the activities and tools outlined in the Protection Prevention M&E, Analysis, and Learning Plan are completely new, they have either not been leveraged for program M&E purposes or have not been widely harnessed by protection actors. Instead, these types of methods have been used more readily in the peacebuilding and development sectors, despite their clear relevance to protection prevention programming. Even identifying consultants who were open to employing such innovative methods proved challenging. Ultimately, NRC invested considerable technical and financial resources to fully design and pilot the toolkit over the course of two years.

The development of the M&E toolkit required creative, multi-sectoral thinking and a substantial investment of resources. While none of the activities and tools outlined in the Protection Prevention M&E, Analysis, and Learning Plan are completely new, they have either not been leveraged for program M&E purposes or have not been widely harnessed by protection actors. Instead, these types of methods have been used more readily in the peacebuilding and development sectors, despite their clear relevance to protection prevention programming. Even identifying consultants who were open to employing such innovative methods proved challenging. Ultimately, NRC invested considerable technical and financial resources to fully design and pilot the toolkit over the course of two years.

Once the M&E toolkit was developed, NRC found that its biggest challenge was to enable effective uptake by program teams. Teams were predominantly used to implementing prescriptive activities focused on delivery of concrete humanitarian relief with predetermined indicators and targets, rather than facilitating the participatory design of community-led programming. Moreover, although the Protection Prevention M&E, Analysis, and Learning Plan presents a clear and sequential set of approaches that teams should conduct throughout the assessment and evaluation phases of a CSP program, it also provides flexibility for teams to select which specific tools are best aligned to the local context. For example, while seasonal calendars are highly relevant in some Colombian communities, where cycles of violence are often associated with the coca harvest, it was not an appropriate tool in others. This flexible approach requires that teams employ critical thinking to make decisions about operating in complex and dynamic contexts.

While many teams are familiar with monitoring of quantitative indicators, the Protection Prevention M&E, Analysis, and Learning Plan requires teams to engage with large amounts of qualitative data to identify intangible social behavior changes within communities. In one case, community participants in a CSP program in Colombia recounted a story of how painting a mural enabled them to reclaim the area as a safe space for community activities and even created a precedent for dialogue with armed actors. However, the program team reported this change as a simple output involving the painting of a wall. This mentality necessitates a critical shift in organizational cultural and team decision making that takes time to nurture.

NRC found that intensive and consistent coaching was the most effective way to encourage this shift in mindset and skills, and to nurture a sense of ownership and confidence among its program teams. Field-level advisors—rather than those based in country or regional offices—played a particularly critical role in supporting an iterative process of conducting demand-driven training on new concepts and resources, testing and refining tools based on feedback from teams, and providing detailed critiques on reporting during regular field visits and virtual sessions. This active and ongoing accompaniment strengthened the analytical and writing skills of the teams and ensured that they had sufficiently detailed and tailored guidance to implement the Protection Prevention M&E, Analysis, and Learning Plan approaches with autonomy. As a result of this investment, NRC has witnessed a considerable improvement in teams’ understanding of what has changed, as well as how and why, which has deepened NRC’s protection analysis contributions to prevention. 

As the focus on qualitative data has increased, NRC has identified a gap in its internal team capacities to effectively collect, clean, analyze, and harness this data. Notably, teams have recognized the potential value of learning new qualitative analysis techniques through both Excel and specialized software, such as MaxQDA, although the latter hasn’t yet been put into practice. The integration of programmatic and M&E activities under the holistic model of the Protection Prevention M&E, Analysis, and Learning Plan also necessitates deeper collaboration between M&E and protection colleagues, especially through Information Management Officers, in order to harness their respective skills sets and balance ongoing demands on M&E teams to continue fulfilling traditional reporting requirements. As such, NRC is investing in providing joint training for its field and country-level protection and M&E teams in Palestine to deepen this collaboration and ensure responsible data management across all team members.

• Integrate programmatic and M&E activities: Protection programs should consider developing methods and tools that leverage the natural synergies between activities intended for protection analysis and program M&E. This integration not only has the potential to generate rich qualitative information, but also to improve efficiency for teams that face time and resource constraints. By extension, organizations should promote deeper collaboration between program and M&E teams in order to realize this integration.
• Balance standardized tools with contextualization: Despite the uniqueness of each operating context, didactic and user-friendly tools are often the most accessible and actionable resources for field teams to adopt. Protection programs should therefore consider developing a menu of approved tools, while decentralizing decision making to field teams to assess which ones are most appropriate and relevant to their context. Field-level advisors should also collaborate closely with field teams to develop tailored guidance and further refine tools for local use.
• Invest in consistent coaching of field teams: Measuring prevention outcomes requires an inherent shift in the norms and culture of protection actors. While developing and disseminating new frameworks and tools is a critical step, protection programs should invest in sustained and intensive coaching to ensure meaningful uptake by field teams. Notably, field-level advisors should engage in iterative accompaniment approaches to nurture critical thinking skills and local ownership by field teams, not only to gather quality qualitative data, but also to analyze it and translate it into action. Organizations should encourage and dedicate time for regular and intentional practice-based learning to deepen this culture and further share emergent lessons.
• Allocate sufficient staff resources: These qualitative approaches require considerable time to nurture internal capacities, conduct thoughtful data collection and analysis, and support consistent collaboration across protection and M&E teams. In order to support these efforts, organizations should plan to build sufficient staff into their budgets, including both field-level team members and technical advisors at all levels for the duration of the program cycle.
• Advocate for outcome-oriented learning within the humanitarian system: Ultimately, adaptive and sustainable outcome-oriented program learning demands that humanitarian actors at all levels are committed to and engaged in the process. Donors must be prepared to invest in multi-year strategies and innovative qualitative M&E techniques. Organizations must have the will to share their experiences across contexts and agencies. Practitioners must have the resources, skills, and culture to effectively document learning. Collective advocacy is essential to embedding these principles within the humanitarian system moving forward.