Module 3: Measurement Considerations

Vigaud-Walsh (2020) reviewed a large number of GBV prevention projects and programs, including their associated logframes[1] and monitoring frameworks. The majority of these programs were monitoring output and activity-level indicators. In some cases, this included community perceptions of the quality of GBV prevention services. But in many, measurements were restricted to the quantity of activities conducted and the number of persons served.

There are reasons for this “retreat” to output measurement. Organizations consulted in the design of this framework cited the following challenges when it comes to monitoring outcomes for GBV work:

• Risk: Surveying community members about GBV incidence rates risks causing harm to vulnerable community members and, in some instances, to program staff. This makes it hard to collect primary data in the first place.
• Trust: Asking community members to share their perceptions of GBV requires a high level of trust in the community, which can be difficult when program staff turnover is high or when M&E staff from outside the program team are collecting data.
• Sensitivity: It is often difficult to discuss sensitive topics around sexual violence with community members, making data collection about GBV incidence unreliable even when it can be carried out without causing harm.
• Privacy and data management: Even when the data is collected in a reliable manner, it is often difficult to share sensitive case-data, or any data that could reveal the identity of vulnerable groups. This makes it difficult for M&E teams to access any results data collected.
• Cost: Robust measurement of changes in GBV incidence over time requires significant investment in measurement tools and frameworks that are very challenging to fund in the current humanitarian funding landscape. This could change if donors and implementing agencies decide to take a strategic approach to evidence generation for GBV prevention. But until it does, it will remain prohibitively expensive for organizations to measure GBV incidence in a rigorous manner that allows for attribution claims to be soundly made.

3.2.1. Why indicators matter

Well-designed indicators allow M&E teams to measure the progress of a project towards its goal. Without clearly defined indicators, even the clearest theory of change can be hard to test.

Monitoring frameworks should include indicators across the results-chain, including indicators for outputs, results and outcomes of the activities:

DEFINITIONS BOX 3:[2]

3.3.1. GBV Guiding Principles

Throughout the program cycle, staff will undoubtedly encounter GBV survivors. Survivors who disclose an incident of GBV are often at high risk of stigma and further violence by the perpetrator or others. To safeguard against this, the survivor-centered approach is employed in all interactions with GBV survivors, including data collection for the purposes of monitoring of evaluating prevention programs.

A survivor-centered approach is a supportive environment in which survivors’ rights and wishes are respected, their safety is ensured, and they are treated with dignity and respect. It is underpinned by the following guiding principles:

Safety: The safety and security of survivors, their children, and those assisting them are the primary considerations. Safety refers to both physical safety and security and to a sense of psychological and emotional safety.

Confidentiality: Survivors have the right to choose to whom they will or will not tell their story, and any information about them should only be shared with their informed consent. Maintaining confidentiality means not disclosing any information at any time to any party without the informed consent of the person concerned. Information about a survivor’s experience of abuse should be collected, used, shared and stored in a confidential manner.

Respect: All actions taken should be guided by respect for the choices, wishes, rights and dignity of each survivor.

3.3.2. The “Do No Harm” approach

The “Do No Harm” centers on taking measures to avoid exposing people to harm as a result of our work. This means making sure that the actions we take do not create further GBV risks or any other kind of harm for survivors and others at risk. The “Do No Harm” principle includes:

• Avoiding any actions that might expose a survivor or person at-risk to acts of revenge or further violence.
• Making all communication and interactions safe and supportive to avoid the traumatization of survivors.
• Ensuring that the collection, storage, and sharing of information abides by the GBV Guiding Principles, and does not create any additional risk.

3.3.3. Ethical considerations for data collection and use

As working with survivors, or those at risk of GBV, engenders significant safety and security concerns, the research for the design, monitoring, and/or the evaluation of GBV programs—including prevention programs—must be undertaken with extreme care and sensitivity to respecting GBV Guiding Principles and the “Do No Harm” Approach. This is particularly relevant when it comes to collecting and using data.

The International Committee for the Red Cross has developed professional standards that we should adhere to ethically to manage sensitive protection information. They include, but are not limited to, collecting information for the use of protection programs, endeavoring to collect such information if the organization has the right information management system to process and store data confidentially, evaluating the scope of information to be collected in relation to its relevance for protection programs.

With regards to GBV specifically, the World Health Organization developed eight safety and ethical recommendations to be considered before researching violence against women and girls. This is applicable to all GBV-related data collection activities, including needs assessments, surveys, and the monitoring and evaluation of interventions. The eight recommendations are:

1. Analyse risks and benefits: Before collecting any data, it is important to consider both: (1) potential risks that respondents and data collectors may experience, and (2) potential benefits to the affected community and the wider humanitarian community. It is critical that the benefits outweigh the risks.

2. Methodology: Data collection activities must be safe and survivor-centered, methodologically sound and not time intensive.

3. Referral services: Basic care and support to survivors must be available locally before commencing any activity that may involve individuals disclosing information about their experiences of violence.

4. Safety: The safety and security of all those involved in information gathering is a primary concern and should be monitored continuously. Safety and security conditions should be regularly incorporated into the security protocol.

5.  Confidentiality: The confidentiality of individuals who participate in any data-collection activity must be protected at all times. Data should be collected anonymously where possible.

6. Informed consent: Anyone participating in data gathering activities must give informed consent. Before collecting data, all participants need to be informed of the purpose of the exercise, the risks they may face, and the benefits (including any monetary or in-kind compensation) they can expect to receive due to their participation.

7. Information gathering team: The data gathering team must include women. All members must be selected carefully and receive relevant and sufficient specialized training and ongoing support.

8. Children: Additional safeguards must be established if children (i.e., those under 18 years old) participate in information-gathering.

For more detailed information, please see:

• World Health Organization (2007). Ethical and Safety Recommendations for Researching, Documenting and Monitoring Sexual Violence in Emergencies. Geneva. https://www.who.int/gender/documents/OMS_Ethics&Safety10Aug07.pdf
• ICRC (2009). Professional Standards for Protection Work Carried Out by Humanitarian and Human Rights Actors in Armed Conflict and Other Situations of Violence. Geneva. https://www.icrc.org/en/ doc/assets/files/other/icrc_002_0999.pdf
• CHS Alliance, Group URD and the Sphere Project (2014). Core Humanitarian Standard on Quality and Accountability. https://corehumanitarianstandard.org/files/files/Core%20Humanitarian%20 Standard%20-%20English.pdf 
• IASC (2019) Guidelines for Integrating Gender-Based Violence Interventions in Humanitarian Action https://gbvguidelines.org/en/

3.4.1. What is evaluability and why does it matter?

The term “evaluability” refers to the extent to which an “activity or project can be evaluated in a reliable and credible fashion”.[8] Projects with poorly specified risk analyses, theories of change, or monitoring frameworks, are invariably hard to evaluate in any meaningful way. By assessing our programs for evaluability during the project design stage, we can help improve those project designs themselves. This is especially important for GBV prevention given the project-design weaknesses discussed in the introduction to this framework.

Evaluability assessments are often conducted by donor agencies and INGOs prior to the commissioning of independent evaluations of their projects and programs. They are usually conducted in order to assess:[9]

1. Theoretical evaluability: given the project design as it currently stands, how possible is it to measure intended results and the desired outcome? Does the project have clear objectives? Are those objectives translatable into measurable indicators?
2. Practical evaluability: is it feasible to collect all the data necessary for the evaluation of the project? Are there any access constraints blocking primary data collection? Are all the key stakeholders sufficiently engaged and supportive of an evaluation at this time?
3. The usefulness of an evaluation: who is most likely to use the evaluation and how? How would the evaluation complement other monitoring and research activities related to the project?

Whilst each of these questions are worth asking prior to conducting an evaluation, they are also valuable questions to ask during project design itself. This can help sharpen the clarity of the project design and highlight any areas of confusion in the theory of change. Considering evaluability at project design stage can also help M&E teams to design appropriate monitoring frameworks from the outset. And ultimately, early consideration of evaluability can be a useful means to improving the evidence-base for future project approaches, a point of particular value for GBV prevention actors.[10]

3.4.2. How to design for evaluability

Designing for evaluability means conducting project design in a manner that supports evaluability throughout implementation. Key elements to consider in this regard include:

• Basing the project design on a clear context-specific risk analysis, following the approach outlined in Module 1.
• Developing a clear theory of change including assumptions and evidence, as outlined in Module 2.
• Identifying clear and observable indicators in a manner that supports relevant data collection against them.
• Planning, and budgeting, for data collection ahead of time.
• Each of these things can help sharpen the project design. And when implemented well, can help project teams course correct during implementation.

The following section presents a quick evaluability check-list, designed for use by project teams during project design. It focuses on the clarity of the project design and the type of evidence that will need to be monitored during implementation, in order to conduct a useful evaluation. No specific evaluation, monitoring or data skills are needed to use this checklist, and the assessment is designed to be possible within the equivalent of 0.5 working days.

3.4.3. Example evaluability checklist

The checklist below is adapted from Davies (2013), ALNAP (2016) and Dillon et al. (2019). It covers key questions for project teams to consider when designing GBV prevention programs, to ensure evaluability and support the design of an appropriate monitoring system during implementation.

3.5.1. What is outcome mapping and how can it help?

Outcome mapping is a method for planning, monitoring and evaluating projects and programs that aim to achieve lasting social and behavioral change. It was originally designed by the International Development Research Centre in Canada, with the first guidebook being published in 2001. Since then, the method has been developed and used across a wide range of development and program contexts, and has been adapted and built upon by many of the organizations using it. An online learning community has been established to help program managers learn about outcome mapping, and includes a range of useful resources for anyone seeking to learn more. The community is available at https://www.outcomemapping.ca/start-here.

Outcome mapping has a range of potential uses for organizations working to prevent GBV in humanitarian contexts, including:

• It can help program teams understand complex behavior change within a community over time. This is useful for teams who want to better understand how their activities are influencing changes in the behaviors of perpetrators, vulnerable groups, and the wider community.
• It can help teams think about the pathways to change underlying their program logic. This is useful when trying to understand how the pre-conditions and underlying factors for GBV change and evolve over time.
• It is particularly useful for mapping and observing wider changes across a community, beyond the direct intended results of the program. This can help teams understand how GBV prevention activities conducted with a specific target audience can influence wider community changes beyond the direct program participants.

But the difficulty of using outcome mapping in humanitarian contexts is time and labor resources needed to make it work; and the need to think about outcome mapping across the project cycle: from initial design stage, through implementation and final evaluation.

To implement a “full” outcome mapping approach, program teams need to follow each of the 12 steps outlined in the illustration below, spread across project design (stage 1) implementation (stage 2) and evaluation (stage 3):

Diagram 4: Outcome Mapping Steps

Each of these steps requires a participatory approach. For example, “Step 1: Vision” involves working with community members to identify the big-picture change that they want to achieve. This is normally a very ambitious picture, covering multiple areas of social interaction. For example, a GBV prevention team in Cox’s Bazar might have a vision such as “we want all migrants living in Cox’s Bazar to be free from the threat of all forms of gender-based violence, including intimate partner violence, discrimination based on gender norms and sexual exploitation and abuse by humanitarian workers.” Designing this vision would typically require running several workshops with community-members and program teams, working together to describe the community they want to live in. And this type of activity needs to be replicated for each of the 12 steps of the outcome mapping process.

3.5.2. A streamlined version for humanitarian contexts

The difficulty of doing this in humanitarian contexts is outlined in Module 1. Humanitarian teams typically don’t have significant time to invest in project design, with proposals often being designed in 2-4 weeks-time.

For this reason, this framework proposes the following streamlined outcome mapping approach, which isolates the key elements of most potential benefit, whilst reducing the number of steps required for successful implementation in a short timeframe:

Diagram 5: Streamlined Outcome Mapping Approach

This version includes just four steps:

• Step  1: Vision. Describing the big-picture vision that the program (or country office) wants to achieve over the medium-term.
• Step 3: Boundary Partners. Choosing a number of key program stakeholders, who will interact directly with the program activities (e.g. as participants to GBV awareness-raising workshops) but who also have influence across the wider community (e.g. through involvement in women’s support groups or men’s social networks).
• Step 5: Progress Markers. Identifying the key behavior changes within the community that will lead up to the overarching change you are trying to bring about.
• Step 9: Results Journals. Designing journal tools for boundary partners to use as a way to track the changes described by the progress markers identified in Step 5.[11]

The following sections outline each of the first three steps of this process, which should all be undertaken during project design stage or, where possible, before proposal design itself. Step 9, results journals, is covered in Module 4 below, as it relates directly to evaluation tools and approaches.

3.5.3. Step 1: Vision

The program vision is the large-scale community-wide scenario that the program wants to contribute to bringing about. It typically combines descriptions of the ideal economic, political, social or environmental situation the program is working towards. For this reason, it is best to think in terms of longer-term goals and social change. One way to do this is to link the vision to longer-term strategic planning, beyond the life-cycle of an individual program. For example, a country-team working on GBV prevention in rural Afghanistan could try to map out where they want rural communities to be in 2-3 years with regards to gender-based violence risk. Questions to ask might include:

The contribution of the organization, and the individual programs that it implements over the next three years, might only be one part of this vision. But it is still important to identify the vision at design stage, particularly in protracted crisis contexts where the organization has, or intends to have, a continued presence over the medium-term.

This tool should be easy to integrate into pre-existing strategic design processes that the organization already undertakes as part of its GBV prevention work. In particular, it doesn’t require any additional data collection as such, so need not add significantly to the time or resource burden on project and program teams. Moreover, the work of defining a medium-term vision is closely related to the task of building a country- or area-wide theory of change, as outlined in Module 2, above. As such, it is recommended that teams wishing to develop a theory of change at this level of analysis try to develop something close to the vision statement outlined above as one outcome of the theory of change design process. This can then help the teams come back to the overarching vision as they adapt their theory of change over the program cycle.

One critical element of the outcome mapping approach is to emphasize participatory approaches throughout. As such, the design and elaboration of a vision such as this should ideally be done through community-based discussion and workshopping. This kind of activity can present risks of harm when discussing GBV risk with community groups. It is therefore essential to take a “Do No Harm” approach to consultations of this type.

Nevertheless, program teams are encouraged to integrate questions about the long-term vision within pre-existing community consultation activities, wherever possible encouraging community members to help co-design the long-term strategy of their organization’s activities in the communities they serve.

3.5.4. Boundary Partners

Boundary partners are key project stakeholders and partners, who will interact closely with the project activities themselves, but who also have the power to influence change across the wider community beyond the life-cycle of the project itself. Examples of boundary partners include:

• Sex workers participating in GBV prevention training and capacity-building programs.
• Senior military personnel taking part in training on the obligations of military actors under international humanitarian and human rights law.
• Community and religious leaders taking part in GBV awareness-raising activities and events.

The nature of GBV prevention work often entails that program teams already work closely with boundary partners during implementation. GBV prevention programming – its design and execution – must be inclusive, participatory, and accessible to all. This requires targeted work with specific at-risk groups, to understand their risks and to ensure that barriers to their participation are overcome. But reaching the people can be hard. They are marginalized and overlooked by others in society, or they must maintain a low profile for security purposes. Reaching them should be done in collaboration with civil society organizations and community associations that are experienced in working with them and meeting their needs safely.

In the outcome mapping framework, boundary partners lie in a program’s “sphere of influence”. That is, they inhabit a space outside the direct control of the program teams but inside the area of indirect influence. For example, military personnel might be influenced by training activities but they cannot be controlled by program teams. But they also lie inside the “sphere of interest”, i.e., the area in which the program seeks to bring about change. The senior military personnel, for example, are part of the military structure whose behavior is of interest to the program in question.

This idea is commonly represented in visual form as below:

Diagram 6: Boundary Partners

Here, the project stakeholders are represented by dots, which lie close to the project itself (e.g. direct participants of training programs) or further away in the broader population (e.g. members of the community group who do not engage directly with the program but whose behavior is of importance to the program vision. The boundary partners are marked using red dots.

Once the program team has identified a collection of suitable boundary partners, it’s important to consult with them on the program’s proposed outcomes and activities. This is, again, ideally done before the design of any individual project proposal, following the development of the strategic vision of the organization outlined in Step 1.

3.5.5. Progress Markers

Progress markers are indicators of community-based change in behavior, attitudes, beliefs and norms, which mark the steps along the path to the broad-based change identified in the program vision. When using outcome mapping to track change in the community, its vital to have a good selection of progress markers to track. It is best to select them in consultation with the program boundary partners, and always before the program begins implementation.

To identify progress markers for a project or program, start from the immediate changes you would expect to see after community members engage in the project activities. For example, an awareness-raising activity might include pre and post-tests for participants to track the change in their awareness of GBV risk factors in their community. The first progress marker towards positive change here could be an improved score on the post-test compared to the pre-test result. This is, so to speak, something you should “expect to see” if the program is operating as planned. You can then map out further changes in the community that go beyond this base-level change, steadily moving towards the overarching change the program seeks to achieve. For example:

There are at least three critical aspects to a good collection of progress markers:

• They should all focus on changes within the community and community members themselves. It can be tempting to think in terms of project activities when thinking about the minimum standards that you “expect to see”. For example, a team might suggest “high participation levels in awareness raising events”. But this is a measure of the program output, i.e. the number of people participating in the awareness-raising activities. It is not an outcome measure. Instead, focus on what changes you expect to see in the life of the community and its members. Increased awareness of participants, for example, or commitments they make.
• They should include a number of possible steps towards change. It is impossible to know exactly how change will occur before it does. So include as many possible steps along the path as possible. When collecting data against these indicators, it is important to be open to the idea that the “love to see” changes might still happen even if the “like to see” ones don’t. Change can happen in different ways.
• They final “love to see” changes should be ambitious. The final progress markers should end with the most profound social transformation you can realistically achieve in the timeframe you are working with. These are the changes that the program was designed to achieve, and the reason for which the program was undertaken in the first place.