Implementing Impact Evaluations of Malaria Control Interventions: Process, Lessons Learned, and Recommendations

Experience and lessons learned.

A broad range of stakeholders were invited to participate in the evaluations, but it was critical to have the MoH/NMCP involved, and where possible to lead the evaluations. Malaria impact evaluations were included in some NMCP’s Monitoring and Evaluation (NMCP’s M and E) plans. From the experience in conducting the evaluations to date, it was crucial for the MoH/NMCP to appoint a representative from the NMCP to serve as a point of contact for the evaluation, especially when the NMCP was not leading the evaluation. For example, in Malawi and Liberia the NMCP’s M and E focal points were involved in all discussions and decisions related to the evaluations. Including stakeholders in addition to the NMCP and funding agencies was particularly valuable when stakeholders were aware of studies done in country and could provide access to the reports and the data or could provide information on contextual factors (e.g., other health interventions) during the evaluation period.

Experience from the initial evaluations showed that stakeholders could be divided into roughly three groups based on when they were involved and their level of engagement in the evaluations: steering committee, broad stakeholders group, and core evaluation team (Figure 1 and Table 1). First, was the steering committee, although this was not always formalized. A steering committee was responsible for advising on directions for the evaluations in Senegal and Mali and also participated in the development of the request for proposals for the consultants for the Senegal evaluation. In countries without a formal steering committee, the NMCP and partners also filled these roles and were responsible for submitting an IRB proposal. In countries where there was a steering committee, it approved the evaluation protocol before submitting to the IRB. The steering committee was also consulted toward the end of the evaluation for review of the final evaluation report. Second, was the broad group of stakeholders who met at the outset of most of the evaluations to help brainstorm ideas around the evaluation, including identifying published and unpublished research studies, case studies, and also identifying pertinent data. It was important to engage the broad group of stakeholders early in the initiation phase. This was conducted at slightly different times in Liberia and Mozambique compared with Rwanda, but in all cases served to initiate the process and generate buy-in. In Liberia and Mozambique, stakeholders met at the outset of the process before any data analysis began and the meeting was used to generate ideas on other data sources (besides household surveys) to include in the analysis. The initial stakeholder meeting was conducted in Rwanda after the survey data had been analyzed. A workshop was held with the Rwandan NMCP, funding agencies, and external and local technical partners to review the preliminary findings, to identify additional data needs, and to discuss coordination of the report writing efforts. In most countries, the broad stakeholders group met again toward the end of the evaluations to review the results and assist with interpreting the findings, which fed into the final report.

In addition to the broad stakeholders group, a very successful approach was to form technical working groups with subsets of the stakeholders who advised on specific aspects of the evaluation. In mainland Tanzania, the local implementing partner convened a number of small technical working groups to review evidence available on topics including malaria in pregnancy, case management, vector control, and facility-based health data. Similarly in Zanzibar, expert working groups were convened to provide topic-specific guidance for the evaluation. Other countries did not constitute these technical working groups but instead reached out to individual experts in the field to provide the necessary data or contextual perspective.

The final group of stakeholders was the core evaluation team, which was involved in the day-to-day conduct of the evaluation. The composition of the core evaluation team varied by country, but included the NMCP point of contact, and representatives of funding agencies, local and external technical implementing partners, and in some countries, additional in-country partners.

In all countries, one major challenge was keeping the stakeholders engaged. This was addressed by having regularly scheduled phone calls and follow-up e-mail communications during the evaluation to discuss progress, process, and methodological issues, and to review results. The core evaluation team was involved in these routine activities and all stakeholders were continuously provided updates through e-mail communications. However, despite these efforts, it was difficult to retain the same level of engagement of the broad group of stakeholders, and in some countries of the core evaluation team, throughout the evaluation process.

Recommendations.

• Establish a steering committee for the evaluation to guide the evaluation process, approve the evaluation protocol and assist in identification of a local implementing partner. If no formal steering committee is formed, the NMCP and key in-country partners will need to initiate the evaluation and if needed submit the evaluation protocol for IRB approval. The evaluation team should use any existing in-country arrangements to assist with these activities.

• The NMCP should lead the evaluation if possible, but where this is not possible the NMCP should appoint at least one staff member to serve as a point of contact for the evaluation who will work as part of the core evaluation team. Malaria impact evaluations should be included in NMCPs’ M and E plans.

• Engage a broad group of stakeholders at the initiation, results review and dissemination stages of the evaluation. Use (or establish if necessary) technical working groups to provide advice on specific topics during the evaluation.

• The core evaluation team should periodically update the MoH/NMCP, funding agencies, and other key stakeholders.

• Engage focused technical stakeholders and partners who can provide information and context for the implementation of the malaria control efforts in country and are familiar with nonmalaria health programs in country.

Experience and lessons learned.

It was valuable having multiple partners work together on the impact evaluations; and where it was possible to coordinate among the partners, there was a reduction in the number of similar evaluations in country. In Ethiopia, PMI and the Global Fund to Fight AIDS, Tuberculosis and Malaria (Global Fund), in agreement with the government of Ethiopia, conducted a joint evaluation in 2013. The coordination was established by including members of both agencies in the planning, implementation, and review of the evaluation. PMI funded and organized the evaluation in Ethiopia looking at impact through 2013 and preparations are underway now for Global Fund to fund the next impact evaluation in Ethiopia in 2017. The Global Fund Technical Evaluation Reference Group emphasizes the importance of partner approaches to evaluation, recommends and supports joint evaluations in countries, and synthesizes evidence from those joint evaluations for global level strategic reviews.⁵ These country evaluations can be, or can complement, national program reviews with a robust assessment of epidemiology and program gaps and form the basis for preparation of funding concept notes to the Global Fund, and reports on progress of grant implementation.⁵

There were some instances where multiple evaluations were unavoidable due to different requirements or methodologies. For example, in Malawi and Uganda, PMI funded impact evaluations^6,7 and the UK’s Department for International Development funded epidemiological profile reports^8,9 at the same time. These reports differed in approach and content, yet there were complementary elements. Some of the data from the epidemiological profile reports were used as an input to the PMI-funded impact evaluations, but there was no overall coordination. Learning from the experiences in Malawi and Uganda, when the evaluation was designed in Kenya, efforts were made to coordinate the two evaluations by including some members on both evaluation teams and building off each other’s evaluations. Another example of concurrent evaluations was in Uganda where PMI and the Institute of Health Metrics and Evaluation (IHME)¹⁰ (funded by the Bill and Melinda Gates Foundation) conducted evaluations. The PMI-funded evaluation followed the plausibility argument approach proposed by the RBM MERG and IHME selected an approach to address causal inference. The two evaluation groups were in communication and reviewed and referenced the others’ reports.

In several countries, the RBM Executive Director sent letters to the MoH/NMCPs to introduce the impact evaluations and request their participation. This served to indicate to the countries that the broader RBM Partnership was supportive of these evaluations and it was not the effort of only one funding agency. Requirements of the NMCPs were included in the planning of the evaluations, including agreements on publications resulting from the evaluations and sections of the report the NMCP wanted to write. The methodology was also discussed and agreed upon with the NMCPs.

Recommendations.

• Partners should support countries in their efforts to evaluate the impact of their malaria control programs and should collaborate to conduct one coordinated evaluation, whenever possible. NMCP requirements also need to be included.

• Partners should discuss planned evaluations in an attempt to coordinate the evaluations or conduct a single joint evaluation. Ideally, there would be a forum where partners could present plans for these evaluations and several venues are possible. For example, globally through the RBM MERG or through donor meetings or malaria coordinating committees in country.

• Where multiple evaluations are deemed necessary, efforts should be made to coordinate review of the results and dissemination of findings to avoid confusion and conflicting messages in country.

Experience and lessons learned.

A common method^1–3 can be applied to evaluate the impact of malaria control interventions in high burden countries with adaptations for the specific country context. Within a given evaluation framework, it was important to consider the availability of high-quality data sources, epidemiological context, and level and duration of intervention implementation when choosing an appropriate design. Alternative evaluation approaches have been proposed.^2,3 The standard methodology consisted of building a plausibility argument through assessing trends of malaria intervention coverage, trends of malaria morbidity and under-five mortality, and trends of key contextual factors potentially affecting child survival. However for countries where data allowed, additional advanced analyses including cox-proportional hazard regression were done to quantify the effect of malaria interventions (e.g., insecticide-treated net ownership) on child survival.^2,3,11

Information about the malaria control program was gleaned from the National Malaria Strategic Plans, the country’s Annual Statistical Reports, Malaria Program Reviews (MPRs) and partner reports (UNICEF, PMI, etc.). MPRs provided information on the program activities, implementation progress against national strategic plan targets, and the potential for impact.¹² MPR reports were used in conjunction with analytic results from impact evaluations to provide additional contextual information on how well the program was implemented, which assisted in interpreting the evaluation results. The impact evaluations can be complementary to the MPRs.

Using lessons learned from the initial evaluations in Tanzania, Malawi, and Angola, a process for determining if it was appropriate to conduct a national-level impact evaluation with all-cause childhood mortality (ACCM) as the key impact indicator was developed (Figure 2). It should be noted there are no definite thresholds for what level of malaria control intervention coverage is necessary to expect national-level impact, nor for how long this coverage needs to be sustained until impact could be expected on malaria morbidity and mortality, although several studies have modeled this.^2,3,13 However, by using ACCM as the key impact indicator (5-year estimate), interventions at least need to be implemented and coverage maintained in advance of the midpoint of this 5-year period, which has proven to be a challenge in these initial evaluations. An increase in intervention coverage only in the last year of the 5-year period covered by the ACCM estimate is likely to have an impact that year, but this may not translate into impact seen within a 5-year mortality estimate. It is also important to note that impact could still be achieved subnationally if intervention coverage is uneven, but the analyses will likely need to be stratified by geographic location to address this. On the other hand, if the intervention coverage is uneven, but targeted to the highest burden areas, this may impact national mortality estimates.

View Full Size

Malaria impact evaluation decision process. When deciding when it is appropriate to conduct a national-level impact evaluation several considerations need to be taken into account including epidemiological context, intervention coverage levels, time since intervention scale-up began, time intervention coverage levels have been sufficiently high, and time since last evaluation. Several alternatives to a full national-level impact evaluation are provided. This decision process is applicable for high burden countries using all cause childhood mortality as the primary impact measure. DHS, Demographic and Health Survey; IE, impact evaluation; IGME, UN Inter-agency Group for Child Mortality Estimation; IHME, Institute for Health Metrics and Evaluation; MICS, Multiple Indicator Cluster Survey.

Citation: The American Journal of Tropical Medicine and Hygiene 97, 3_Suppl; 10.4269/ajtmh.17-0064

• Download Figure
• Download figure as PowerPoint slide

The importance of evaluation timing was evident in the case of the Angola evaluation. In 2002, Angola emerged from a civil war in which much of its infrastructure, including health infrastructure, was destroyed. The NMCP in Angola had been scaling up its malaria control interventions since emerging from the war, but the evaluation revealed that intervention coverage was not at a level where national-level impact on ACCM could be expected.¹⁴ In response to the determination that the evaluation was premature, an attempt was made to focus on one of the provinces where many malaria control activities were focused. In retrospect, an evaluation of progress in intervention coverage may have been preferable to the full impact evaluation assessing impact on ACCM that was conducted. Technical challenges of a different nature were experienced in Ethiopia,¹⁵ where using ACCM as the primary impact measure may not have been the best approach given the heterogeneity in malaria burden, with some areas experiencing a high level of malaria transmission and other areas experiencing very little. A national evaluation using ACCM masked a lot of this difference as reliable estimates of ACCM restricted to just those areas with malaria risk were not available.

The experience from conducting the evaluation in mainland Tanzania emphasized the need for an agreed-upon analysis plan at the outset of the evaluation. This evaluation was the first one completed following the plausibility evaluation approach.^16,17 The final set of analyses, including indicator stratifications,^2,3 were determined through an iterative process throughout the evaluation implementation. General guidance documents generated at the outset of the process in Tanzania were updated with lessons learned from that evaluation and then used to inform subsequent evaluations. The core evaluation team sought advice from external experts in the fields of malaria and child health program evaluation through a technical advisory group. This group reviewed the mainland Tanzania impact evaluation report and guidance documents, and met with evaluation funders and implementers at a 3-day meeting to revise the methods for the subsequent evaluations. On the basis of these recommendations, a more detailed standardized evaluation design with a detailed analytical plan was developed, which was first implemented in Uganda and then further adapted for the subsequent evaluations.

A limitation of the malaria impact evaluations conducted to date has been that they were all retrospective. The evaluation team therefore used what data was available, instead of being able to plan data collection to best measure impact. As discussed below under the use of the evaluation results, it is recognized that one of the results of the evaluation is the identification of gaps in the available data. NMCPs and partners have been encouraged to strengthen their data systems as a result.

Recommendations.

• Review the availability and quality of data, malaria epidemiology, and timing and level of malaria control intervention coverage (e.g., what level of coverage has been reached and how long has coverage been at this level) prior to the design phase of an evaluation^2,3 to determine if a national-level impact evaluation is appropriate.

• A standard methodology^2,3 and operational framework was developed and can serve as a guide for future impact evaluations. It is recommended that evaluators implement this approach, reviewing and adapting for the country context as needed. Where data allow, additional advanced analyses can be conducted to quantify the effect of malaria control interventions on child survival.

• Although an analysis plan needs to be agreed upon at the outset to guide the evaluation, it is equally important to also have some degree of flexibility to adjust the analytical plan based on any new evidence that emerges during the course of the evaluation, including interim findings from the ongoing evaluation.

• The evaluation approach was designed for estimating impact in high malaria burden countries, using ACCM as the primary impact indicator. There is a need to develop an impact evaluation approach suitable to moderate and low transmission settings and discussions on such a methodology are underway.

• Prospective evaluations are ideal and provide the evaluators the opportunity to collect data prospectively thus overcoming the limitations of relying on data already available. Particularly for countries looking at conducting another malaria impact evaluation in the future, plans for improving (or adding) data sources should be developed following the completion of the first evaluation.

Experience and lessons learned.

In addition to large household survey data, other data sources added depth and completeness to the evaluations. Sources included data from anemia and parasitemia surveys, Health and Demographic Surveillance Systems (HDSS), and Health Management Information Systems (HMIS), including sentinel surveillance sites. In addition, information on socioeconomic conditions, nonmalaria health interventions and meteorological (rainfall and temperature) data were considered.^2,3 When requesting access to data it was important to reach an understanding with the party that holds the data, regarding who will have access to the raw data and its role in the evaluation. For most of the evaluations, complete data sets were provided, but in others the owner of the data performed the requested analyses for the evaluation team without sharing the data. For example, in Malawi, researchers at the National Center for Chronic Disease Prevention and Health Promotion at the Centers for Disease Control and Prevention provided and reanalyzed parasitemia results from the National Micronutrient Surveys.⁶ In some instances, a memorandum of understanding (MOU) was requested before access to data was granted. For example, a written request was submitted to the NMCP in Malawi for access to the 2010 MIS dataset. Ideally, MOUs regarding data access (specifically in reference to making the data publically available) should be developed at the time a survey or study is conducted.

Another solution to the data access issue was having an impact evaluation analyst work with the holders of the data. This approach was used in Zanzibar, where one of the consultants working on the evaluation went to Zanzibar and analyzed the HMIS data in conjunction with the Zanzibar Malaria Elimination Program. This had the added benefit of strengthening capacity for data analysis and evaluations within the malaria control program.

In some countries, the evaluation team was unable to gain access to the data. It was important to move forward with the evaluation using the main data sets, including household surveys and HMIS where possible, and not hold up the evaluation for the additional data. Where possible the evaluation referenced published reports when the data could not be analyzed directly.

Recommendations.

The evaluation team should:

• Begin working with the MoH/NMCP and the other stakeholders as early as possible to identify useful data sets, including malaria, other maternal and child health information, and meteorological data, and to request access to them.

• Establish a robust and transparent decision-making process for determining the quality of various data sets, with a clear decision point regarding whether to use the data in the evaluation. For national household survey data, review the methodology and pay particular attention to the sample size to make sure it is powered to measure key indicators in a sub-population or at sub-national level. Specific attention should be given to data from routine health information systems by setting a minimum cutoff point for data completeness. Regardless of the indicators or data elements, the level of completeness should be at least 85% to avoid random results which are difficult to explain over time.

• Identify alternative approaches when the raw data sets are not available or not of sufficient quality. Options include: request that select indicators be abstracted; request that the owners of the data sets perform the requested analysis for the impact evaluation; have a member of the evaluation team work with the NMCP or local organization to analyze the data in country; or provide data cleaning and analysis assistance to the owners of the data.

Experience and lessons learned.

A common evaluation methodology framework^2,3 was developed and adapted to each country’s context. Similarly, a common operational framework was developed as described in this paper and adapted to each country’s capacity. In all evaluations, technical expertise in analyzing the large household survey data sets was provided by an external technical partner, but the evaluation lead and partner responsible for writing the reports varied by country. In some countries the NMCP led the evaluation; however, in others the NMCP remained engaged but preferred that the process be led by another partner.

In each country in which PMI initiated the process, PMI negotiated with the NMCP (and in some countries with the steering committee) to determine the best fit to accomplish the evaluation in a reasonable period of time, while preserving transparency and affording opportunities for all partners to contribute.

Although it was critical to complete the evaluation in a timely manner it was also important to take advantage of opportunities to build and strengthen local capacity. Capacity building of the NMCP and/or local institutions needed to be agreed upon at the outset, with the understanding that some of the analyses would be completed externally. Providing capacity strengthening opportunities through working closely with external technical assistance partners incurred additional time and costs and these needed to be planned for at the outset. For example, in Liberia the international technical partner organized a multiple day workshop with the NMCP and local technical partner to explain the impact evaluation methodology and to work through interpreting the analyses of the Liberia data.

Conducting the evaluations as a team resulted in an increased exchange of ideas, improved transparency, and capacity strengthening of local institutions and NMCP staff, but also contributed to lengthening the evaluation timeline. Although not explicitly defined at the beginning of these evaluations, there were three general implementation approaches that emerged:

Recommendations.

• Identify national and international partners to complete a high quality evaluation in a reasonable period of time, based on in-country capacity and time available for the NMCP to work on the evaluation.

• Work with the NMCP (and steering committee) to identify the most appropriate partner to lead the evaluation, whether this is the NMCP, an in-country technical partner, or an external technical partner.

• Provide opportunities to improve technical capacities of the NMCP and/or local implementing partner. Budget time and funding for agreed upon capacity strengthening activities at the outset of the evaluation.

• Ensure in-country technical partners have strong ties with the MoH/NMCP and are able to work with the external technical partner.

Experience and lessons learned

Dissemination of the findings.

The presentation of impact evaluation results needs to be tailored to the audience. In most countries, there were multiple audiences that benefited from and used the results, necessitating the production of multiple documents. The evaluation teams found that lengthy, detailed impact evaluation reports were not ideal for advocacy nor for the malaria community. As a result, after the first set of evaluations the impact evaluation reports were streamlined, while still retaining the detailed methods and analyses in a set of annexes to the report. In addition to the impact evaluation reports themselves, in some countries the results from the impact evaluations were used to produce two additional types of documents: RBM Progress and Impact (P and I) Series Country Focus reports^18,19 and scientific journal articles.^{11,16,20–22} The RBM P and I reports, which were advocacy-focused documents, afforded an opportunity to present findings from the impact evaluation in a clear and concise format, to tailor the report to a wider audience (e.g., politicians, media, advocates, development partners), and to share lessons learned during the scale-up phase of malaria control interventions in a given country. Results from the evaluation in mainland Tanzania were also used as a country showcase for impact in the Global Fund results reports.²³

Dissemination events (e.g., high-level events where the report was released) were held in both mainland Tanzania and Malawi surrounding the release of the RBM P and I reports^18,19 and in Senegal around the release of the core report.

For dissemination of the results in country (when the dissemination was not part of an RBM P and I report release), the evaluation team developed a summary of the key findings. This took the form of an expanded executive summary and was used for the dissemination events in Senegal, Uganda, and Kenya. This format presented the results of the full impact evaluation in a more digestible way for a broad range of stakeholders.

Use of the findings.

National malaria control programs used the results to provide evidence of the impact of their malaria control program. Funding agencies (e.g., PMI and Global Fund) used the findings as part of their reporting requirements to demonstrate the contribution of their support to the impact of malaria control and also to justify sustained funding of the malaria programs. The results of the impact evaluations were used by countries to report against the Millennium Development Goals and, more recently, the Sustainable Development Goals (SDGs). Donor agencies and the malaria community have used the results of the impact evaluations to assess progress toward national and global malaria control targets. The impact evaluation results could also provide epidemiological data to feed into the MPRs.

National malaria control programs and partners also used these evaluations to identify gaps in their data collection systems. In some countries, it was not possible to analyze the routine data or the multiple routine data sources gave conflicting trends. For example, in one country aberrant malaria mortality findings from the routine data resulted in a team from the NMCP and MoH visiting health facilities to review the data to understand how malaria deaths were being captured in the routine information system. As part of the dissemination process, malaria control programs and their partners were encouraged to review the results and limitations of the evaluation and develop an action plan for building off the current malaria control efforts in country, including identifying gaps in malaria control interventions and gaps in data systems.

Recommendations.

• As appropriate, use multiple formats to present the impact evaluation findings (e.g., full report, key findings report, journal article, and/or policy brief) to meet the needs of the various stakeholders. A detailed scientific report can be produced and serve as the main reference report for the other documents released in association with the evaluation. Consider a key findings report for World Malaria Day events or to accompany the dissemination of the final report.

• Prepare journal articles and/or present at scientific meetings to inform the broader malaria community. Peer review and publication of the evaluation findings provides further validation of the impact evaluation results and creates buy-in.

• Impact evaluations can be used to report progress against targets of the SDGs and as a key input to MPRs and Global Fund applications.

• The evaluation process will likely uncover gaps and limitations in the data. These should be noted and discussed with partners during the evaluation, and at the end of the evaluation an action plan should be developed as part of the dissemination process to strengthen the data, especially routine data collection systems in country. This can inform prospective planning of future evaluations to better meet the needs of NMCPs and partners.

• The results of the evaluation should also be used by NMCPs and partners to strengthen their program in response to low levels of intervention coverage. If there is no demonstrated impact the NMCP and partners should investigate the root causes of this and address these to strengthen their programs.

Experience and lessons learned.

In the initiation phase, additional time was needed in some countries primarily due to the identification and hiring (procurement process) of a local technical partner. The execution phase was prolonged because of the time consumed by multiple partners in reviewing numerous drafts of the core report, competing priorities for the NMCP and partners, and inaccessibility of data. Finally, delays have occurred during the final stage of review and clearance of the evaluation report.

The experience with the first thirteen evaluations has shown that any longer than a year and the evaluation became less efficient and risked being less useful to stakeholders. As discussed above, it is difficult to maintain stakeholder engagement over a long period of time. PMI with its partners have undertaken efforts to streamline the process for the subsequent evaluations to keep them near a 12-month time frame (Table 2).

Recommendations.

• Concrete steps (Table 2) need to be taken to ensure the evaluations are completed within a year.

• It is recommended that a clear management/decision-making/communication strategy among partners be established at the outset. In addition, clear procedures for clearance and approval of the reports and associated documents need to be established up front.

• Limit the number of drafts of the report that key stakeholders (core evaluation team and additional identified stakeholders) are asked to review to the first complete draft and the final draft, with the core evaluation team reviewing specific sections in the interim.

Experience and lessons learned.

The evaluation in Senegal was conducted in French with the analysis plan, protocol, and all drafts of the report produced in French. Additional time was needed at the outset to translate the standard analysis plan and protocol to French. When it came time to finalize the report, there was a requirement from some partners to review the report in English. This required a significant amount of time to translate the technical document from French to English and then resulted in the need to make simultaneous edits to the French and English versions of the report. A similar situation arose in Angola, where the initial drafts of the background chapters were written in Portuguese. Given the need to review the report throughout the process in Angola, the core evaluation team switched all drafts to English part way through the evaluation.

Recommendations.

• Decide at the outset the language(s) the final report and associated documents will be produced in. One solution is to produce the key findings report in the official language and English, but maintain the full report in the official language.

• Translation services need to be budgeted for both in terms of the time needed for translation of documents and also the cost.

Experience and lessons learned.

Funding.

A detailed budget that includes costs for human resources, stakeholder meetings, payment for data sets, subcontracts, translation, and dissemination was prepared before the evaluations were implemented. The cost for an evaluation using in-country and external technical partners ranged from $230,000 to $405,000 (Table 3). This does not include salaries of the NMCP and funding agency staff involved in the evaluations. It was important for all partners to agree to the funding, either in-kind or through direct financing. This was particularly important as the impact evaluations were a large undertaking that involved multiple partners. As mentioned above, NMCPs contributed by providing staff time to serve as points of contact for the evaluation team and were involved throughout the evaluation.

Funding was necessary primarily for the analysts’ and writers’ time and this was broken into funding for in-country technical partners and external technical partners. A second cost was workshops and/or stakeholder meetings. In some cases, a payment was required for accessing data, but this was a minimal cost for evaluations based on secondary analysis of existing data sources. In some countries, funding was required for access to and analysis of the meteorological data. Planned dissemination events and printing the report or associated documents also needed to be budgeted for.

Human resources.

For analysts and writers, it was ideal if the evaluation team was comprised of individuals capable of analyzing large household survey data sets as well as other data sets such as those from HMIS and HDSS. In addition, there needed to be individuals on the evaluation team who were familiar with the malaria control program and could work with the NMCP to write sections on the history of malaria control in the country, policy changes, and implementation of the various malaria control interventions, and to provide context for interpreting the findings.

The core evaluation teams found that there needed to be a balance between having a large set of stakeholders involved in the evaluation and getting the evaluation done within a year. It was efficient to hire external and in-country technical partners who could devote the necessary time to the evaluations, while still consulting with the NMCP and funding agencies throughout the process.

Recommendations.

• A detailed budget that includes costs for human resources, stakeholder meetings, payment for data sets, subcontracts, translation, and dissemination needs to be prepared before the evaluation is implemented. This provides the basis for stakeholders to define their contributions, be it in-kind or direct financing of the evaluation.

Experience and lessons learned.

The evaluations described here were led by the NMCPs, funding agencies and partners, all of whom stood to gain or lose based on the results of the evaluation. The evaluation teams agreed to present the results of the evaluations whether the outcomes demonstrated impact or not; however, there still remained a potential conflict of interest, particularly because of the level of interpretation of the results that was required.

Recommendations.

• Encourage the evaluation team to release the evaluation findings whether they demonstrate impact or not.

• Maximize the independence of the evaluators as resources allow. Consider hiring an independent evaluator for malaria impact evaluations especially when the results are open to interpretation.