Review of M4P Evaluation Methods and Approaches, ITAD, April 2013

The reviewers analysed 32 M4P programme reviews and evaluations, monitoring and eReview of M4P Evaluation Methods and Approaches, ITAD, April 2013: valuation (M&E) frameworks and programme reports to help guide the design and implementation of future M4P evaluations. Generally, M4P evaluations are found to be weak in terms of considering systemic changes, triangulation practices and use of theories of change, among other things.

Description
This report presents the findings of a study undertaken by ITAD for DFID to review the methods used to evaluate M4P programmes. It is intended to help guide the design and implementation of future evaluations.

M4P programmes are defined as playing a facilitative, adaptive role – in order to contribute to systemic, large scale and sustainable market changes that positively affect the poor. The nature of the approach, and the complexity of the markets within which it operates, present a number of challenges for evaluation. Evaluation approaches must address these challenges, if they are to avoid inaccurate estimations of impact.

Methods for info gathering
The reviewers analysed 32 M4P programme reviews and evaluations, monitoring and evaluation (M&E) frameworks and programme reports, including 14 evaluation reports, and consulted with 14 individuals and organisations in the field.

Summary of results
The M4P evaluations reviewed were generally found to be weak, in terms of:

– consideration of systemic, sustainable changes in market systems;
– data quality (small sample sizes with little consideration of sampling frames, statistical significance or bias);
– triangulation practices (particularly with regard to qualitative data collection);
– the use of theories of change (those used were often linear, not externally vetted, with assumptions not adequately tested);
– consistency in units for facilitating accurate aggregation; and
– consideration of unintended negative effects.

Advice from the DCED Standard for Results Measurement is included, for example on measuring systemic change.

The report concludes that, while evaluations must be objective, they also need to be based on in-depth understanding of the interventions. As an example of this combination, it cites the DCED Standard – which combines internal monitoring with an external audit of the monitoring system.

Review of M4P Evaluation Methods and Approaches, ITAD, April 2013
Source of information: The Donor Committee for Enterprise Development.