Evaluation has to change

01 May 2010

"There is a demand for UNDP’s role as a traditional development partner to change in Turkey" said UNDP’s Evaluation Office Director Saraswathi Menon.

New Horizons - Menon who visited Turkey in the scope of a stakeholder workshop to discuss UNDP's work in the country in mid-April, drew attention to the fact that evaluation methods have to change to cater to individual country needs. New Horizons interviewed Menon after the meeting.

What is the purpose of your visit to Turkey? What did you do? How did you do it? What was the process? What was the result? What would be the outcomes of this evaluation?

The Evaluation Office has completed an evaluation of UNDP’s work in Turkey over the last 5 years. This evaluation is more or less finalized, it’s in draft form and we held a stakeholder meeting on the draft report with our partners. Among stakeholders we count everyone, who is engaged with UNDP here, the Country Office and the primary partner the government and the State Planning Organization both on the implementation side and the strategic side. Among the stakeholders are other UN agencies, other donors but also its very important that we listen to civil society, the academia and so on.We wanted to get their comments on the report, especially on the recommendations and see how we could make the report even more stronger and useful in Turkey.

Did you find the comments from the participants useful?

Extremely useful. What struck me especially was the diverstiy of comments. There were many implementing partners of UNDP Turkey who focused on their particular experience in formulating or managing a project. And others were looking at the UNDP engagement more generally, like the SPO. Issues for Turkey but could be addressed by working with UNDP such as the whole question of concepts in development. There are different perceptions of what gender means, what capacity development means and so on. They felt that the way UNDP goes about solving some of its problems would help them solve their problems in Turkey. For instance on capacity development UNDP could organize a meeting on what that means and people in Turkey could what that means to them. If there is a better agreement on these concepts, then there would be a better agreement on the next step, on actions to follow. Above all what I saw was that UNDP is in a very interesting position in Turkey. We have been a traditional development partner for a long time but now more and more the demand is for UNDP to play a different role because Turkey is changing. That came through in many of the comments and I think the report will really have to think that point through very carefully before we finalize it.

Does this relate to some of the new trends in M&E?

Not so much in Monitoring and Evaluation but in terms of development, how the world is changing. But I would also say that evaluation has to change. Even the criteria we use today of effectiveness, efficiency, and so on. These were all developed around the evaluation of development projects. More and more, we are not evaluating projects, we are evaluating Joint Programmes, we are evaluating sectoral strategies, non-project work (a lot of UNDP work is advocacy, promoting UN values, conventions and norms) so you are evaluating at a different level from the project and evaluation will have to change in terms of the level. Evaluation will also have to change because we have traditionally done evaluation from the outside, looking at what an organization brings to a country and what contribution that makes. But more and more as countries are taking greater leadership over their development, when they have the capacity to define what they do in aid and development cooperation you start to have a smaller catalytic part. Then you have to look at evaluation from the perspective of the country. It must be what the country wants to monitor, what the country wants to evaluate and what we do has to fit into that. That is a challenge for us because we are not used to it.

How do you plan on overcoming this?

You can only concretely address this not theoretically but you actually have to work together. And I hope Turkey can be an example of that because the UNDP Turkey and government partnership is different. It’s almost an opportunity that monitoring and evaluation has been weak according to the report’s findings and has to be addressed. Now they can address it differently. Rather than inventing their own systems of monitoring and evaluation for the office, maybe they should be doing is to work with the government to see where government needs support needs to be strengthened, they can learn from other countries with strong support systems, and see how UNDP will plug into that. National programmes, it doesn’t make sense for UNDP to monitor a project if it is not part of the national effort.

Can you comment on the dual position of UNDP? Does this concern Turkey’s recent position as a higher MIC country or does it have to do with the South South Cooperation project?

A little bit of both. Many countries are engaged in the South South Cooperation. In Turkey, this has been formalized. You have TIKA, an institutional arrangement promoting this kind of cooperation. UNDP has to come to terms with not just supporting an institution but supporting South South Cooperation in Turkey that promotes the development principles of the UN. After all, we are not just looking at the replication of the old and unequal aid relationships. South South Cooperation is among equals, among countries who share development challenges. So how do we help Turkey to be a good equal partner of other developing countries. That’s the kind of experience UNDP can create.

What do you expect from the stakeholder meeting? What do you think will improve?

Two things. Most immediately, from our point of view, we will have a better report. Because we had so many good stakeholders in the room, we hope that hey will read the report and use it .For UNDP at two levels. Because you have a new RR and you have the bureau director. I hope that UNDP will benefit from this, in terms of a new and stronger programme. We heard that the RR has a new vision and the report will help him carr that out. Because the regional director was here, we hope that the experiences in Turkey will be shared much more broadly and that the bureau will be able to support Turkey better and that they will use these of a new kind of partnership, etc.in other countries in the region and beyond.

Do you have any further recommendations for the CO aside from the ones in the report?

We will have more recommendations because the draft recommendations will be deepend, elaborated, and maybe some prioritizing will be necessary.

When will the report be finalized?

The report will be finalized over the next month, and it will be printed and made available to the executive board of UNDP by the end of May, beginning of June.

Are all ADRs distributed throught the UNDP?

Yes. We send a copy to every country office, so they can learn from other country examples.

For further information evaluation and monitoring at UNDP as well as online copies of ADR reports, please visit www.undp.org/eo.


Backround information on evaluation

What is monitoring and evaluation? Are they separate entities? What kinds of questions do they answer?

Evaluation is judgment made of the relevance, appropriateness, effectiveness, efficiency, impact and sustainability of development efforts, based on agreed criteria and benchmarks among key partners and stakeholders. It involves a rigorous, systematic and objective process in the design, analysis and interpretation of information to answer specific questions. It provides assessments of what works and why, highlights intended and unintended results, and provides strategic lessons to guide decision-makers and inform stakeholders.

Monitoring is distinct from evaluation. It is a continuous function providing managers and key stakeholders with regular feedback on the consistency or discrepancy between planned and actual activities and programme performance and on the internal and external factors affecting results. Monitoring provides an early indication of the likelihood that expected results will be attained. It provides an opportunity to validate the programme theory and logic and to make necessary changes in programme activities and approaches. Information from systematic monitoring serves as a critical input to evaluation.

What is the role of evaluation at UNDP?

Evaluation in UNDP provides an objective assessment of contributions to development results, through assessing its programmes and operations, including advocacy, advisory services, knowledge networks, technical assistance, coordination and partnerships. Evaluation addresses what works and why, as well as what does not work and unintended outcomes. This supports accountability, informs decision-making and allows UNDP to better manage for development results.

Evaluation also improves learning and knowledge for development among UNDP and its partners. Engagement of all key stakeholders enhances capacity for evaluation as well as its utility. The development of knowledge-management systems, learning groups and communities of practice increases access to knowledge and enhances knowledge-sharing, collaboration and innovation.

Evaluation is critical for UNDP to progress towards advancing human development. Through the generation of ‘evidence’ and objective information, evaluations enable managers to make informed decisions and plan strategically. UNDP success depends, in part, on the ability of UNDP and its counterparts to carry out credible evaluations and use them to make evidenced-based decisions. The effective conduct and use of evaluation requires adequate human and financial resources, sound understanding of evaluation and most importantly, a culture of results-orientation, learning, inquiry and evidence-based decision making. Everyone in UNDP and its stakeholders have to share the same vision and be open to change. When evaluations are used effectively, they support programme improvements, knowledge generation and accountability.

How does evaluation work? What is the process? Are their certain benchmarks or criteria to adhere to? What are they?

The evaluation process should involve key government counterparts, donors, civil society and UN organizations, as well as beneficiaries of initiatives and ‘informants’, who may not necessarily have a direct stake in the subject of an evaluation. Such broad-based involvement of national stakeholders will enhance not only the ownership of and mutual accountability for results, but also the credibility and transparency of the evaluation exercise.

UNDP emphasizes the centrality of national ownership in evaluating results. The achievement of the outcome is dependent upon contributions from a range of partners, including UNDP.

The evaluations conducted by UNDP fall into two categories: independent evaluations conducted by the Evaluation Office, and decentralized evaluations managed by country offices, regional bureaux and practice and policy bureaux, and conducted by external experts. Together they represent a coherent system of evaluation that provides the necessary coverage to manage for results and to support organizational accountability. The Evaluation Office is mandated to conduct evaluations for corporate accountability, strategic planning, and the development of information for global knowledge use.

Country offices, regional bureaux, and practice and policy bureaux commission decentralized evaluations in the programmatic frameworks for which they are responsible. Outcome evaluations address the short-term, medium-term and long-term results of a programme or cluster of related UNDP projects. They include an assessment of the effectiveness, efficiency, sustainability and relevance of the programme against their own objectives, their combined contribution, and the contribution of external factors and actors. Project evaluations assess the efficiency and effectiveness of a project in achieving its intended results. They also assess the relevance and sustainability of outputs as contributions to medium-term and longer-term outcomes. Projects can be evaluated during the time of implementation, at the end of implementation (terminal evaluation), or after a period of time after the project has ended (ex-post evaluation).

Why is evaluation necessary? Who foresees M&E activities?

The evaluation function works to enhance the development effectiveness of UNDP to help men and women build a better life. It strengthens accountability and learning through evaluation and partnership.

The Evaluation Office is the custodian of the evaluation function. The office prepares and periodically reviews and updates UNDP’s policy for evaluation; and reports annually to the Executive Board on the function, findings and recommendations of evaluations, on compliance, quality assurance, and follow-up to evaluations conducted by UNDP and its associated funds and programmes. The office also maintains a system to record management responses to all evaluations and alerts senior management to emerging evaluation-related issues of corporate significance.

The Evaluation Office is responsible for developing an agenda for independent evaluations, based on consultations with the Executive Board, senior management, the associated funds and programmes and other stakeholders. In response to emerging issues that the Evaluation Office may identify; strategic and thematic evaluations, programme evaluations such as the Assessment of Development Results (ADRs) at the country level, evaluations of global, regional, and South-South programmes, and other evaluations as required are conducted. The office ensures that the evaluations provide strategic and representative coverage of UNDP programmes and results, and that mandatory evaluations are carried out; sets evaluation standards for planning, conducting and using evaluations, developing and disseminating methodology and establishing the institutional mechanisms for applying the standards. Supporting the setting of frameworks and standards for monitoring in the context of the UNDP results-based management system to facilitate the evaluation of programmes and activities, the office assures mandatory decentralized evaluations and supports quality assurance of the evaluations conducted by the associated funds and programmes.

In carrying out these tasks, the Evaluation Office has to ensure that evaluation in UNDP contributes to and remains consistent with UN policy and reforms including supporting and participating in joint evaluations.