JICA\'s CBA guide

JICA Guideline for Project Evaluation ~ Practical Methods for Project Evaluation ~ September 2004 Office of Evaluation, Planning and Coordination Department Japan International Cooperation Agency (JICA) i TABLE OF CONTENTS Part +: JICA’s Project Evaluation Chapter 1 Outline of JICA’s Project Evaluation 1. Objectives of JICA’s Project Evaluation 2. Types of JICA’s Project Evaluation 3. JICA’s Evaluation System 4. Evaluation Feedback 5. Criteria for Good Evaluation Chapter 2 Frameworks and Basic Steps of JICA’s Project Evaluation 1. Evaluation as a Management Tool 2. Framework of Evaluation 3. Major Steps of JICA’s Project Evaluation 1 2 3 5 9 11 13 15 16 17 23 Part II: JICA’s Evaluation Methods Chapter 1 Confirming Evaluation Purposes and Organizing Information on Target Project 1. Confirmation of Evaluation Purpose 2. Grasping the Whole Picture of a Target Project 3. Utilization of the Logical Framework 4. Grasping the Implementation Situation: Information on Performance and Implementation Process Chapter 2 Planning Project Evaluation 1. Developing Evaluation Questions 2. Basis for Judgment 3. Considering Necessary Data and Information Sources 4. Data Collection Methods 5. Formulating an Evaluation Grid Chapter 3 Interpretating Data and Reporting Evaluation Results 1. Data Interpretation 2. Making Recommendations and Proposing Lessons Learned 3. Reporting Evaluation Results 26 27 28 30 33 46 48 51 61 66 72 82 84 85 87 89 ii Part III: Management of Project Evaluations Chapter 1 Issues in Managing Evaluations 1. The Role of JICA Project Operational Departments 2. Preparation of the Advertisement of a Contract 3. Preliminary Preparation for the Evaluation 4. Managing On-site Evaluations 5. Preparation of the Evaluation Report 6. Feedback of Evaluation Results Chapter 2 Issues in Ex-ante and Ex-post Evaluation 1. Key Issues of Ex-ante Evaluation Studies 2. Issues of Monitoring and Mid-term Evaluation Studies 3. Key Issues of Terminal Evaluation Studies 4. Key Issues of Ex-post Evaluations 95 96 97 100 107 109 113 114 115 118 139 152 193 Frequently Asked Questions regarding JICA’s Project Evaluation Attached Materials 1. What is the Logical Framework? 2. What is Participatory Evaluation? 3. What is Performance measurement? 4. Bibliography 198 220 221 230 232 235 NOTE: This guideline is a translation of major parts of the Japanese original version; ‘ JICA Jigyo Hyoka Guideline ~ Project Hyoka no Jissenteki Shuho~’, Office of Evaluation and Post Project Monitoring, Planning and Evaluation Department, JICA, March 2004. Part I JICA’s Project Evaluation 2 Chapter 1 Outline of JICA’s Project Evaluation In order to provide an overview of JICA’s Project Evaluation, this chapter explains its objectives, types, implementing system and feedback system. It also explains the “criteria for good evaluation” for improving the evaluation quality. It suggests the goal and direction of JICA’s Project Evaluation. Tips! - There are three objectives for the use of JICA’s Project Evaluation: 1) tool for the project cycle management; 2) tool for enhancing the “learning effects” for more effective project management; 3) ensuring accountability. - Types of evaluation are mainly classified into two levels: program-level and project-level. - Project-level evaluation is classified into four types conducted at different stages during the project cycle: ex-ante evaluation, mid-term evaluation, terminal evaluation, and ex-post evaluation. Program-level evaluation is comprehensive evaluation which mainly applies to ex-post evaluation, and has several types according to its target and evaluator. - JICA’s Evaluation system consists of 1) Evaluation Study Committee, 2) Advisory Committee on Evaluation, 3) Office of Evaluation and Post-Project Monitoring, and 4) Departments and Overseas Offices Involved in Project Implementation. - JICA regards the function of evaluation feedback as twofold: 1) feedback for project management and operation and 2) feedback toward public. JICA makes an effort to implement strategies for effective feedback. - Criteria for good evaluation are 1) usefulness, 2) fairness and neutrality, 3) credibility, 4) participation of partner countries and so forth. Becoming fully responsible for those criteria leads to more qualified evaluation. 3 1. Objectives of JICA’s Project Evaluation It is important to evaluate the outcomes that a project achieves and to feedback the evaluation results, lessons, and recommendations obtained for a more effective and efficient implementation of development assistance. The harsh economic and fiscal situations at home have generated strong calls in Japan for more effective and efficient implementation and ensuring accountability for ODA. The enhancement of evaluation has drawn attention as one of a major improvement measures. In addition, there are changes in the political landscape such as the adoption of public sector evaluation by ministries and the reorganization of agencies into Independent Administrative Institutions (IAI) that ask for improvements of the evaluation system. JICA’s evaluation is a tool for judging as objectively as possible the relevance and effectiveness of JICA’s cooperation activities at four different stages during the project cycle: ex-ante, mid-term, terminal, and ex-post. The primary objective of evaluation is to improve the effectiveness and efficiency of projects by using evaluation results for better planning and implementation. JICA also intends to gain public support and understanding by using them to ensure accountability. JICA has been focusing its effort to bolster its evaluation with the following three objectives. (1) Using Evaluation Feedback as a Means for Project Operation and Management By using them in the decision-making process, JICA refers to evaluation results when formulating its aid strategies and JICA Country Programs. It also uses them when making decisions regarding project execution, selecting target projects, reviewing plans, and determining the continuation or termination of a project. (2) Enhancing the “Learning Effects” of the Personnel and Organizations Concerned for More Effective Project Implementation Evaluation feedback enhances how effectively the various people involved can learn and develop their skills. The term “Learning Effects” refers to how successfully the process of learning from evaluations enables JICA staff and stakeholders to better implement their projects and programs. For instance, the lessons from past projects serve as useful references for jica staff and officials of partner countries when they plan and implement similar projects. Also, the evaluation process itself contributes to expanding the knowledge and developing the capacities of the people involved, and thus serves as a “learning process”. 4 (3) Disclosing Information Widely to Secure JICA’s Accountability Disclosing evaluation results to the public and explaining that JICA is fulfilling its responsibility for its undertakings is indispensable for winning public support and understanding. In order to ensure accountability to taxpayers, JICA needs to ensure adequate information disclosure. Figure 1-1-1� Utilization of JICA’s Evaluation Results� � � � Using Evaluation Feedback as a Means for Project Operation Enhancing the Learning Effect of the Personnel and Organizations concerned Securing JICA’s Accountability Ensuring Accountability to Taxpayers / Implementing Projects more Effectively and Efficiently Conducting Evaluation 5 2㧚Types of JICA’s Project Evaluation � The Ministry of Foreign Affairs’ “Report on Improvement of ODA Evaluation System,” released in March 2000, classified ODA evaluation into three levels: policy-level, program-level, and project-level as shown in Figure 1-1-2. The report called for enhancement of policy- and program-level evaluation. The Ministry of Foreign Affairs conducts policy level evaluation such as Country Assistance Program and Sector-specific Initiatives, as well as program-level evaluation by sectors and schemes. JICA conducts program- and project-level evaluation. How to evaluate JICA’s management cycle is illustrated in Figure 1-1-3. There are two cycles: the “small cycle” (project cycle) and the “large cycle” (program cycle). Figure 1-1-2 ODA System and JICA Evaluation Country A’s Development Issue JICA’s basic policies ࡮� Mid-term Plan ࡮� Study committee reports ࡮� Jica Thematic Guidelines etc࡮Country Assistance Program ࡮Sector-specific Initiatives Medium-term Policy on ODA ODA Charter Country A’s National Development Plan JICA Country Program (for Country A) Development Issue Development Issue Country A’s Development Issue Country A’s Development Program Cooperation by other donors Country A ’s independent projects Country A ’s independent projects Cooperation by other donors JICA Projects JICA Projects JICA Projects JICA Projects Project Level Program Level Policy Level Evaluation System of entire O DA JICA Cooperation Program JICA Cooperation Program 6 Figure 1-1-3 Evaluation Types by Stages during the Project Cycle (1) Evaluation Types by Project/Program level Project-level evaluation covers individual projects. This type of evaluation, conducted both by JICA’s departments responsible for project implementation and by overseas offices, is intended to be reflected in planning and reviewing individual projects, in making decisions as to the continuation of projects and the revision of project plans, in planning and executing other similar projects, and in ensuring the accountability of operations. Program-level evaluation includes comprehensive evaluation applied to such groups of projects as those that share the same overall goal and development issues. It is also directed at a set of projects under a specific cooperation scheme. These evaluations are principally conducted by the Office of Evaluation in JICA at the ex-post stages as country-program evaluation or thematic evaluation. The evaluation results are used mainly for improving JICA’s Country Program and for finding and formulating new projects. Program-level evaluation is classified into two types as follows: i) Country-program Evaluation This comprehensive evaluation examines the overall effects of JICA’s cooperation on the The Japanese Public ޝPlanޞ Development or Revision of JICA Country Program Issue-specific request suevey Ex-ante Evaluation Mid-term Evaluation Terminal Evaluation ᅆPlanᅇ Ex-post Evaluation Project Cycle � � Feedback � � Feedback Feedback � � Feedback Program Cycle Ex-post Evaluation Thematic, Country -Program ޝImplementationޞᅆPost Imple- mentationᅇ 7 development of a targeted country across projects. After clarifying and analyzing the overall effects of JICA’s cooperation and the difficulties it faced, this evaluation derives lessons and recommendations for the improvement of future JICA Country Program of the country in question. ii) Thematic Evaluation This evaluation looks at a number of projects by focusing on specific sectors, issues (environment, poverty, gender, peace-building, etc.) or cooperation schemes (Japan Overseas Cooperation Volunteer Program, etc.). After clarifying and analyzing the overall effects and common impeding factors of JICA’s projects with regard to the issue in question, this evaluation derives lessons and recommendations for the implementation of future projects focusing on those themes. It also considers effective approaches and methods to implement projects focusing on the specific theme. (2) Evaluation Types by Stages During the Project Cycle Project-level evaluation is classified into four types conducted at different stages during the project cycle: ex-ante evaluation, mid-term evaluation, terminal evaluation, and ex-post evaluation. i) Ex-ante Evaluation Ex-ante evaluation is conducted on a project requested by a recipient country. It first involves a study of the project to determine its necessity as well as its conformity with JICA Country Program. Details of the project and its expected outputs are clarified. Then, the relevance of the project is comprehensively examined and evaluated. In ex-ante evaluation, evaluation indicators are set and they are used to measure the effect of the project in subsequent evaluation, from the mid-term evaluation to the ex-post evaluation. ii) Mid-term Evaluation Mid-term evaluation is conducted at the mid-point of projects. This evaluation aims at examining the achievements and process of the project, focusing on the efficiency and relevance among the Five Evaluation Criteria. Based upon its results, the original project plan may be revised or the operation structure strengthened if necessary. iii) Terminal Evaluation Terminal evaluation is performed upon completion of a project, focusing on its efficiency, effectiveness, and sustainability. Based upon the results of the evaluation, JICA determines 8 whether it is appropriate to complete the project or necessary to extend follow-up cooperation. iv) Ex-post Evaluation Ex-post evaluation is conducted after a certain period has passed since the completion of a target project, and it is conducted with emphasis on the impact and sustainability of the project. This evaluation aims at deriving lessons and recommendations for the improvement of JICA Country Programs and for the planning and implementation of more effective and efficient projects. (3) Evaluation Types by Evaluators JICA’s evaluation can be classified by evaluators as follows: i) Evaluation by third parties (External Evaluation) In order to improve the quality and objectivity of its evaluation, JICA entrusts a certain portion of its evaluation studies to external third parties that were not involved in the planning and implementation of the projects to be evaluated. For the same reason, JICA also includes those that have high expertise in the targeted fields for evaluation, such as universities, research institutions, academics, consultants, etc. ii) Evaluation by JICA (Internal Evaluation) In order to derive lessons and recommendations that meet the actual condition or needs of recipient countries, this evaluation is conducted mainly by JICA with the knowledge of those systems and other things that surround a project or an issue. JICA also promotes the review of such internal evaluation results by third parties (academics, journalists, NGOs, etc.) with expertise in development assistance and familiarity with JICA’s undertakings to assure transparency and objectivity. iii) Joint Evaluation This evaluation is conducted in collaboration with organizations in the target countries or with the aid agencies of other donor countries. Joint evaluation with partner countries is effective for sharing recognition with JICA about the effects of and the issues regarding those projects. It also contributes to learning evaluation methods and improving the capacity of those countries in carrying out evaluation. This evaluation is effective in promoting the mutual learning of evaluation methods and aid coordination. 9 3. JICA’s Evaluation System (1) Development of JICA’s Evaluation System JICA has long been committed to improving its evaluation system. In July 1981, JICA set up the Evaluation Study Committee to deal with issues and challenges with its evaluation. The Committee has been leading JICA’s effort to develop new approaches and techniques for better evaluation. In April 1988, the Office of Evaluation was set up within the Planning Department, as a unit specializing in evaluation. In April 1990, the Office was reorganized as the Evaluation and Post Project Monitoring Division, and then put under direct supervision of the President in October 1996 for greater independence of evaluations. In January 2000, the Office of Evaluation and Post Project Monitoring was merged again with the Planning and Evaluation Department as a step to enhance the feedback of evaluation results for better project planning. In an effort to ensure objectivity and transparency in evaluation, an Advisory Committee on Evaluation composed of external experts was established in June 2002 as an advisory body for the Evaluation Study Committee. A further step to upgrade the evaluation system was taken in May 2003, when an evaluation chief was assigned to each of the departments and overseas offices directly involved in project management. The step is aimed at controlling and improving the quality of evaluation so that evaluation can meet the needs and conditions at the forefront of aid operations. The evaluation chiefs’ main responsibilities include quality control for project evaluations by their departments and offices and the promotion of evaluation feedback to those concerned. In April 2004, the Planning and Evaluation Department was reorganized into Planning and Coordination Department as a part of a reform that the Headquarters carried out after reorganizing JICA into an Independent Administrative Initiative (IAI). (2) Current Evaluation System and Roles JICA’s current evaluation system involves four main parties: the Evaluation Study Committee, the Advisory Committee on Evaluation, the Office of Evaluation and Post Project Monitoring and the project implementation departments (departments and overseas offices responsible for project operation). The principal roles of these respective parties are as follows: i) Evaluation Study Committee The committee is headed by the JICA Vice President in charge of planning and evaluation and is composed of managing directors of related departments. The committee examines and discusses JICA’s basic evaluation policies as well as the methods for giving evaluation 10 feedback. Under this committee, an “Evaluation Study Working Group” has been set up to study, deliberate and report on related issues and problems. ii) Advisory Committee on Evaluation This committee is made up of external experts (academics, NGO members, journalists, etc.) well informed about issues concerning development aid and evaluation. The committee advises the Evaluation Study Committee on evaluation systems and methods. It also reviews the results of internal evaluations to improve their objectivity. iii) Office of Evaluation, Planning and Coordination Department The Office is responsible for planning and coordinating the overall evaluation activities within JICA, including efforts to improve the evaluation methods and promote evaluation feedback. It also carries out ex-post evaluations such as country program evaluation and thematic evaluation. The Office supports and supervises evaluation activities by departments and overseas offices. iv) Departments and Overseas Offices Involved in Project Implementation Departments and overseas offices involved in project implementation conduct ex-ante, mid-term, terminal, and ex-post evaluations of individual projects under their responsibility. These evaluation results are used in managing the projects and identifying their effects. As mentioned above, the evaluation chiefs assigned to these departments and offices lead their efforts to improve the quality control for evaluations and evaluation feedback. (3) Efforts of Fostering Human Resources for Evaluation In addition to the establishment of a good evaluation system, building human resources with evaluation capacity is also essential for improving the quality of JICA’s evaluations. JICA has provided training programs for its staff both at its headquarters