Table 2: Client Feedback Indicators for FY 2012 Effectiveness in terms of: Count Mean Standard Deviation Median Achievement of agreed objectives 192 4.89 0.79 5.00 Relevance 195 5.06 0.75 5.00 Technical Quality 191 5.05 0.79 5.00 Engagement 195 4.95 0.81 5.00 Timeliness 192 4.60 1.00 5.00 Policy Change? (Yes=1) 113 0.77 0.38 1.00 Source: World Bank Operation Policy and Country Services. Note: The Effectiveness indicators are measured on a scale of 1 to 6, while the Policy Change indicator is a binary indicator. iii) Performance assessment reviews are also used to evaluate policy reports. Performance assessment reviews (PAR) are part of the IEG initiative to assess the impact of Bank ESW and TA. This initiative is meant as a replacement for the evaluations formerly performed by the Quality Assurance Group (QAG), which was disbanded in FY10. Until now there has not been an autonomous review process of AAA (IEG, 2012). These ratings consist of the following categories: results, relevance, technical quality, and dialogue/dissemination. All measures use six-point scale ranging from “highly satisfactory” to “highly unsatisfactory”. They are based on four sources: reviews of the content of the policy report to establish substantive content; reviews of the documentary record to probe into process issues, including inception, client and stakeholder engagement, quality control, and dissemination; interviews of country directors and task managers that were familiar with the reports as well as the network responsible for the reports; and interviews of government officials and other stakeholders in the public and private sectors in the client countries as well as staff in the resident offices (World Bank, 2010b; World Bank, 2011b).16 This study measures the demand for and use of policy reports through downloads and citation counts. These two indicators are objective measures which serve as a complement to the three above- mentioned approaches to measuring the quality of these types of knowledge products. Download counts capture the intent to use World Bank policy reports. More specifically, download counts refer to the number of times a PDF has been downloaded from the World Bank’s external website. As noted in Wagstaff (2012b), downloads are an excellent indicator of the use of knowledge created by the Bank, particularly the intent to use the document since it is reasonable to assume that the person who downloads a policy report would at least take a look at the contents of the document. We consider this indicator an objective measure of whether policy report meets the development objective of informing the public debate. Citations counts measure how often a report was cited by other publications. In academia, research papers cite other publications that they have used in performing research. Citations are considered a good indicator of the influence of academic research and are widely used for this purpose in all fields. While citations are a commonly used metric for analyzing the impact of published academic articles, they have not been used to assess the demand of policy reports, in part because policy reports are 16 While these documents are available within IDU, there does not appear to be an organized database that contains these reviews, nor is there an official guideline provided by IEG explaining the methodology of PARs. 10