Table 1: Policy report Average Self-Assessment Scores Development Objectives (DO) Inform Build Client Influence Inform Government Analytical Inform/Stimulate Development Year Lending Policy Capacity Public Debate Community 2008 0.79 0.72 0.76 0.81 0.69 2009 0.79 0.72 0.71 0.84 0.67 2010 0.79 0.72 0.74 0.83 0.71 2011 0.75 0.72 0.77 0.84 0.70 2012 0.85 0.72 0.71 0.83 0.75 2008–2012 0.79 0.72 0.74 0.83 0.70 Reports with DO 484 861 489 773 496 % of Policy reports 49% 32% 31% 46% 26% with a score of 1 Source: Business Warehouse. Note: A score of zero indicates that the DO was not met; a score of one indicates that the DO was fully met. ii) Client feedback on knowledge products is routinely being sought. Recently, the World Bank has begun to systematically gather and incorporate client feedback for ESW and TA. The Bank sends surveys to government counterparts of selected ESWs and TA to seek their view regarding the quality, relevance, and impact of the provided knowledge service. The most recent survey was completed at the beginning of FY13. Furthermore, work is ongoing to have client feedback information disseminated for economic sector work, non-lending technical assistance, external training and internal knowledge products (World Bank, 2012c). A product-specific survey was sent to users of ESW and TA for 210 projects that were completed in FY12, with preliminary results showing that many of these products were considered by their users to be effective at achieving their agreed objectives. The effectiveness of these reports was measured on five characteristics: i) how effective they were at addressing the specific development goals of their agency, ii) their technical quality; iii) their use of best available data, iv) how effective they were in engaging the clients during the design, implementation, and completion of the work, and v) whether the product or service was delivered in a timely manner. Table 2 below shows the descriptive statistics of this most recent report, measured on a six point scale, with 1 being considered to be “Very Ineffective” and 6 considered to be “Very Effective.” As mentioned in World Bank (2013b), the average ratings for quality and likely impact are deemed to be effective, with median ratings of 5 on a 6 point scale. This is commensurate with the self-assessment that TTLs gave regarding whether their policy report informed public debate. The client survey also asks whether or not the knowledge product or activity has led to specific changes, either through policy, regulations, or institutional changes. According to the latest survey, reports scored very high with respect to relevance and technical quality and more than three-quarter led to a change in policies, regulations or institutions.15 15 There are a few shortcomings with the client feedback surveys. First, the survey is likely to suffer from significant selection biases arising from a high non-response rate and the fact that the TTL could choose the respondents. Feedback was provided only for 113 out of 210 projects for the question over whether the policy report had led to a policy change within the client agency or institution and the number of feedback providers varied significantly across reports. Second, feedback was requested for a smaller number of projects than in our dataset (210 ESW and TA projects were analyzed in the client feedback survey for FY10, while our dataset covers 275 policy reports). 9