Effective evaluations of funding programs and instruments are widely recognized as fundamental to informing the preparation of public policy for science. The relevance of effective impact evaluation is especially pertinent in times of increasing budgetary constraints. However, in OECD member states, the landscape of impact evaluation of public research (done in public and semi-public institutions) is diverse and inconsistent, so there is a clear need to reflect on the different practices in place, on implementation, and on the effectiveness of impact evaluation as underpinning policy decisions.
This was the main objective of the workshop "Assessing the Impacts of Public Research Systems", which took place in Lisbon at the end of April. In a joint FCT-OECD organization, researchers, practitioners of public research systems and policy makers from 14 OECD countries, as well as representatives of the European Commission, Science Europe and the OECD itself, met to reflect on and learn from the experiences of the various countries in assessing the impact of public research systems. The results of the workshop pave the way for the definition of priorities, indicators, and best practices that can serve as a foundation for cross-country analyses.
The international panorama of impact evaluation is indeed disparate: in several countries, ex-post evaluation is mandatory when programs end; some countries also require ex-ante impact analysis to support programs and policies; a small group of countries go further, implementing long-term evaluations to measure the economic and social impact of programs or organizations. Portugal is in the group (made up mostly of southern European countries) that, as a general rule, implement evaluation exercises as a requirement for using European Structural Funds. Clearly, these countries, and others in the OECD, can benefit from the knowledge of countries with more experience in this area, and thus develop not only expertise but also a solid knowledge base (namely through the enlargement of the scientific community).
Despite existing disparities, the scope of evaluations has been increasing since the first studies in the 1960s. Any evaluation of public research systems today covers individuals and their academic and professional paths, organizations and institutions, networks and, more recently, ecosystems and policy-mixes. This comprehensiveness implies the simultaneous and rigorous analysis of several instruments and, in particular, of the often complex interactions between the instruments.
The challenges increase when moving to the level of indicators and methodologies to be applied in transnational analyses. In this sense, the Lisbon workshop showed that sharing knowledge, identifying common issues and concerns, and reflecting on shared experiences and results are essential steps for the success of the process.
Workshop participants identified that the different challenges associated with impact assessment studies at the regional, national or international level may require different methodological approaches. They also stressed the importance of incorporating impact assessment studies into regular institutional practice in the policy making process. The issue of excellence in impact evaluation was also addressed, recognizing that only studies conducted according to the highest international quality criteria effectively contribute to the optimization of public policy design.
The results of the workshop will be directly used in the Knowledge Triangle project of the Working Party on Innovation Policies (TIP) of the OECD Committee for Scientific and Technological Policy (CSTP). This new project aims to analyze the various aspects of the involvement of universities, research centers and business in promoting innovation. Some of the issues that the project will address are: incentives for researchers and other actors in the system, the role of governments as coordinators and agents in the system, the identification of new forms of collaboration between universities and society, and also the policies that best harmonize the different functions (current and new) of research, education and innovation in the Knowledge Triangle. Luisa Henriques (FCT) and Ricardo Mamede (ISCTE) are members of the Steering Group of the project.
Impact assessment is one of the four modules of the Knowledge Triangle project, the other three being (1) Higher education institutions in the Knowledge Triangle; (2) New funding, cooperation, and governance regimes; and (3) Place-based policies in support of the Knowledge Triangle.