Manel Torralba Flickr

Making an impact

09 November 2016

What impact are public policy schools having on society and how can that be better measured and evaluated? Paul Harris and Björn Dressel offer some suggestions.

The “impact” debate is here to stay. Universities around the world are grappling with how to respond to a rising tide of metrics and rankings, not to mention the expectations of government, industry, the community – and their own staff and students – that they will engage more actively with society beyond the walls of academia. All this comes together in the debate about what impacts universities can be said to have in society and how they could be better evaluated and encouraged.

The National Innovation and Science Agenda which the Australian Government launched in December last year states that “for the first time, Australia will introduce a systematic national assessment to measure these impacts”. The Australian Research Council (ARC) is now working on the design of an assessment framework to be tested in 2017 and applied widely in 2018.

This effort builds on more than five years of debate and policy work, underpinned by the idea that while we perform well on traditional indicators of things like research quality, Australia “underperforms in measures of university and end-user collaboration”. By better measuring and understanding what is already happening, the government’s aim is to stimulate collaboration in order to produce more desirable impacts.

As a part of the innovation agenda, these initiatives tend to focus on engagement with industry and generating commercial returns. For us, that raises a serious question: what should public policy schools – defined here as professionally-oriented tertiary education institutions that offer postgraduate degrees in public policy – be doing about this impact agenda? What kinds of impacts do they, and should they, have not just in public policy but also in society more generally?

The dominant story about the relationship between academia and public policy is a persistently negative one about a “gap” or “chasm” between two separate worlds. So the first challenge is to simply work out what, if anything, is already happening.

We know from impact assessment exercises elsewhere in the world that the work of academics does indeed connect in significant ways with public policy and the work of government. The UK Research Excellence Framework (REF) exercise conducted in 2014 assessed research impact for the first time, with almost 7,000 case studies produced by 154 universities. Analysis of these cases found that the three most common types of impact cited were “informing government policy”, followed by “supporting Parliamentary scrutiny” and “technology commercialisation”. More than 40 per cent of all the REF case studies mentioned impact on public policy.

Closer to home, we know that a significant portion of the partners in the ARC’s Linkage programmes (which are designed to fund collaboration between university researchers and the end-users of research) are government agencies. A 2013 “mapping” report of links between science and policy in Australia similarly found hundreds of government-funded programs, advisory committees and contract research projects awarded to universities. So there is a lot already going on that is relevant to assessing impacts in public policy, but neither universities nor governments currently gather and evaluate information about these activities in any systematic way.

A second challenge comes from the flood of new metrics and public policy school rankings that are now popping up, measuring schools based on publications and citations. But the staff in these schools do more than just publish and cite. For instance, they teach, offering a distinctly interdisciplinary curriculum that students can apply in their careers and lives as citizens. Staff in public policy schools also offer tailored professional development programs for government officials, and academics all across universities serve on advisory committees and councils, undertake specific consultancies for government and many other forms of engagement and outreach, including writing for a general audience about policy issues and speaking with the media.

We believe that public policy schools can lead the way in thinking about the design of impact assessment frameworks that reflect both the diverse activities and diverse impacts of universities in society, including in public policy. Staff in these schools also have something important to contribute on the relationships between expertise, evidence and end-users; they have, after all, been studying (and living) them for decades.

The public policy literature shows us that the interactions between universities and policy-making are complex, dynamic and context-specific. So in seeking to better evaluate the long-term impact of these interactions, simple quantitative metrics or a “one-size-fits-all” model probably won’t be much help. This complexity represents a third challenge for public policy schools, but again the literature can help.

Forty years ago, Professor Carol Weiss from Harvard University sought to understand the uptake and use of research in public policy, at a time when there was “well publicised concern” among both academics and government officials about the usefulness and public value of research. In her 1979 paper The Many Meanings of Research Utilization, Weiss sets out seven modes of research utilisation that remain influential today, including:

  • Knowledge-driven: a linear model derived from the physical sciences, where basic research leads to applied research which in turn leads to application
  • Problem-solving: seeking knowledge to address a pre-existing problem or gap in decision-making, with government commissioning research
  • Political: research used as “ammunition” in political debate, often to support predetermined positions
  • Enlightenment: research findings “percolating through informed publics” and shaping the ways in which people think about particular issues.

Weiss crucially allows for the role of politics in policy, as well as the role of universities to challenge government from time to time, as well as working constructively with it. Weiss notes that while it is the “enlightenment” model which is perhaps the most common pathway to policy impact, it can be indirect and inefficient.

She concludes that “it probably takes an extraordinary concatenation of circumstances for research to influence policy decisions directly”. But, she cautions, this does not mean that the work of academics, in all its variety, does not have significant impact in public policy when more broadly understood. As Weiss noted almost 40 years ago, frustration at the apparent “gap” between research and policy may stem from the fact that observers have only one model in mind, when the full picture is richer.

What we now need to do is learn from these conceptual models and apply them in impact assessment frameworks that can incorporate not just research, but all the diverse activities and positive roles of universities.

At a time characterised by fears of a “war on experts” and a potential weakening of trust in the social structures that produce both knowledge and policy, it is crucial that public policy schools engage with these questions. We believe that talking about impact, and working through what that means for designing and evaluating all the diverse kinds of work we do, can itself have a positive impact by improving dialogue, accountability and transparency.

Many people are busily working on new ways to strengthen collaboration between universities and industry (and assess their impact) – it is time for public policy schools to get on with it too. Given the growth in the number of public policy schools in the Asia-Pacific region since the late 1990s, we think these questions are important to other universities and governments as well. There will be no single, simple, silver bullet solution to the challenges posed by the impact debate – public policy schools will need to respond in ways that fit their own context. But this is an important opportunity for collaboration, engagement and better demonstrating their significant value to society.

This article was published by the Australian and by Policy, the website of the Asia and the Pacific Policy Society and Crawford School.

Filed under:

Updated:  18 July 2024/Responsible Officer:  Crawford Engagement/Page Contact:  CAP Web Team