Lining up the dominoes: lessons from art research on how to evidence impact
Drawing on case studies from UK's 2014 Research Excellence Framework
For research to have impact is becoming more and more important, so being able to convincingly evidence that impact is a valuable skill. Lesley Brook has studied how the impact of art research was evidenced during the 2014 Research Excellence Framework and shares lessons also applicable to a broader range of disciplines. While achieving impact is not a simple linear pathway, a complex domino fall can serve as useful analogy: starting with one domino, a cascade of research outputs and related activities fall along multiple branching pathways to contribute collectively to the overall impact of research and to evidence of that. Be sure to remove superfluous dominoes, fill in any gaps, and don’t end your lines too soon!
Research impact matters. In some countries, like the UK through its Research Excellence Framework (REF) exercise, funding for higher education institutions depends partly on evidence their research is having impact. But there are additional reasons why impact is important:
- To maximise the benefits of the research for our societies
- To strengthen applications for research funding and provide accountability for research funding
- To build the reputation of the researchers and their institution.
These reasons apply to all research disciplines. Art therefore needs to be able to evidence research impact, but this is particularly difficult where the impact occurs in people’s hearts and minds, and where the art researchers may not be able to define, let alone identify, who their research has reached, for example through a public exhibition.
To find out how the impact of art research might be evidenced, I analysed a selection of REF2014 impact case studies classified under the Studies in Creative Arts and Writing research subject area. I was looking for insights into the types of evidence that could be used, different ways to gather evidence, and how to use evidence persuasively. Having filtered my selection down to 63 unique impact case studies, I extracted 24 different types of evidence of impact, grouped into eight categories under three themes.
The full article is available in the Journal of Arts Management, Law, and Society. In this post I’d like to focus on three key lessons about how to use impact evidence persuasively. Though these lessons have been drawn from an analysis of art research, each can be applied to a broader range of disciplines.
Although achieving impact is not a simple linear pathway, an analogy with a complex domino fall may be useful. Starting with one domino, a cascade of research outputs and related activities over time can fall along multiple branching pathways to contribute collectively to the overall impact and to evidence of that.
1. Remove superfluous dominoes
Some of the impact case studies I analysed included information that was not directly relevant to impact, or added little value to the impact case study.
- Evidence of activity or “busy-ness”: the number of events held, for example, adds nothing to the number of people who attended, unless accompanied by information showing that different types of events reached different audiences. In a similar vein, if you know the number who attended, then it does not matter whether that exceeded expectations.
- Evidence of the quality of research: this might be evidence of an award won or of critical acclaim for the research. Unless impact is expressly identified as one of the reasons for the award or acclaim, it is not safe to assume that high-quality research has impact – isn’t that why the REF2014 required separate evidence of impact?
- Evidence of invitations: for example, invitations to present. For the audience at the presentation, I suggest it makes no difference whether the presenter was invited or not. The evidence of the reach and significance of the impact depends upon the audience, not on the circumstances giving rise to the presentation occurring.
- Evidence of funding for the work: if one of the criteria for funding is the impact of previous research, then yes, this can be evidence of prior impact but otherwise funding is evidence of impact for the researcher and/or their institution, not for anyone else.
If evidence is not relevant to the impact of the research, leave it out!
2. Fill in big gaps between dominoes
In some impact case studies I had to guess why some of the evidence was being relied upon. I suggest it would be far more effective to tell readers why the evidence is relevant. If you want readers to draw inferences, don’t just give them the evidence, lead them!
- “Before and after” evidence is gold but was rare. If you want to show that awareness or understanding or appreciation has changed, that should not be assumed but supported by evidence. What were levels of awareness like before the event? What do audience members say about what changed for them?
- Context can be useful, for example how do attendance numbers compare with the city population or with attendance at similar events?
- Many impact case studies assumed that reaching varied or “harder-to-reach” audiences was a Good Thing, but did not explain why or how the impact for those audiences was different in quality or significance than for other audiences.
If you can explain how the evidence supports the inference you want readers to draw, your claimed impact should be more credible.
3. Don’t end domino lines too soon
I found examples of impact case studies which missed opportunities to go further and highlight additional impact.
- Evidence of further dissemination: for example, media reviews of an exhibition. What is the circulation? How large is the secondary audience exposed to the research in this way, even if they might not have visited the exhibition?
- Evidence of implementation: for example, policy change. When did the policy change take effect? Who has benefited from the policy change? What are those benefits?
- Evidence of acquisition of an artwork by a public institution: has that work gone on display? Approximately how many people have seen it as a result?
- Evidence of purchases at an art gallery shop: these may not merely be evidence of economic impact for the gallery but also evidence of audience members’ desire to remind themselves of what they have seen or even dig deeper into the subject matter.
As well as identifying these opportunities to improve the use of evidence, this study helps researchers to identify which types of evidence they might use and to plan how to gather that evidence. Third parties may have relevant evidence, and audience members could be provided with opportunities to engage and communicate the impact for them, for example by recording their thoughts in response to what they’ve experienced, commenting in a workshop or discussion, reflecting on their learning, or taking relevant action.
This blog post is based on the author’s article, “Evidencing Impact from Art Research: Analysis of Impact Case Studies from the REF 2014”, published in the Journal of Arts Management, Law, and Society (DOI: 10.1080/10632921.2017.1386148).
About the author
Lesley Brook works in the Research and Postgraduate Studies portfolio at Otago Polytechnic, New Zealand.