06 February 2018
Richard Clubbe, Research and Policy Officer at Sense about Science, talks about a new report on how transparent the government is about the evidence behind policy and why it matters.
Last week we published a review of how well government shares the evidence and reasoning behind its policies. Transparency of Evidence: A spot check of government policy proposals July 2016 to July 2017 assessed 94 policies from 12 ministerial departments on how transparent they were about the evidence considered. This research was conducted in partnership with the Institute for Government and funded by the Nuffield Foundation and the Alliance for Useful Evidence.
In order to hold policymakers to account, citizens need to be able to follow the “chain of reasoning” that has led to a policy decision. If the evidence that has been considered is unclear or unreferenced, it becomes increasingly difficult to determine whether that evidence provides a sound basis for the government’s actions. Without transparency, researchers find it harder to contribute to the evidence base, and future policymakers are less able to build upon past government research and experience.
Our report does not judge the merits of policy or the quality of the evidence base, but whether a reader can identify what evidence was used and how it was assessed. In 2015, the Institute for Government, Sense about Science and the Alliance for Useful Evidence developed an evidence transparency framework, applicable to any policy proposal, which broadly asks: ‘can the public see what the government is proposing to do, and why?’ That framework examines four components of policy – diagnosis, proposal, implementation and testing and evaluation – and scores them on a scale of 0 to 3.
To get top marks, policy proposals should clearly present and reference the evidence that has been considered, discuss its strengths and weaknesses and acknowledge uncertainties or gaps in the evidence base. Conversely, when readers can’t see what evidence has been used or the rationale for a policy proposal, policies achieve low scores. The best policies display a clear chain of reasoning throughout the proposal.
In 2016 we published a preliminary review of evidence transparency, highlighting good and bad practice across government. Since this, we have worked together with government departments, presenting our findings in meetings and workshops for policy professionals and analysts from across government, as well as convening a methods review group with senior analysts from a dozen departments and agencies.
This week, we published the results of a spot check on transparency performance, one year on. We scored 94 randomly selected policies announced by 12 government departments between 13 July 2016, when Theresa May became Prime Minister, and the end of July 2017 (more on our methodology can be found in Section 2 of this year's report). For the review, we assembled the documents available at the point when the government first set out a policy publicly. This is important - it is at this point that the public, parliament and the media first have the chance to assess a new proposal - and when it is important that the government exposes the evidence behind its initial thinking in order to promote informed engagement.
A group of 27 volunteers gave up their valuable time to score these policy documents against the evidence transparency framework. Many of them were early career researchers. They did not need expertise in the policy subject or any prior experience in policy analysis - the framework is designed to be applied rapidly and by any motivated citizen.
Reviewers found that departments can attain high standards of transparency in diverse policymaking situations - showing, importantly, that evidence transparency is achievable. The most consistently high scoring departments were the Department for Transport (DfT), Department for Business, Energy and Industrial Strategy (BEIS), Department of Health (DH), Department for Environment, Food and Rural Affairs (DEFRA) and Department for Work and Pensions (DWP). DfT produced the highest scoring policy, 'Cutting down on noise from night flights', which received a 3 in every section of the framework. This was a strong example of how it’s possible to not only share the evidence being discussed, but also explore its limitations in the context of the policy area.
The spot check reveals signs of real improvement in evidence sharing across government. However there is still a long way to go before departments consistently display the evidence and reasoning behind their policy proposals. One of the most alarming findings in 2016 was the volume of research evidence gathered internally but omitted from publication. We observed fewer cases of that in this spot check, as well as broader improvements in referencing. However, on several occasions reviewers found that over-sharing of less relevant evidence muddied the chain of reasoning; often the briefest policy documents were the clearest.
Use of the evidence transparency framework is not restricted to formal reviews. Our aim is to increase transparency to allow those outside government to better engage with policymaking. A transparent evidence base enables a better conversation between policymakers and researchers, or others with expertise, about the pros and cons of different interventions, and about the quality of the evidence base itself. Analysts and academics should welcome the use of the framework to better identify where new studies are needed and when they can submit evidence to help inform policy development.
We are currently planning to conduct a further spot check in two years’ time. The time period to cover and scope of departments, agencies and devolved governments to which we might extend the review will be decided over 2018.
Richard Clubbe is Research and Policy Officer at Sense about Science and contributed to the research for this report. Sense about Science is an independent campaigning charity that challenges the misrepresentation of science and scientific evidence in public life.
This project was funded by the Nuffield Foundation, but the views expressed are those of the authors and not necessarily those of the Foundation. More information is available at www.nuffieldfoundation.org.
You can also read CaSE's 2017 report, 'Improving use of evidence in UK government policymaking' , which explores the structures, processes and practice for science advice in government and makes recommendations to strengthen them.