Analysis: The benefit cost of benefit cost ratios

11/03/2021
Tom van Vuren

Tom van Vuren, regional director UK and Europe at Veitch Lister Consulting, and visiting professor at the Institute for Transport Studies at the University of Leeds, discusses the benefit cost ratio (BCR) process for transport projects.

The business case for BCRs

The benefit cost ratio or BCR has been getting a poor press recently for transport projects.

An Institute for Government report (1) commented on transport evidence: ‘While the BCRs calculated are comprehensive, with this sort of appraisal, the focus on this single number can hide more than it illuminates and is poorly suited for some kinds of project.'

HM Treasury’s review of the Green Book (2) was also critical, and it has led to some stating their expectations that the role of BCRs in decision-making will diminish.

The greatest problem identified by the review is, however, not the usefulness or otherwise of cost benefit analysis (CBA), but the general failure of appraisals to engage properly with the strategic context of the proposal.

That lack of strategic direction has left the presentation of the ‘best’ option heavily reliant on a benefit cost ratio that tends not to reflect the decision makers’ original objectives.

The review does not dismiss the need for the identification, quantification of benefits: 'It is vital (that) interventions … both achieve government policy objectives [in my words: the strategic case] and deliver social value for money - i.e. that maximise the delivery of economic, social and environmental returns for () society for every pound of public funds spent [in my words: the economic case]'.

You could argue that CBA struggles to appropriately quantify social and environmental returns as well, however well-intentioned economists are.

And as a reminder, a business case is of course more than a strategic case and an economic case. It must also provide evidence for the commercial case, the financial case and the management case – is the proposal viable, affordable, and achievable. It’s just that the economic case and the BCR have received most attention in the press.

We’re not alone. My VLC colleague Rhiannon Rutledge points out this same intention in the Australian Transport Assessment and Planning (ATAP) guidelines (3): 'Ensure that proposals to improve transport systems in Australia (through policies, strategies, plans and specific initiatives) achieve jurisdictional goals and objectives, provide maximum net benefit to the community and represent value for money.'

Single figures and complex problems

There are many valid reasons to decry transport planning’s real or perceived reliance on BCRs in decision-making. One is that a single figure cannot reflect the complexities and trade-offs that play when deciding whether to invest, or when choosing between different policies or options.

Some argue for a multi-criteria analysis framework, but the Institute for Government’s assessment of practice in neighbouring countries does not show a large uptake. And as Sae Chi and Jonathan Bunker say in a recent article in Research in Transportation Economics (4): 'designing a comprehensive set of MCA criteria that are appropriate for a wide range of transport projects is extremely complex'.

Others suspect that, given the complexity and opacity of transport modelling processes, the single value represented by the BCR is easily gamed by a careful choice of assumptions, many hidden in the depths of the model.

The Institute for Government report actually quotes an interviewee saying as much, but I have never experienced this, nor do I know any colleagues who have done so, or have been explicitly asked to do so. And in any case, this concern can be quite easily addressed by insisting on forensic reporting of model structure, parameters and other assumptions, and carried out through peer review and carefully designed sensitivity testing.

Fair criticism

What I do consider fair criticisms of BCRs are the following.

In the first place, as every impact, positive and negative, needs to be quantified and monetised, we end up measuring and valuing what we can measure and value not what we should. For instance, are travel time (dis)benefits a good enough and defensible proxy for all other, secondary impacts that may result from how each individual chooses to use or trade these benefits for quite different but more difficult to quantify medium to long-term choices?

The second fair criticism is the precision with which BCRs tend to be presented and used in decision-making, particularly when comparing alternatives. Given the uncertainties in modelling and forecasting, such precision isn’t warranted and gives the false impression of accuracy. Is a BCR of, say 2.21, really significantly different from one of, say 2.34? And does it matter as they both deliver high value for money? I doubt it and here’s when qualitative and strategic rather than bare quantified arguments must come into play.

Scenario testing, using quite different assumptions about the reference situation without the project, the representation of the project, or about the key behavioural parameters in the model, should also help illustrate how the assumed accuracy of some of the highly precise BCRs reported is only valid within that assumed future.

The third fair criticism is that subjectively chosen boundaries in modelling and appraisal may cloud the BCR’s message, by missing important responses to the project or policy that should also be incorporated. Examples are unintended consequences outside of the assumed geographical boundaries, or outside of transport’s immediate domain. But it could also be that the impacts of plausible social or economic policies on the transport system in the future are not recognized and captured.

And finally, as estimated benefits are esoteric, transitory or plainly distrusted, predicted costs become the driver of the BCR (and as real-life occasions have shown, these can and have also been challenged).

Fair compliments

The Treasury review makes excellent recommendations. They are not really new, but I think they are a useful reminder and confirmation of what the Green Book and its implementation in TAG have always said: 'Only options with a strong strategic case should be short listed for detailed cost benefit analysis. The BCR will then only be calculated for options which pass this test.'

The ATAP guidelines copy that: 'A top-down strategic focus – with bottom-up information.'

The BCR is a useful metric to assess variants of options that meet the strategic needs for a project, no more and no less, and ideally to compare like-for-like projects.

Rhiannon Rutledge argues: 'They shouldn’t be used to evaluate across modes, as the modelling and appraisal approaches differ. For instance, the techniques for evaluating active transport projects are much less developed than for mechanised modes. A BCR comparison implies a greater level of confidence than is warranted.'

Fair conclusions

Will the BCR disappear? Unlikely. I would suggest that the BCR has actually been a victim of its own success.

Whereas 25 years ago the quantified (dis)benefits of a project were limited to analyses of travel times, vehicle operating costs and 'accidents', this quite short list of quantified and monetised impacts in the BCR has grown since to allow, for instance, wider economic benefits, landscape impacts, noise, air quality and health benefits.

A reasonable question is whether some of these impacts should really be monetised, if the methods and input parameters are sufficiently understood by practitioners, and if this growth in monetised impacts has reduced the perceived necessity to provide a qualified strategic narrative rather than a single, precise number.

A greater focus on the strategic case also does not in any way diminish the need for solid modelling.

I agree with Phil Goodwin who said in a recent article (5): 'Greater attention to strategic coherence could see the volumes of engineers’ calculations and modelling replaced by shorter, glossier text worded by PR agencies and management consultants, which defy any attempt at challenge, evidential testing or clarification.'

And of course, there is no guarantee that a scheme or programme of schemes that fit the strategic narrative well, can provide an acceptable value for money.

There is definitely a case to make for more sensitivity testing and the modelling of alternative scenarios to support the BCR, but a range of values won’t make the life of decision-makers any easier.

I like to refer to a quote ascribed to Lyndon B Johnson: 'Ranges are for cattle. Give me a number.'

BCRs have been that number and the pressure for a single all-encompassing figure will remain.

Having said that, a very early point in the Treasury review must be realistically achievable: improving 'how results are presented and the process around investment decisions'.

A final and sobering thought: 'The review has discovered that the strategic case for many proposals is weak.'

Will a course adjustment - an increased role for the strategic narrative compared to the maligned BCR - increase rather than reduce the funding problem for transport projects?

Tom van Vuren is Regional Director UK & Europe at Veitch Lister Consulting, and a Visiting Professor at the Institute for Transport Studies at the University of Leeds.

1. https://www.instituteforgovernment.org.uk/sites/default/files/publications/evidence-transport-policy.pdf

2. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/937700/Green_Book_Review_final_report_241120v2.pdf

3. https://www.atap.gov.au/about/overview/1-introduction

4. https://www.sciencedirect.com/science/article/abs/pii/S073988592030144X?via%3Dihub

5. https://www.transportxtra.com/publications/local-transport-today/comment/67529/new-green-book-paves-way-for-shake-up-of-road-and-transport-appraisals/

Latest Issue

latest magazine issue

Inside

  • Smart motorways' BCR problem
  • New lighting decision support tool

 

 

View the latest issue


Highways jobs

Senior Project Manager

Up to £50,823 per annum
The role of this post is to oversee the project management, development and implementation of the County Council’s Major Transport Projects.. Leicestershire
Recruiter: Leicestershire County Council

Programme and Change Delivery Manager (Spatial Planning and Urban Design)

£43,833 - £53,115 Depending on skills and experience
Are you a change delivery manager with expertise in England’s local government spatial planning and urban design sector? Wandsworth, London (Greater)
Recruiter: London Borough of Richmond upon Thames and London Borough of Wandsworth

Bridges Manager – Highways and Transportation

£49,320 to £57,376
You will be responsible for... Northallerton, North Yorkshire
Recruiter: North Yorkshire County Council

Head of Major Projects and Infrastructure – Highways and Transportation

£59,526 to £67,815 plus 10% recruitment payment
This role will be pivotal to the effective delivery of our Highways and Transportation service, therefore requires an... Northallerton, North Yorkshire
Recruiter: North Yorkshire County Council

Principal Travel Plan Officer

£34,728 to £38,890 per annum
We have a fantastic opening within our team for a Principal Travel Plan Officer. Taunton, Somerset
Recruiter: Somerset County Council

Director of Economy & Environment

£96,414 to £106,650 per annum
You will play a leading role in driving forward the county’s ambitious growth programme. Gloucester, Gloucestershire
Recruiter: Gloucestershire County Council

Director of Transport & Highways

£96,414 to £106,650 per annum
You will play a leading role in driving forward Gloucestershire’s ambitious, multi-million pound transport and highways programme. Gloucester, Gloucestershire
Recruiter: Gloucestershire County Council

Board and Partnership Board Chair

Up to £60,000 per annum
This is unique opportunity to lead the Northern agenda and play a pivotal role in creating new opportunities for 15 million people in the North. Manchester/Leeds (Flexible)
Recruiter: Transport for the North

Waste Strategy Manager

£47,672 per annum
We are looking to recruit a waste strategy specialist to add the necessary technical expertise to our team. North Tyneside, Tyne and Wear
Recruiter: North Tyneside Metropolitan Borough Council

Highways Officers

£26,999 to £29,636
Are you someone who is looking to progress their career with one of the most highly regarded highway services in the country? Selby, North Yorkshire
Recruiter: North Yorkshire County Council

Highways Operative

£19,312 to £21,748
We are looking for Highways Operatives to work as part of our operations teams here at Kirklees Council. Kirklees, West Yorkshire
Recruiter: Kirklees Metropolitan Council

Senior Environmental Health Officer

£44,370 - £50,964 per annum
As a Senior Officer in the Food Safety & Infectious Disease team, you will be one of the operational leads on... Kensington and Chelsea, London (Greater)
Recruiter: The Royal Borough of Kensington & Chelsea Council

Permit Officer

£24,982 - £27,041
In October 2016 Cambridgeshire County Council launched it Permit Scheme for Road Works and Street Works. Huntingdon, Cambridgeshire
Recruiter: Cambridgeshire County Council

Traffic Systems Engineer

Competitive Salary
This is a full time role, with a minimum of 37 hours a week for a 6 month interim period, with the potential of the role going permanent. York, North Yorkshire
Recruiter: City of York Council

Transport Compliance Officer

£27,741 - £30,451 per annum
An exciting opportunity has arisen for two Transport Compliance Officers to join our Transport Services team at... Kirklees, West Yorkshire
Recruiter: Kirklees Metropolitan Council

Client Team Leader - Highway Maintenance

£40,876 - £42,821 per annum
Are you looking for a challenge that widens your experience, adds something unique to your CV and helps to make Warwickshire the best it can be?  Warwickshire
Recruiter: Warwickshire County Council

Head of Waste & Parking

£62,235–£68,615 pa
Are you passionate about waste recycling and the bigger picture of sustainability and tackling the climate emergency? Horsham, West Sussex
Recruiter: Horsham District Council

Project Engineer x2

Grade H £35,745 - £43,857
We’re looking for experienced civil engineers to join our well-regarded design and implementation team. Havant, Hampshire
Recruiter: Havant Borough Council

Environment Locality Officer

£27,741 - £29,577 per annum
The officer will play a key role in collaborative working with other internal & external stakeholders specifically... Telford, Shropshire
Recruiter: Telford & Wrekin Council

Senior Principal Structures Engineer

£39,880 - £42,821 per annum
We are looking for a dynamic structurally competent and relevantly experienced officer, to deliver the structural requirements and disciplines on... Bristol
Recruiter: Bristol City Council

Highways Presents

Highways on Fridays

Register now!

Latest Video