Tom van Vuren, regional director UK and Europe at Veitch Lister Consulting, and visiting professor at the Institute for Transport Studies at the University of Leeds, discusses the benefit cost ratio (BCR) process for transport projects.
The business case for BCRs
The benefit cost ratio or BCR has been getting a poor press recently for transport projects.
An Institute for Government report (1) commented on transport evidence: ‘While the BCRs calculated are comprehensive, with this sort of appraisal, the focus on this single number can hide more than it illuminates and is poorly suited for some kinds of project.'
HM Treasury’s review of the Green Book (2) was also critical, and it has led to some stating their expectations that the role of BCRs in decision-making will diminish.
The greatest problem identified by the review is, however, not the usefulness or otherwise of cost benefit analysis (CBA), but the general failure of appraisals to engage properly with the strategic context of the proposal.
That lack of strategic direction has left the presentation of the ‘best’ option heavily reliant on a benefit cost ratio that tends not to reflect the decision makers’ original objectives.
The review does not dismiss the need for the identification, quantification of benefits: 'It is vital (that) interventions … both achieve government policy objectives [in my words: the strategic case] and deliver social value for money - i.e. that maximise the delivery of economic, social and environmental returns for () society for every pound of public funds spent [in my words: the economic case]'.
You could argue that CBA struggles to appropriately quantify social and environmental returns as well, however well-intentioned economists are.
And as a reminder, a business case is of course more than a strategic case and an economic case. It must also provide evidence for the commercial case, the financial case and the management case – is the proposal viable, affordable, and achievable. It’s just that the economic case and the BCR have received most attention in the press.
We’re not alone. My VLC colleague Rhiannon Rutledge points out this same intention in the Australian Transport Assessment and Planning (ATAP) guidelines (3): 'Ensure that proposals to improve transport systems in Australia (through policies, strategies, plans and specific initiatives) achieve jurisdictional goals and objectives, provide maximum net benefit to the community and represent value for money.'
Single figures and complex problems
There are many valid reasons to decry transport planning’s real or perceived reliance on BCRs in decision-making. One is that a single figure cannot reflect the complexities and trade-offs that play when deciding whether to invest, or when choosing between different policies or options.
Some argue for a multi-criteria analysis framework, but the Institute for Government’s assessment of practice in neighbouring countries does not show a large uptake. And as Sae Chi and Jonathan Bunker say in a recent article in Research in Transportation Economics (4): 'designing a comprehensive set of MCA criteria that are appropriate for a wide range of transport projects is extremely complex'.
Others suspect that, given the complexity and opacity of transport modelling processes, the single value represented by the BCR is easily gamed by a careful choice of assumptions, many hidden in the depths of the model.
The Institute for Government report actually quotes an interviewee saying as much, but I have never experienced this, nor do I know any colleagues who have done so, or have been explicitly asked to do so. And in any case, this concern can be quite easily addressed by insisting on forensic reporting of model structure, parameters and other assumptions, and carried out through peer review and carefully designed sensitivity testing.
Fair criticism
What I do consider fair criticisms of BCRs are the following.
In the first place, as every impact, positive and negative, needs to be quantified and monetised, we end up measuring and valuing what we can measure and value not what we should. For instance, are travel time (dis)benefits a good enough and defensible proxy for all other, secondary impacts that may result from how each individual chooses to use or trade these benefits for quite different but more difficult to quantify medium to long-term choices?
The second fair criticism is the precision with which BCRs tend to be presented and used in decision-making, particularly when comparing alternatives. Given the uncertainties in modelling and forecasting, such precision isn’t warranted and gives the false impression of accuracy. Is a BCR of, say 2.21, really significantly different from one of, say 2.34? And does it matter as they both deliver high value for money? I doubt it and here’s when qualitative and strategic rather than bare quantified arguments must come into play.
Scenario testing, using quite different assumptions about the reference situation without the project, the representation of the project, or about the key behavioural parameters in the model, should also help illustrate how the assumed accuracy of some of the highly precise BCRs reported is only valid within that assumed future.
The third fair criticism is that subjectively chosen boundaries in modelling and appraisal may cloud the BCR’s message, by missing important responses to the project or policy that should also be incorporated. Examples are unintended consequences outside of the assumed geographical boundaries, or outside of transport’s immediate domain. But it could also be that the impacts of plausible social or economic policies on the transport system in the future are not recognized and captured.
And finally, as estimated benefits are esoteric, transitory or plainly distrusted, predicted costs become the driver of the BCR (and as real-life occasions have shown, these can and have also been challenged).
Fair compliments
The Treasury review makes excellent recommendations. They are not really new, but I think they are a useful reminder and confirmation of what the Green Book and its implementation in TAG have always said: 'Only options with a strong strategic case should be short listed for detailed cost benefit analysis. The BCR will then only be calculated for options which pass this test.'
The ATAP guidelines copy that: 'A top-down strategic focus – with bottom-up information.'
The BCR is a useful metric to assess variants of options that meet the strategic needs for a project, no more and no less, and ideally to compare like-for-like projects.
Rhiannon Rutledge argues: 'They shouldn’t be used to evaluate across modes, as the modelling and appraisal approaches differ. For instance, the techniques for evaluating active transport projects are much less developed than for mechanised modes. A BCR comparison implies a greater level of confidence than is warranted.'
Fair conclusions
Will the BCR disappear? Unlikely. I would suggest that the BCR has actually been a victim of its own success.
Whereas 25 years ago the quantified (dis)benefits of a project were limited to analyses of travel times, vehicle operating costs and 'accidents', this quite short list of quantified and monetised impacts in the BCR has grown since to allow, for instance, wider economic benefits, landscape impacts, noise, air quality and health benefits.
A reasonable question is whether some of these impacts should really be monetised, if the methods and input parameters are sufficiently understood by practitioners, and if this growth in monetised impacts has reduced the perceived necessity to provide a qualified strategic narrative rather than a single, precise number.
A greater focus on the strategic case also does not in any way diminish the need for solid modelling.
I agree with Phil Goodwin who said in a recent article (5): 'Greater attention to strategic coherence could see the volumes of engineers’ calculations and modelling replaced by shorter, glossier text worded by PR agencies and management consultants, which defy any attempt at challenge, evidential testing or clarification.'
And of course, there is no guarantee that a scheme or programme of schemes that fit the strategic narrative well, can provide an acceptable value for money.
There is definitely a case to make for more sensitivity testing and the modelling of alternative scenarios to support the BCR, but a range of values won’t make the life of decision-makers any easier.
I like to refer to a quote ascribed to Lyndon B Johnson: 'Ranges are for cattle. Give me a number.'
BCRs have been that number and the pressure for a single all-encompassing figure will remain.
Having said that, a very early point in the Treasury review must be realistically achievable: improving 'how results are presented and the process around investment decisions'.
A final and sobering thought: 'The review has discovered that the strategic case for many proposals is weak.'
Will a course adjustment - an increased role for the strategic narrative compared to the maligned BCR - increase rather than reduce the funding problem for transport projects?
Tom van Vuren is Regional Director UK & Europe at Veitch Lister Consulting, and a Visiting Professor at the Institute for Transport Studies at the University of Leeds.
1. https://www.instituteforgovernment.org.uk/sites/default/files/publications/evidence-transport-policy.pdf
2. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/937700/Green_Book_Review_final_report_241120v2.pdf
3. https://www.atap.gov.au/about/overview/1-introduction
4. https://www.sciencedirect.com/science/article/abs/pii/S073988592030144X?via%3Dihub
5. https://www.transportxtra.com/publications/local-transport-today/comment/67529/new-green-book-paves-way-for-shake-up-of-road-and-transport-appraisals/
