As I write, the United Kingdom is in the midst of a national election campaign. A month during which politicians vie to confuse the electorate with big numbers. Politics is suddenly ravaged by intangibility, because the national economy is unable to sustain the usual tangible proxies for a better life – “more schools and hospitals” – and because the tangible results of fixing that economy tend to be unattractive – “less schools and hospitals”. So the best political strategy is not explaining the consequence of choices in a language ordinary people can understand.

Do you like the sound of £100 million ($150 million)? Can I tempt you with £160 billion? Expressing these figures per person in the population can be useful. The first figure is one bar of luxury chocolate for everyone. Doesn’t sound so big now, does it? The second figure is like everyone having a £2,500 bank overdraft (loan). Strange that, because indirectly, we do.

Unfortunately, applying the economics of household groceries to major items of government expenditure introduces certainty. The idea that one can visit a store where luxury chocolate bars are sold for precisely £1.70. Yet many large elements of government expenditure are akin to ordering a chocolate bar years before it can be eaten, for a price that transpires to be somewhere between £1 and £5.

Larger businesses will be familiar with this concept. It’s called risk. Such businesses are often far more interested in what “it might cost” (£5) than what “it will cost” (£1.70), because what it might cost might lead the business to bankruptcy.

The national economy is chaotic in its complexity, but overall, things should average out. So long as all the assumptions are broadly reasonable: Ultimately some will earn/cost more, some less. Short-term in-balance can be solved by (basically) printing more money, and then down-grading future assumptions until everything is back in balance.

However, this breeds a form of arrogance. A sense that government doesn’t need to consider the possibilities. That we can deliver a radical new policy – that has never been done before – and, in spite of it never having been done before, we know precisely how much it is going to cost. Just like a bar of chocolate.

Unfortunately, assumptions tend towards optimism. On average, projected costs are less than actual costs. This isn’t just a problem for accountants. It means that decisions are taken which do not reflect reality. Potentially leading to a Disneyland scenario, where everything is affordable until after the decision is taken, when suddenly everything has become too expensive. It ultimately challenges the validity of decisions, and in doing so, the moral authority of those that take them.

This article uses the Edinburgh Tram project to demonstrate the inherent uncertainty of large government infrastructure projects. It discusses the role of optimism in planning, and the methods used to reconcile planned optimism with subsequent reality. The article describes how the involvement of the private sector in public projects has evolved over the last 20 years, and the highlights the different time-scales applied to private investment and public choices. It concludes that optimism is not only unavoidable, but necessary. Rather, the true problem lies in tendency of people to demand certainty from the public sector, while accepting uncertainty in the private sector. On this page:

Digging Holes

Edinburgh’s Haymarket used to be one the busiest transport junctions in the city. Used to, because for the last 2 years, the highway has played host to a unusual urban art form: Holes.

The exact location of these holes changes from month to month. An elaborate game to confuse pedestrians and drivers, as they fail to navigate the shortest path to their destination. Like the best urban art, the artist is rarely seen – although the presence of various pieces of mechanical digging equipment indicate that the work is somehow still incomplete. Yet the most intriguing part of this sculpture is it’s name, “Edinburgh Trams“. Because in spite of the preponderance of construction activity in the locality, the only track to be seen at Haymarket is the same mainline railway that has run under the road junction for the last 170 years. No new stations, no over-head power cables, and certainly no trams.

“Light rail” requires slightly heavier engineering than road traffic. So laying tram tracks often involves rebuilding almost everything underneath the road surface before the rails can be laid. In a large British city, this means modernising about 200 years worth of poorly (that is, not) planned subterranean infrastructure. A maze of pipes carrying all those important things that normally remain unseen – water, power, communications.

That doesn’t just cause the appearance of a lot of track-free holes. It means that we aren’t entirely sure what’s down there until we start digging. This is one example of what makes the construction planning process especially uncertain for trams: How many holes should we plan to dig, and, hence expect to pay for?

Now that most of the utility work is complete, we know that almost 50 kilometres of utilities were diverted, almost double the 27km originally estimated. Estimation which appears to be fairly typical of the way the rest of the project is progressing:

The Edinburgh tram project was originally conceived as part of a package of transport measures, funded by a new road pricing scheme. The road pricing was abandoned by popular vote, but the tram scheme proceeded anyway.

Before construction had started, a perceived shortage of funds reduced the 3 proposed tram lines (and some extensions), to 2 core routes. Yet by the time the first hole had been dug, the estimated cost of this (smaller) network had already risen above £500 million. That’s a thousand pounds for every city resident – including those that don’t have £1000. By the start of 2010, the (generally accepted) estimated cost had risen to £600 million, with some private estimates closer to £1 billion.

This in a period when (money) inflation has been almost stationary, especially in an otherwise recession-hit construction industry. And all the while, the completion date continues to slip backwards – curiously, always 3 years from now. The Edinburgh Evening News provides a timeline.

This inherent uncertainty doesn’t just make a mockery of announcing something “will cost £375 million”, when the only thing we can be reasonably sure of, is that it won’t cost precisely that much. It creates scope for optimism in planning: To bias all the cost assumptions down. Nobody can convincingly dispute an estimate, because nobody has accurate figures about the cost of building trams through Edinburgh. Estimates are biased down because securing popular (specifically political) approval for a project is easiest when the cost is lowest. Naturally, once a proposal has political support, there is considerable reluctance to abandon it. So everyone keeps digging.

Optimism Bias

The tendency for costs to be under-estimated has been apparent for several decades. Indeed, it has been formalised in the planning process as “optimism bias“. Literally, add n% to the projected cost, because, on average, past infrastructure projects have cost n% more to finish than originally estimated.

For light rail schemes, n is at least 40%. That’s quite a big error for a sector that claims to employ professional people for forecasting and planning.

Of course, adding n% still doesn’t solve the problem: We merely proceed to make even more optimistic assumptions. Claim that that much optimism bias shouldn’t apply to us. Like a gambler, convinced that their luck is somehow above average. Yet (unfortunately) secure in the knowledge that British government doesn’t “go bankrupt”: However bad the bet, those gambling debts will always be funded from somewhere. And you’d be amazed how few people working in government can explain where somewhere actually is.

This isn’t a grand conspiracy. It’s the same kind of optimism that concludes parents to believe their children are somehow “better” than they actually are at school. It’s inbred in planners and politicians alike: They’re fundamentally the kind of people that want to affect change. To get things done. They have a natural disinclination to believe proposals are unachievable.

Extend the planning horizon over (often) decades, and, frankly, almost anything can be made plausible.

Consider an operational, rather than construction example: Optimism fills our new trams with people that were previously causing road traffic congestion (policy objective), and who happily pay a premium over bus fares (operational objective). Yet pessimism would fill our new trams with former bus passengers (minimal policy impact), and who are paying bus fares (unfortunately trams have higher operating costs than buses). Only a couple of inherently uncertain assumptions separate these 2 scenarios. Separate success from failure.

The Downside of Up

There are several methods of managing the financial crisis that (invariably) occurs once reality challenges optimism.

For example, we can always do less. Build 1 tram line now, and another one “later”, maybe. Commission a single aircraft carrier, rather than a pair. It’s a terrible fix:

  • 1 tram line clearly delivers less benefit than 2 lines. If the prime objective is perception, not transport (as I contend), then running trams past the Royal Bank of Scotland headquarters and on through the city centre, may be sufficient. But it is not always so easy to scale back. A single aircraft carrier reduces the solution to farce: If one wishes to maintain an active sea-based air capability, then you can’t do it with one ship: Wars rarely adhere to maintenance schedules. Strategically, the choice is between 2 ships or no ships.
  • An initial decision to build 2 tram lines triggers an immense amount of preparatory work, the cost of which tends to be lost. Even if no work has been done, contracts have been signed, organisations have been scaled up to deal with the initial (larger) proposal. In defense procurement, the naval architecture associated with a one-ship fleet is rather similar to a 2-ship fleet.

We can also try and pass costs on to commercial contractors. Transfer the risk from government to the private sector. When the price doubles, it is the private contractor that picks up the bill. Sounds great, except:

  • Short term, relations between contractor and client can become exceedingly unpleasant. Not least because government is a less aggressive environment than (most) commerce, so public officials often aren’t prepared for unpleasantness. In Edinburgh, work has been stopped completely by disputes with the main contractor, Bilfinger Berger. Disputes that rumble on.
  • Medium term, a commercial contractor may default on the contract. The largest businesses struggle to manage hundreds of millions of pounds of unexpected cost. The result can be a half-finished thing, eventually completed late, by a far less optimistic contractor – who has inflated the cost still further.
  • Long term, the risk associated with all government contracts rises. Potential contractors increasingly regard public sectors clients as liabilities. Or more likely, greater liabilities.

Evolution of Public-Private

That final observation contributes to the view that it is more cost-effective for the private sector to do everything, with the public sector reduced to buying the result like a service. That the private sector is better at managing risk, and the public sector can only be trusted to purchase at fixed prices – like chocolate bars. It’s one of the logics behind the 1990s-era Public Finance Initiative, Design-Build-Finance-Operate, Public Private Partnerships, and. There’s another problem:

The public sector isn’t sure what kind of chocolate bar it wants. Not just that difficult choice between Almond and Toffee. Rather, today it is buying luxury chocolate, but at some point in the near future its political objectives will change, and the public appetite will be for… somewhat cheaper milk chocolate. Or maybe… biscuits. Feeling hungry yet?

The private sector process is akin to trying to order groceries a decade in advance. A nonsense to consumers that expect to decide while standing in the store, seconds before they buy.

The catch is that, unlike chocolate bars, physical infrastructure lasts decades. The private sector can only invest in, and fund, such infrastructure over decades. Faced with the certainty of political uncertainty – whatever it builds likely won’t be wanted for some of its lifetime – the only sensible solution is not to build.

Box: Over the Sea to Skye
The Skye Bridge connects the Isle of Skye, in Western Scotland, to the mainland. Although technically sea, the crossing traverses a narrow channel of water, about a mile wide. The bridge was constructed as a “Public Finance Initiative” project. The contractor both built and then continued to own the bridge, charging road users a toll. The bridge had replaced a ferry crossing, so rational (economist) planning assumed that bridge crossings could be priced much like the old ferry. Unfortunately people perceive a ferry more like a “cruise”, while a road bridge is considered more like a (“free”) road. A situation that was further confused by the economics of the original ferry operation: In the best tradition of public sector transport operation, the Skye (Kyle) ferry had been cross-subsidising other ferry routes in the Western Isles. So where paid, the Skye ferry fare had been excessive for the distance travelled, and hence bridge tolls became rather excessive. Where paid, because local people had tended not to pay to use the ferry, yet found the privately owned toll-booth less accommodating. Opposition to the toll was initially small (the area is sparsely populated), but intensely passionate. As Scotland gained devolved government (including transport responsibilities) in the late 1990s, the toll was subjected to increased political pressure. Under the previous Conservative administration, Scotland was governed much like an imperial province – devolved government was a very dramatic change to the political landscape. Tolls were first subsidised by government – at significant public cost – until complete capitulation occurred: The bridge was bought (effectively at the price of the contractor’s lost future earnings, not the price of construction), and tolls removed. This political climate then spread beyond Skye – a belief that any Scottish bridge toll was unacceptable, apparent in current discussion about the Forth Bridge. In less than 20 years, the politics of bridge tolls have been inverted. The legacy of the Skye Bridge is not just funding inefficiency (George Monbiot calculated that the bridge should have cost no more than £25 million, but actually cost over £90 million). It also clearly demonstrates how the commercial construction/investment time-scales (in this case, originally a 27-year operating contract) can be completely at odds with far shorter public/political timescales.

Uncertainty can be reduced by what civil servants used to nervously refer to as “the P word”: Partnership. By close communication, shared understanding of each others’ problems, and, ultimately, strong personal relationships between the individuals involved, both the public and private sectors can better predict and adjust to future changes in objectives. Demands that might otherwise appear “from nowhere”. In a stable political environment, this can work – that is, one with rather limited democracy. But it rarely survives the sudden and dramatic shocks to the political system, that are associated with elective democracy: 15 years of gradually evolving, but broadly similar policy by one government, may be completely reversed by a newly elected government. In the British system, literally overnight.

This big uncertainty – the risk of a sudden change in direction – makes everyone cautious. Government officials are reluctant to commit to processes their future political masters may not be committed to. Commercial organisations become reluctant to invest in a process that might be “taken away” at a later date.

The result is a move back to formal contracts, as a way of protecting everyone against this big uncertainty. Except that a “partnership-contract” looks rather too cosy. At best uncompetitive. At worst outright corrupt. And, as the Skye Bridge box above shows, contracts still won’t stop a determined politician. Such contracts merely confuse the strategy, and inflate the final cost.

Now consider that, having merged public and private into one almost seamless project-completing entity, we logically start to wonder why we ever bothered to maintain any distinction? Why not just let government do everything itself?

Why – well, among other things, because the public is too optimistic. Yes, that again. We’re back where we started.

Paradoxes of Uncertainty

The underlying problem is one of time. The paradox, that people excel at evolving and adapting from day-to-day, but struggle to predict their own evolution and adaptation. They cannot conceive in the future, what they can do in the present.

Of course, people tend to live in the present. This isn’t just reflected in the inherent difficulty of trying to order groceries a decade before they are consumed. There’s a fundamental disconnection between the present self and the future self, even though the 2 states of being are logically related.

William Hazlitt is attributed with the observation that our present self views our future self the same way as it views other people. This starts to explain why we tend towards future optimism – we’re applying the same sense of optimism to the future, that we do to other people. That hints at how deep optimism runs through the human condition.

(Optimism could be equally well described as empathy. Which, in turn, probably reveals why depression – arguably a lose of human empathy – tends to make people more rational.)


  • Clearly, we are not going to solve the problem by solving optimism. It’s a part of what it is to be human.
  • Indeed, managing optimism is no easier than managing people – a largely empathic process that tends to fail when applied rationally, or into the future.
  • Meanwhile, optimism appears fundamental to the types of decisions that push society forward. Optimism is what biases human endeavour away from the currently accepted average. Possibly a necessary condition of “trial-and-error” evolution.

I reluctantly conclude that optimism is not only unavoidable, but entirely necessary.

So why is it such a problem for the public sector? Simple: Government doesn’t do uncertainty. Much as it doesn’t make mistakes. Of course it does, it just has terrible difficulty acknowledging that it does.

An investor in a commercial business accepts that some of the money they invest will be spent on unprofitable activities. They judge the success of their investment on the overall return of the entire business. In contrast, public investors in a government enterprise – or taxpayers, as they are called – expect every penny (cent) to be “well spent”. Anything the population deems to be “waste” can approximate to a moral right to “demand their money back”, with almost no consideration of overall performance.

Culturally this makes it hard for the public sector to acknowledge misjudgements, consequently harder to learn from them, and hence, exceptionally hard to learn from them while they are happening – at the time when something still might be done to improve the situation. In practice, either the situation gets so bad that it dissolves into scandal and farce, or once the dust has settled, the National Audit Office (or Scottish equivalent) reports about the woes of British tram procurement. The current generation of local politicians and officials will learn not to try that again (I suspect they already know). But nobody will address the underlying structural problem.

Which leaves an important question: Why do people demand financial certainty from the public sector, while simultaneously accepting uncertainty from the private sector?

8 thoughts on “Optimism”

  1. Hi Tim

    Interesting view on Edinburgh Trams and of Skye Bridge.

    The biggest problem in the trams project is not the ability to add bias but that of managing a project to a cost plan to deliver what was sold to most as a fixed price project. The bespoke contracts drawn up by Edinburgh were incompetent in the extreme.

    As the person who led the Skye Bridge Project the same would be true of the then Scottish administration. To use you analogy of procuring products, Skye was like the Government ordering a new aircraft on a 20 year lease and then demanding to own it 10 years into that lease.

    The break clauses would factor in the change for breaking the lease. By the way in this case there was a huge niaive error built into the deal and what was called a reversion clause.

    I think rather than there being an optimism problem there is more on a competence in the people that will work for government on public infrastucture projects!

  2. John – I think we’re both heading towards a similar conclusion: The key issue is the management of an inherently uncertain process. Our disagreement would be the extent to which this is a societal problem, or a competence problem. I suspect both factors are present, but that the societal problem may be the most difficult one to solve…

  3. Hi Tim

    I would not disagree, it is all down to which one has had the most influence.
    It need not be inherently uncertain, this too comes down to competence!
    Take the trams for instance, the utilities could easily have been separated from any impact on the infrastructure contract. It was conceit on the part of Tie to think they could manage that interface.

    All of the English tram projects were done on a PFI contract where the bid consortium took the risk!

  4. […] Optimism – Tim Howgego An investor in a commercial business accepts that some of the money they invest will be spent on unprofitable activities. They judge the success of their investment on the overall return of the entire business. In contrast, public investors in a government enterprise – or taxpayers, as they are called – expect every penny (cent) to be “well spent”. Anything the population deems to be “waste” can approximate to a moral right to “demand their money back”, with almost no consideration of overall performance.Culturally this makes it hard for the public sector to acknowledge misjudgements, consequently harder to learn from them, and hence, exceptionally hard to learn from them while they are happening – at the time when something still might be done to improve the situation. […]

  5. […] My classic explanation for Edinburgh’s trams is that “we like trains”. Specifically people outside of Edinburgh, but whom the city might like to attract, “like trains”. For historic and geological reasons (a mix of sustained service quality and hills never giving railways a chance) bus use has remained socially acceptable for a large proportion of Edinburgh society. Hence the native enthusiasm for trams was never overwhelming, even before the troubled construction began. And by the time it began again opinions tended towards the negative. I eventually lost track of all the holes. […]

Comments are closed.