Why the NBN's cost-benefit analysis is flawed

The assumptions underlying the NBN Cost Benefit Analysis (CBA) just don't stack up and no amount of justification will bring the CBA anything more than criticism. The speeches made by NBN Co chairman Ziggy Switkowski and Professor Henry Ergas at the Committee for Economic Development of Australia (CEDA) business luncheon held in Melbourne provided a valuable insight into the NBN’s new direction and raised a few eyebrows as well. In Business Spectator the Switkowski and Ergas talks are discussed along with why the CBA is nothing more than a flawed analysis.

Read the full article below

The recent Committee for Economic Development of Australia (CEDA) business luncheon held in Melbourne provided a venue for NBN Co chairman Ziggy Switkowski and Professor Henry Ergas to provide a valuable insight into the NBN’s new direction.

Ergas was a pivotal member of the Cost-Benefit Analysis and Review of Regulation panel led by Dr Michael Vertigan. Two of the three reports that the Vertigan panel was tasked to complete by the Coalition government last December have been released and the final report should be released soon, potentially after the Telstra and Optus agreement renegotiation is finalised.

This final report is eagerly anticipated by participants in the NBN debate because it will provide clear guidance on how the Coalition government intends to complete the multi-technology mix NBN. In addition it should reveal how it will adjust the regulatory and legislative environment to facilitate infrastructure competition in high-value urban areas while protecting NBN Co’s bottom line.

Unsurprisingly, the Vertigan panel reports have mirrored the support given to the Coalition’s NBN plan by the raft of NBN related reviews and audits carried out over the past year.

Ergas, a cost-benefit analysis expert, has been very vocal over the past couple of weeks about his disdain for governments that commence major projects without adequate prior analysis of a project's economic and social costs and benefits. Ergas is right to be concerned and his stance should be supported by the taxpayer.

But the question remains as to why the Vertigan panel utilised a European-based consultancy to develop a single demand and connection speed data set. There are many organisations and universities in Australia that could have provided four or five reasonable data sets for the cost-benefit analysis. 

There can be no doubt that the Vertigan panel’s cost-benefit analysis report highlights what can go wrong when politics is involved and how this affects the selection of panel members and consultants.

During his talk, Ergas said: “Where good policymaking is involved, almost always it is based on careful analysis”.

“Obviously when you’re looking at a project such as this [the NBN], the Sydney Harbor Bridge or the Snowy, you’re looking at projects that have very large impacts over very long periods of time. And to do that before the funds have been dispensed, you have to make assumptions about costs and you have to try to project what the benefits will be.

“So you have to make a range of assumptions and the results will invariably be sensitive to the assumptions you make, and that too has been a source of criticism of what we’ve done.”

Ergas highlights that a key benefit of a cost-benefit analysis is the identification and publication of assumptions necessary for the study to reach reasonable and justifiable outcomes.

OK, great motivation. But what Ergas said next was a surprise.

He says that the whole point of the cost-benefit analysis, from his perspective, was to build a “rigorous conceptual framework” for one base set of assumptions around future internet use in Australia.

He added that this could then be used to “test the sensitivity of the results to alternative assumptions”.

“If other people want to test other assumptions, then of course that is perfectly fine,” Ergas said.

But shouldn’t the average Australian taxpayer expect that the millions of dollars spent on the Vertigan panel would produce a rigorous study that included all of the reasonable alternative assumptions?

It is this point that has been at the centre of the criticism surrounding the cost-benefit analysis.

Ergas argues that a sensitivity analysis of the results based on alternative assumptions was carried out, and that this was adequate to reflect the range of possible outcomes. This basically means that any base assumption is varied from 10 per cent to 200 per cent to provide a range of outcomes to see how the variations affect the overall model results.

But a review of the cost-benefit analysis finds that the assumed values for demand, connection speed and other key inputs to technical model such as projected rates of change were low.

We don’t have the exact figures, but assuming the cost-benefit analysis concludes that Australian internet speeds will grow at 1 per cent per year over the next ten years.

This is what its forecast would look like. We’ve also included two lines to account for the sensitivity analysis.

For argument's sake, let’s assume that 1 per cent per annum is a modest measure of download speed growth. Here’s another graph -- without the sensitivity lines -- showing what download speed growth would look like if considered that it like if growth was projected at 10 per cent and 30 per cent per year.

The way Ergas explained the cost-benefit analysis, the data in the document will most likely represent the first graph, rather than the second, and this is a problem.

The point of the cost-benefit analysis to have a range of input values applied and to carry out sensitivity analysis on each set of assumptions made, not to use one set of input values ,and then to argue that by carrying out a sensitivity analysis on this set of input values the results are OK.

History is a good guide when it comes to predicting technology change and it remains uncertain why the Vertigan panel dismissed well known predictors for technology change, one of which put technology demand growth above 30 per cent.

So where do we go from here?

Well, nowhere really.

The cost-benefit analysis report includes a number of omissions, ad the conceptual framework cannot be reconstructed from what is provided. This makes it impossible to build an equivalent model so that alternate assumptions can be applied.

If Ergas believes that the whole point of the exercise was to develop a “rigorous conceptual framework”, then it is important that he ask Communications Minister Malcolm Turnbull to release the conceptual framework in its entirety in a form suitable for further analysis to be undertaken.

Mark Gregory is a Senior Lecturer in the School of Electrical and Computer Engineering at RMIT University