Assessing Value for Money of the PPP

A key objective of governments in implementing PPPs in infrastructure is to achieve value for money (VFM). Value for money means achieving the optimal combination of benefits and costs in delivering services users want. Many PPP programs require an assessment of whether a PPP is likely to offer better value for the public than traditional public procurement—often called value for money analysis.

A VFM analysis can be done for a specific PPP project, and at a program level, for projects with common characteristics. For example, the United Kingdom Treasury's manual on assessing value for money (UK 2011b) described how value for money should be assessed at both the program and project levels (that methodology was later considered biased and recalled by government).

VFM analysis typically involves a combination of qualitative and quantitative approaches. Qualitative VFM analysis consists of sense-checking the rationale for using a PPP. This involves asking whether a proposed project is of a type likely to be suitable for private financing, and whether the conditions are in place for the PPP to achieve value for money—for example, that the PPP has been structured well, and that competitive tension is expected during the bidding process. This often takes place at a relatively early stage of PPP development—as such, qualitative VFM analysis may constitute part of the PPP screening described in Screening for PPP Potential.

Some PPP programs also require quantitative assessment of value for money. This typically involves comparing the chosen PPP option against a Public Sector Comparator (or PSC)—that is, what the project costs would look like if delivered through traditional procurement. This comparison can be made in different ways. The most common is to compare the fiscal cost under the two options—comparing the risk-adjusted cost to government of procuring the same project through traditional procurement, to the expected cost to government of the PPP (pre-procurement) or the actual PPP bids (post-procurement). An alternative is to compare the two options with an economic cost-benefit basis—that is, to quantitatively weigh the expected benefits of a PPP over traditional procurement against its additional costs.

Value for money analysis—particularly using quantitative public sector comparator methodologies—has been widely debated. Some question the value and relevance of a PSC approach, which can appear to be more scientific than is the case, potentially misleading decision-makers; or conversely, may simply come too late in the process to be a genuine input to decision-making. A World Bank report on Value for Money (WB 2013a) analysis presents evidence on practices from several countries, and on trends regarding the scope of value for money analysis and the relative advantages of quantitative and qualitative approaches.

For more discussion on approaches to assessing value for money, and their relative advantages and disadvantages, see also:

• Farquharson et al's section on selecting projects (Farquharson et al. 2011, 41–43), which briefly describes value for money and cost benefit analysis, and considers the value of qualitative versus quantitative approaches.
• Grimsey and Lewis's article on PPPs and Value for Money (Grimsey and Lewis 2005, 347–351) includes a section on approaches to value for money describing examples of different countries' approaches.
• The OECD's publication on PPPs (OECD 2008a, 71–72), which also describes the range of methods used by different countries, on a spectrum of complexity, from simply relying on competition, to full cost-benefit analysis of different procurement options.
• The PPIAF Toolkit for PPP in Roads and Highways has a section on value for money and the PSC (WB 2009a), which describes the logic behind value for money analysis, and how the PSC is used.
• The European PPP Expertise Centre (EPEC) value for money assessment report (EPEC 2015) outlines and compares value for money assessment methodologies in several European countries.

The remainder of this section briefly describes and provides further resources for readers on qualitative and quantitative value for money assessment methodologies.

Qualitative value for money assessment

Qualitative VFM analysis involves sense-checking the rationale for using PPP as a delivery mechanism—that is, asking whether a proposed project is of a type likely to be suitable for private financing; as well as whether the conditions that are necessary to achieve value for money are in place, as described in Farquharson et al, (Farquharson et al. 2011, 42–43). This often takes place at a relatively early stage of PPP development—as such, qualitative VFM analysis may overlap with the PPP Screening process described in Screening for PPP Potential above—but may be repeated throughout the project development process.

Some jurisdictions have clearly-defined criteria for this analysis. For example:

• The UK Treasury has defined criteria for assessing suitability, and unsuitability, for a Private Finance Initiative (PFI)—the UK’s availability payment PPP model. Suitability criteria include the long-term, predictable need for the service; the ability to allocate risk effectively—including through performance-related payments and ensuring sufficient private capital at risk; the likely ability of the private sector party to manage risk and take responsibility for delivery; presence of stable and adequate policy and institutions; and a competitive bidding market. Unsuitability criteria include projects that are either too small or too complicated; sectors where needs are likely to change or there is a risk of obsolescence (for example, PFI projects are no longer used in the ICT sector in the UK); or where the contracting authority is inadequately skilled to manage PPP (WB 2013a).
• In France, preliminary analysis of a PPP includes checking against several criteria under three categories: PPP relevance—for example, appropriateness of an integrated, whole-of-life approach to managing a project; commercial attractiveness; and the potential for optimal risk allocation (WB 2013a).
• In the Commonwealth of Virginia, United States, assessment of a potential PPP at high level and detailed screening stages also considers proposed road projects against specific criteria to determine if the project is delivered under the Public-Private Transportation Act (PPTA)—that is, as a PPP. These criteria include whether a project is sufficiently complex to benefit from private sector innovation; whether a PPP can achieve appropriate risk transfer; and the degree of stakeholder support. The extent to which a project can generate revenues from tolls is also taken into consideration when assessing possible PPP structures (WB 2013a).
• The Caribbean PPP Toolkit (Caribbean 2017, Module 4, Section 8) presents Jamaica’s methodology for assessing value for money, and other globally-relevant guidance.

The EPEC Guide to Guidance also includes a list of key conditions that should be met to have a higher probability of achieving value for money (EPEC 2011b, Chapter 1.2.4).

Public Sector Comparator—comparing fiscal cost

The most common quantitative tool for value for money assessment of a PPP project is derived from the approach originally used in the United Kingdom's PFI program in the early 1990s as described in Leigland and Shugart's Gridlines article on the PSC (Leigland and Shugart 2006). It involves comparing the fiscal cost of a PPP delivery option with that of a conventional public delivery option—not a single conventional option, but a range of infrastructure options as noted in the 2011 Treasury Guidance on Valuing Infrastructure Spend (UK 2015a). NAO evidence presented in the House of Commons 2014 report (UK 2014a) discusses several shortcomings in the identification of PSCs.

The focus of the Fiscal Cost approach to value for money analysis is the construction of a PSC—the cost to government of implementing the project through traditional public procurement. Calculating the PSC can be complicated, as several adjustments are needed to ensure a fair comparison. How the Public Sector Comparator is calculated, highlights some methodological debates.

This type of PSC can be used at two stages of the procurement process, as described in the OECD book's chapter on the economics of PPPs (OECD 2008a, 71–72). These are:

• Before the bidding process—the PSC can be compared with a shadow or reference PPP, or market comparator—a model of the expected cost of the project under the PPP option. This can help identify whether the PPP can be expected to provide value for money, before deciding to go ahead with detailed preparation and procurement. The reference PPP model would be the same as the financial model described in Assessing Commercial Viability.
• During the bidding process—the PSC can also be compared with actual PPP bids received, to assess whether the bids provide value for money. This approach is used in Australia, and is described in a PSC Technical Note (AU 2016a).

Despite the appealing logic of the concept, there have been many criticisms of the usefulness of the PSC and fiscal cost comparison approach in countries where it has been used frequently, such as the United Kingdom and Australia. A United Kingdom House of Lords' review of the PPP program (NAO 2013a), for example, argued that shortage of relevant data and methodological issues limit the value of the PSC. The government's response to the review agrees that the PSC provides only a partial picture, and highlights that its use is balanced with qualitative analysis, as described above.

Leigland and Shugart's Gridlines article on the PSC (Leigland and Shugart 2006, 2–3) summarizes these criticisms, which include the inevitable inaccuracy of estimates over a long-term project, lack of consensus on methodology, and so the possibility of manipulation to reach the desired conclusion. Grimsey and Lewis (Grimsey and Lewis 2005, 362–371) describe some of these criticisms in more detail. Given these challenges, Leigland and Shugart's Gridlines article (Leigland and Shugart 2006, 3–4) also discusses whether and how the PSC approach could make sense in a developing country context.

Economic cost-benefit comparison of PPP and public procurement

One of the criticisms sometimes leveled at the PSC is that it focuses solely on the financial cost to government of PPP or traditional procurement. A more comprehensive approach would also consider the differences in expected benefits, and compare the net economic benefit under PPP or under public procurement. On the other hand, as Grimsey and Lewis note (Grimsey and Lewis 2004, 353), this adds further complexity to the value for money analysis over the PSC approach, and could risk making the results even more subjective.

For example, the EPEC's note on non-financial benefits of PPP (EPEC 2011c) suggests how some of the benefits of PPP—as described in Infrastructure Challenges and How PPPs Can Help—could be quantified, and added to a more typical PSC analysis.

Few countries have introduced this kind of analysis in practice. New Zealand's new PPP program is an exception. Cost-benefit analysis is the main tool for assessing procurement options. New Zealand's PPP guidance material (NZ 2016, 6–12) asks practitioners to identify the possible benefits of PPP over traditional public procurement and where possible to assign dollar values to each benefit.

In many developing countries' PPP programs, the aim is not just to reduce cost, but to transform service delivery. For example, governments hope that roads will be better maintained, thus delivering additional trade and economic benefits. These changes in service levels and quality cannot be captured by comparing fiscal costs of PPP and public procurement. Where these expected benefits are deemed important, and quantitative value for money analysis is desired, economic cost-benefit analysis may be the better approach.