Introduction

This document briefly sets out some key features of how Quantitative Cost Risk Analysis (QCRA) has been undertaken on Crossrail. The main focus of this report is on Crossrail’s approach to QCRA which has a degree of originality in comparison to other major projects, particularly in relation to the Trend Management process
This document covers the Quantitative Cost Risk Assessment (QCRA) process used for regular cost reporting from 2009 to 2016. A modified version has been in place from 2017 as outlined in the managing risk in the later stages of Crossrail Learning  Legacy paper.

Context

The funding structure for Crossrail Ltd (CRL) includes the Anticipated Final CRL Direct Costs (AFCDC) which is defined as the P50 cost estimate (the cost estimate where there is 50% probability of delivering the project), delegated contingency funding up to the P80 cost estimate to CRL, contingency between P80 and P95 held by Transport for London (TfL), and Department of Transport (DfT) responsible beyond P95. Over time the range between the P50, P80 and P95 values have been reducing, as is to be expected on a project on the scale of Crossrail.
The Project Sponsors have a series of intervention points at which they can take action if the project is forecast to cost more than the P50 cost estimate. These clear intervention points have provided a focus on active programme and cost management whilst providing CRL with the freedom to manage risk.
The Project Development Agreement (PDA) which sets out the relationship and requirements between Crossrail Ltd and Sponsors has a number of requirements for cost reporting. As a result, CRL implemented a comprehensive process for QCRA throughout the organisation which was fully integrated within the cost management process.
Each quarter a QCRA is produced based on potential outcomes to what extent the risks impact. The QCRA output feeds into the Crossrail Investment Model (CIM) S-curve demonstrating the P50, P80 and P95 likelihood of exceeding the intervention points.

Figure 1 – Example S-curve

Constituent Parts of a Cost Forecast

The QCRA process in CRL defines the overall contingency that is required. The QCRA output is an assessment of uncertainty and risk as a ‘modelled risk forecast’. This forms part of the forecasting of total cost, known as Anticipated Final Cost (AFC), which intends to provide a realistic assessment of the outturn cost of the scope in question. The forecasting of cost is maintained independently of budget and/or authority.
Contingency is that part of the budget which is identified to manage risk and the impacts of risk. Prior to centralisation of risk contingency to the programme in 2017, the Anticipated Final Cost (AFC) was defined as having two distinct elements: Resolved Trend Forecast (RTF) plus Risk Forecast (this included all unresolved trends (URTs)), as shown in Figure 2. At this stage of the programme the AFCDC calculation does not include new URTs. New URTs only affect AFCDC when they have been reviewed, accepted and are resolved (as detailed in managing risk in the later stages of Crossrail).

Figure 2 –  AFC = Resolved Trend Forecast plus Risk Forecast

Tools Used

There are two principal data sources and one set of spreadsheet templates used to undertake the QCRA:

• PRISM – Cost Management Database. Both agreed and uncertain costs are captured. The uncertain costs are called unresolved trends.
• ARM (Active Risk Manager) – Risk Database. This consists of both threats and opportunities. The process requires that cost impacts, (positive and negative), transferred into the spreadsheet are based on post-mitigation values.
• The above information is fed into a series of Excel QCRA templates to be used with @RISK (a risk modelling software package).

The intention was that the Excel spreadsheets should carry out the complex calculations and ARM would be used to store the data and would remain the most up-to-date version of risk and mitigation data. However, in practice, risk data was frequently maintained from one quarter to the next in the Excel Spreadsheets, which then became out of sync with ARM data. Furthermore although the QCRA templates are clearly labelled “post-mitigation” for a variety of reasons, (see the recommendation section below), in reality, pre-mitigation entries were often input.

Quarterly Process

The QCRA is undertaken every quarter and feeds into the overall AFCDC. Although the AFCDC is calculated each period, the QCRA is only run each quarter. The following steps take place each quarter –

1. PRISM and ARM are reconciled. The resulting conclusion is set out in the Excel QCRA Template. There can be significant differences between the two systems for several reasons –
   I. Project wide risks are held in ARM, but not in PRISM
   II. A risk and its equivalent unresolved trend may be set out differently in PRISM and ARM. For instance there may be one risk impacting two different aspects of a project which is then shown as two PRISM entries.
   III. There may be different assumptions about the potential for mitigations to reduce the cost impact.
2. The conclusions from the above are set out in the spreadsheet as either –
   I. An Unresolved Trend (URT), a cost that has or will be incurred but the value is uncertain.
   II. A cost item that may or may not be required i.e. a risk.
3. For each project, risks are categorised as project, sector or programme. A project QCRA model is run using the QCRA template producing the risk exposure for each of those three categories.
4.  All risk exposure (project, sector and programme) is then combined with the assessment of Programme wide risk which feeds into the overall Crossrail Risk Model (CRM). Once the overall CRL programme risk exposure is calculated, this then feeds into the CIM. The CIM calculates the overall AFCDC and the likelihood of remaining within the intervention points. The following diagram illustrates the flow of risk data:

Figure 3 –  Crossrail Risk Model

Four Weekly Process

Cost forecasting is required every four weeks to fit with Crossrail’s reporting periods. Therefore, in-between the Quarterly assessment, a more limited review is carried out. The three principle types of calculation carried out at these reviews are:

• Resolution of URTs – As URTs move to a position of Resolved they move into the project AFC risk forecast.
• Revised assessment of an URT – In cases where the value of the URT has gone up or down the value of the risk forecast moves accordingly.
• Change Papers – Where change papers are approved and require a change to cost forecasts, the programme wide risks can be used to offset if there is an applicable risk.

Lessons Learned

A number of potential improvements are outlined below that could be used to improve the approach to QCRAs.

• The QCRA process enabled complex cost information to be reported reliably in the format required by Sponsors. But for the reasons stated below, this led to duplication of effort (updates to multiple systems) and time consuming use of resources. A more streamlined system could have been considered from the outset.
• The close working between teams. The Risk team worked closely with the Cost and Change teams and this led to a joined up approach to risk management and cost management. This was key to ensuring that all parts of the cost forecast were being managed inclusively. However the points below highlight some of the issues that also ensued.
• The QCRA consists of both trends and risks. The Cost and Change team are the experts for the former and the Risk team for the latter. Each team can add limited value to the other. Cost forecasting does require reconciliation between the two sets of data. However this does not require one overall modelling tool. It is recommended that separate processes and tools are used.
• Extensive reconciliation between QCRA sheets and ARM. By using a combination of tools (ARM and Excel) to undertake the QCRA it meant that updates were required in two places. This meant there were occasions when there were differences between the QCRA spreadsheets and ARM data. If the above recommendation concerning the separation of the Trend and Risk reporting processes was adopted then there would be a case for carrying out the risk analysis within ARM, which in turn would tend to drive improved ARM data freshness and quality.
• Post versus pre-mitigation values. The risk management procedure was designed to work on post-mitigation values. On occasion there have been circumstances when pre-mitigation values were used instead. The template did not include mitigation information, and hence, when carrying out reconciliation exercises there was often a tendency to over-look the potential for mitigation. The recommendation is that both pre and post-mitigation data should be included in the modelling, as well as the cost of mitigation, so the importance of mitigation can be examined as part of the reconciliation process. This would provide more focus on mitigation, assisting the overall risk process.
• The Four Weekly Process does not utilise updated risk data. It reviews movement of trends against risks identified in the quarterly QCRA. The above recommendations would simplify the QCRA process which would facilitate using fresh ARM data and risk modelling so it could be carried out every 4 weeks rather than quarterly.
• The programme level prolongation cost risk, (risks associated with delays), could have been informed by the Quantitative Schedule Risk Analysis (QSRA) carried out as part of the Semi Annual Construction Review (SACR). Each period the schedule could be used to inform of changes i.e. delays and subsequent costs associated (that may not have been captured by the QCRA). The QSRA for SACR could be used to inform much wider costs to the end of the programme. For example, where large mitigations have been put in place and would require additional cost, or where there has been significant slippage on the overall schedule.
• Simplification of QCRA processes. Within the above recommendations there are several that, if adopted on Crossrail, would have resulted in a simplified process. In addition, the concept of Programme wide risk being measured at lower as well as upper levels, is an obvious area for simplification. Complex processes are prone to communication issues, are difficult to update and change, tend to be difficult to provide training for, and can result in those undertaking them to use short-cuts. For all those reasons less complex cost QCRAs would be a better option.