Introduction

During the passage of the Crossrail hybrid bill through Parliament, the Crossrail Environmental Minimum Requirements (EMR) were negotiated with various stakeholders including local authorities and statutory bodies.  The requirements in the EMR were set at current industry best practice in 2008.  Crossrail also set an aspirational environmental vision of ‘Setting the standard for environmental performance and leaving the legacy of a sustainable railway’ with the aim of encapsulating the aims and direction of environmental management within the Crossrail organisation.

Crossrail needed a means by which it could be assured that its Tier 1 contractors (hereafter referred to as contractors), were complying with the EMR and performing to a level that allowed the vision to be achieved.  Crossrail developed an Environmental Management System (EMS) to deliver that assurance, including mechanisms such as audits, site inspections and processes for reporting and investigating environmental incidents.  Contractors were also required to report every 4 weeks on a suite of environmental key performance indicators (including waste, energy, incidents, complaints, water use and recycled content of new construction material) and this information was collated into dashboard reports.  Contractors were also encouraged to share information and lessons learned using an internet based collaboration site and at regular fora.

Despite these various mechanisms, in the early stages there was an initial lack of visibility at programme level of each contractor’s overall environmental performance against all of the relevant environmental requirements, and in particular those requirements that were not covered by the regular environmental key performance indicator reporting.  The development of the Crossrail performance assurance process in early 2012, following consultation with key members of the programme and Crossrail’s contractors, provided an opportunity to obtain greater transparency of contractor overall environmental performance.  It complemented the other assurance mechanisms by providing a structured framework which enabled the objective assessment of the contractors.  It provided visibility of the performance of each contractor and it enabled benchmarking of their performance relative to one another.  Although the performance assurance process was developed initially for commercial assurance, it soon became apparent that it could also provide assurance for other functions.

Performance Assurance covered 6 core areas in addition to environmental performance (commercial, quality, health and safety, community relations and social sustainability).  Performance Assurance assessments commenced in 2012 and subsequent iterations were undertaken thereafter every 6 months so that progress made by the contractors could be tracked.  In addition to complementing the other EMS assurance mechanisms, performance assurance also operated alongside Crossrail’s reward and recognition schemes, namely the Green Line Recognition Scheme for promoting positive environmental behaviour change and the Annual Sustainability Awards.

Further information about the performance assurance process is provided in the paper on the performance assurance framework contained on the Crossrail learning legacy website.

This paper describes the approach taken to conduct environmental performance assurance, the framework that was used to measure performance and the process for undertaking the assessment.  It also sets out the benefits that have resulted from implementing the process.

Setting Objective Performance Measures for Environment

In common with each of the other core areas of supplier performance, a detailed suite of objective environmental performance measures were defined covering two aspects:

• Inputs: These were indicators that measured the contractor’s approach to delivering environmental management; and the quality of their systems and processes. They assessed qualitative information such as audit schedules, training matrices, and evidence of proactive planning for consents.
• Outputs:  These were indicators that directly measured environmental performance. These were usually quantitative in nature and were collated from data submitted through the usual reporting process, for example percentage reduction in energy use.

Across these two aspects, performance was measured on a scale of 0 to 3:

0 = Non Compliant performance – not meeting the obligations under the contract (known as the Works Information or WI)

1 = Compliant performance – meeting the obligations of the WI

2 = Value Added performance – working beyond the obligations of the WI

3 = World-Class performance – working significantly beyond the obligations of the WI.

The objective performance measures were set out in an environmental performance assurance framework.

Development of the Environment Performance Assurance Framework

The Environment Performance Assurance Framework is provided as a supporting document.  It sets out the full range of measures that were used during environmental performance assurance of the civil construction phase of the project, including tunnelling works and the construction of the portal, shaft and station structures between 2012 and 2015.  It does not include measures specifically aimed at assuring the fit-out of the station and shaft structures or the installation of rail systems.

Level 1 of the framework (compliant performance) drew heavily on the Crossrail Works Information (WI) which set the benchmark of compliance and was environment industry best practice.  Discussions were held with organisations such as Constructing Excellence, Network Rail and Crossrail’s contractors.  These organisations provided information on the current environment practice in the industry which helped to develop definitions for the “value added” and “world class” sections of the framework.  Crossrail’s contractors were consulted on the draft framework.   The framework included 16 inputs and 8 outputs.

Not all of the environmental performance measures were used continuously during all of the iterations of supplier performance.  The Performance Assurance Framework presented with this paper represents the culmination of all the measures that were developed during the life of the civil construction phase of the project.  Anyone applying it to projects in the future will need to select measures that are appropriate to the stage and nature of the project.

Early phases focused on basic compliance and as the project matured the focus shifted towards ‘value added’ and ‘world class’ performance

At Crossrail, during the “early” phase of the performance assurance process, the aim was to obtain assurance that Crossrail’s contractors were compliant with the environment requirements of the WI and hence there was a focus on measuring compliance across a wide range of environmental performance measures.  All of the measures listed in the framework were included to ensure that the assessment provided a comprehensive and robust mechanism to check compliance.

As contractors’ knowledge of the process developed and performance improved, the assessment moved into the “mature” phase where the focus shifted from achieving compliant performance and towards ‘value added’ and ’world class’ performance.   The assessments started to focus on certain measures in particular whilst others, that had once been rightly the target of most effort, could be taken as met and therefore did not require further consideration. As a result, measures such as setting objectives and targets, status of environment management plans, status of the environmental management system and provision of suitable and adequate environmental staff resources were not selected for assessment during this phase.  Measures such as the control of construction noise and vibration (which was a significant issue at Crossrail) were developed further.  The noise management measure was segregated into areas such as leadership, engagement with community relations and the local authority, section 61 consent applications and monitoring techniques. The benefits of disaggregating the noise criteria allowed for clearer visibility of areas performing well and other areas requiring improvement.

During the “late” phase of the assurance process, major civil construction work was either reducing for the station contracts, or coming to a close for the tunnelling contracts and the number of measures were streamlined to reflect the stage of the project and the nature of the work being undertaken.   For example, during this phase most of the contracts had completed excavation works and had procured the majority of their construction materials.  Measures such as performance against excavated material reuse targets and undertaking sustainable procurement of materials were not selected for assessment because these items were no longer applicable across multiple contracts.  Similarly, although performance with ecology and contaminated land management were still included at this stage, these items were applicable to a minimal number of contracts.

Performance measures should be clearly based on the requirements of the contract.

A key lesson learned was that compliance performance measures should be clearly based on the requirements set out in the contract.  This provided a robust basis for undertaking the assessment.  Another lesson was that performance measures should be equally applicable to all contracts otherwise comparison between contracts was inconclusive. For example, not all Crossrail contracts were required to meet CEEQUAL and BREEAM targets. Therefore, inclusion of measures associated with meeting these targets were not always applicable and resulted in the need to adjust the overall performance scoring so that those that did not have them could be compared with those that did.

Another lesson related to the need to develop of a balanced set of meaningful input and output measures.  Input measures (or leading indicators) help to identify future issues and provide a forward-looking picture that is not identified by only recording traditional retrospective key performance indicators. For the Crossrail project, it was possible to develop input measures for all of the environmental topic areas, but in some cases these measured the presence of certain documentation (e.g. the preparation of plans) and were not able to provide information on the contractor’s implementation of the plans.  Some of the topic areas (such as noise management) were more developed in relation to inputs and were more reflective of the performance of the contractor.

It was not possible to provide meaningful output measures for all of the different environmental topic areas.  This was because even if a process (i.e. an input) is robust, an output does not always directly relate to that process. For example, noise output levels, or the number of complaints received about noise were not always directly related to performance in noise management (e.g. production of management plans, training of personnel, communications to third parties or the use of noise mitigation measures) because other factors such as the nature and timing of the works and the sensitivity of the local community had significant influence.  Also, the framework did not use environmental incidents as an output measure because of the potential that to do so could inadvertently result in a culture of not reporting incidents for fear it would adversely affect the performance score.  As a result of these two issues, only a limited number of output measures were identified, and this tended to bias performance results towards those that had them i.e. contractors that performed well in those few topic areas were shown as better performing overall.

It is recommended that future projects consider the balance of input and output measures and how these can be further developed to provide a more balanced measure of performance.  One option would be to develop a full set of input measures that reflect actual performance (and avoid the use of output measures at all).  If, however it is desired that output measures are included, then it is recommended that future projects focus on development of a suitable suite of such measures.

Regardless of which measures are used, it is recommended that a collaborative approach to their development is adopted.  The involvement of contractors in the development of the framework, and the regular feedback sessions as the framework developed, helped to reduce the number of challenges that contractors made to their scores.

The Assessment Process

Each iteration of supplier performance review was undertaken within a defined “review period” which was a window of time of four Periods (a Period being defined as four weeks) equating to roughly 4 months, during which all the contracts were assessed.  Each function followed a slightly different process that was tailored to the measures being assessed, but collaboration between functions ensured that the overall approach taken was consistent and that contractors were not overburdened by the process.

The process that was used for conducting Environmental Performance Assurance was undertaken in 9 steps:

1. Framework Distribution
2. Verification visits including feedback to project team and site visits
3. Distribution of draft reports
4. Collation of quantitative data
5. Moderation sessions
6. Distribution of the final reports
7. Published Reports
8. Feedback Sessions
9. Contractor Action Plans

The main personnel involved in the assessment process were:

Client Performance Assurance Team – This team was responsible for the overall performance assurance process (covering all six functions) including the overall strategy and programme and for collating the reports from each of the functional areas into a performance assurance report for each contract.

Client Environment Team – The client environment team took the lead on the environmental assurance for the project, primarily through the development and implementation of the EMS (in particular via auditing and reporting processes).  The team was responsible for developing and distributing the environmental performance assurance framework and running the environmental performance assurance assessments, specifically carrying out verification visits, writing and distributing reports, collating quantitative data, running moderation sessions and seeking and collating feedback.

Client Environmental Advisors – The client environmental advisors provided environmental assurance of contracts on a day to day basis and worked directly with the contractors to achieve this.  Their main involvement in the assessment process was in the moderation sessions (to ensure that the assessment was a fair reflection of contractor performance) and working with the contractors to implement contractor environmental improvement action plans.

Client Environmental Specialists (e.g. noise specialist, air quality specialist) – The client specialists provided technical advice to the client team, including the review of technical deliverables from the contractors.  Their main involvement in the assessment process was the provision of technical advice to the client environment team as part of the verification visits.

Contractor Environmental Managers – The contractor environmental managers provided information during the verification visits, commented on the draft environmental performance assessment reports, cascaded results within their contract and developed and implemented contractor environmental improvement action plans.

Client Project Managers and Contractor Project Managers – The project managers in both client and contractor teams were invited to attend the feedback session after the verification visit.  Their support of both verification visits and the environmental improvement action plans was encouraged.

The main steps in the process are described in more detail below.

Step 1 Framework Distribution

The framework was distributed via email to all contractors.  Briefings on the framework content were also held (e.g. through fora such as the Environment Managers Forum, a bi-monthly meeting of Crossrail and the contractors’ environmental managers).

Step 2 Verification visits including feedback to project team and site visits

Verification visit dates were scheduled approximately 2 months in advance of the visit and undertaken within the review period. Early scheduling of verification visits was vital as this provided sufficient notice which reduced the likelihood of cancellation and unpreparedness.

The verification visit was undertaken by a member of the client environment team and consisted of a site walkover and a review of documentation. Qualitative measures were focused on during this review such as processes addressing audits and inspections, training and awareness, waste, air quality and water management (please refer to the framework).

The visits varied in duration dependent upon the size and complexity of the site, but typically a day for large contracts and half a day for smaller contracts. Throughout the visit, the contractor was updated on its scores and those areas where there was capacity for improvement.  A feedback session with the client team responsible for managing the contractor was held at the end of the visit.

Where contracts were operating across multiple sites, it was not possible to undertake a verification visit to all of the sites because of limited resources.  In such cases, verification visits were undertaken at either one or two sites to provide a representative sample of the contract.  Over the following iterations, the sites visited would be varied so that all the sites were eventually covered.

At this stage of the process, technical input from Client Environmental Specialists was incorporated.

Step 3 Distribution of draft reports

A draft report was completed after each of the visits and presented to the contractor and client environmental advisors for comment.  This was issued as soon as possible after the visits and usually within 48 hours.  The draft report was usually incomplete as quantitative results would not have been available at this stage (the quantitative data was collated at the end of the review period).  Outstanding evidence that was required before the end of the review period was also identified in the draft report.

Step 4 Collation of quantitative data

Quantitative data was collated at the end of the review period and was taken directly from databases and other reporting information held by Crossrail.  Any inconsistencies with data were followed up with the contractor prior to the final report distribution.  Examples of quantitative environment data described in the framework include:

• the number of audits and inspections
• performance in meeting targets for construction and demolition waste and excavated material
• use of timber from sustainable sources
• recycled content of construction materials
• incident reporting
• energy models

Step 5 Moderation sessions

Moderation sessions were conducted after each round of supplier performance with client environmental advisors.  This was to ensure that the scores were a fair representation of the contractor’s performance.  A session was also held with the Crossrail Environment Manager to ensure that a consistent approach was taken to the scoring across all the contractors assessed.

Step 6 Distribution of the final reports

Once the evidence and moderation sessions were completed, the final reports were distributed to the Performance Assurance team and the environmental results were compiled into an overall report (together with results from the other core areas) and distributed to the client team who have responsibilities for managing the contractors.  A variety of different mechanisms were used to disseminate the results, for example at fora for project managers and environmental managers. An example of a final report is provided as a supporting document.

Step 7 Published Results

The environment results were totalled and an average score per contract plotted on a graph as demonstrated in Figure 1. Each contractor knew its position on the graph, however, the other contactors’ results remained anonymous.  An example of this graph is shown in Figure 2.

Figure 1 – Environmental performance scores calculated

Figure 2 – Distribution of contractor results

Step 8 Feedback Sessions

After each iteration of supplier performance assessment a feedback workshop was coordinated with the Crossrail contractors to gather thoughts on the process and the framework to ensure it remained suitable and robust. This was usually held about 2 months after the results were published and in advance of the next iteration.  This helped to maintain motivation and contractor support for the process, which reduced the number of comments on scores and queries when the final reports were submitted.

Step 9 Contractor Action Plans

After each iteration of the supplier performance assessment, environmental improvement action plans were produced. These were developed by each contractor in response to the information in the final reports.  The contractors were responsible for producing these plans so that they had ownership of the actions and the programme for their completion.  They identified actions to improve environmental performance and encouraged a process of continual improvement.  Progress towards meeting actions was followed up during subsequent performance assurance iterations.

Resource Requirements for Performance Assurance

A total of six supplier performance iterations were undertaken during the civil construction phase of the project.  This case study reports the results from five of the iterations that were available at the time of writing.  Over the duration of Performance Assurance on Crossrail between 12 and 16 contracts were assessed during each of the iterations.  It took approximately two days per contract to complete each review which included verification visits, collection of data, report writing and moderation sessions.

The assessments were undertaken by staff within the client environment team who did not have day to day contact with the contracts.  This helped to ensure an independent assessment.

During the initial stages of performance assurance, the process was conducted by one assessor and then as the project matured by two assessors. The advantage of using one assessor was consistency in the scoring.  The benefit of using two assessors was that a greater breadth of experience was available which helped to make the scoring more robust and also reduced the potential for subjectivity, thereby providing greater surety of a balanced score.  When conducting the performance assurance process, the size of the project and time undertaken to complete each iteration had to be considered when establishing the resources necessary to carry out the task.

A lesson learned during the process was that assessors who conduct the supplier performance process should be competent, qualified and experienced in environmental management.  In addition a lead auditor qualification or experience in conducting environmental audits was found to be beneficial during the verification visits.

Benefits of conducting the Performance Assurance Process 

The use of the Performance Assurance process was successful in increasing environmental performance across the contractors.  The average score achieved by the contractors improved by 42% between 2012 and 2015.  This comprised an increase in the average output score of 47% and an increase in the average input score of 37%. Figure 3 above demonstrates how environment performance evolved over 5 iterations as represented by the yellow triangles. The yellow triangles are the average score of all the contracts assessed. The large improvement in output scores between iterations 2 and 3 relates to an improvement in the matrix that was developed for iteration 3 which included a greater number of output measures and which allowed contracts to improve their output scores.

Figure 3 – Average Score from Round 1 to Round 5

Figure 4 demonstrates that in 2012, 78% measures were assessed to be ‘compliant’ zone, with 14% non-compliant and 8% of measures in the ‘value added’ or ‘world class’ zone.  By 2015, 55% measures were assessed to be either value added or world class.  Non-compliant measures had reduced to 4%.

Figure 4 – Environmental Performance 2012-2015 

The value of the supplier performance process was that it provided clear visibility of Crossrail’s contractor performance as measured objectively across the programme.  This visibility enabled action to be taken where needed to address areas of non-compliance.  Although there were only very few instances of non-compliance, the process enabled visibility of these and rectification could be actioned swiftly (without waiting for the reporting process to be complete).

The process of visiting sites and collecting information also aided the identification of best practice/innovations in environment management and thus sharing of this best practice across the programme.  It also helped to identify areas that were weaker and would benefit from greater support or awareness raising.  The process itself provided a driver for contractors to perform better as a result of competitiveness to become the “top performing contract”.

One of the lessons learned was that the supplier performance process was developed in isolation from the Green Line Recognition Scheme which aimed to promote positive environmental behaviour change.  With hindsight; there could have been more synergy between the two schemes rather than operating separately.  For example, the criteria in the Green Line Scheme could have been included as criteria in the environmental supplier performance matrix.  For each of the criteria, different levels of performance could have been clearly defined (for example compliance performance, value added performance and world class performance as defined by supplier performance).

Crossrail is the first major infrastructure project to conduct environmental performance assurance of its contractors in this level of detail. Through the development of the process, Crossrail has provided the content and process for other major projects to adapt and take forward with the aim of increasing environment performance across the construction industry. For example, the process has been shared with Network Rail who has since rolled out the process to its supply chain on the Crossrail project (when it is carrying out works on Crossrail’s behalf).