Introduction

The Crossrail project has generated in the region of 15 million pieces of highly complex documented information in support of planning, designing and constructing the railway.  It was therefore critical that Crossrail established and maintained a well structured and rigorously controlled information environment.

Failure to effectively control and manage data risks would have lead to wasteful overlaps, duplications and omissions – thereby obstructing information sharing and reporting.  This would in turn have hindered and delayed project completion with additional costs and reputational impact.

This case study reviews the role of the Information Compliance team within the Crossrail Technical Directorate; in ensuring quality of information and that the right people were able to retrieve the correct information throughout the project lifecycle.

The paper reviews the various assurance methods used to measure, analyse and communicate document compliance and use of dynamic Key Performance Indicators to improve performance and manage document delivery risks.

The Information Environment

The Document Data and Compliance (DDC) team was collectively responsible for assuring all documented technical information collected by Crossrail from 2008 (the approval of the Crossrail Act) to 2018 (handover to Operations).  This encompasses all facets of specifications, geographical surveys and geotechnics, consents, land management, drawings and computer modelling, construction works (including enabling), asset labelling and management, interface operations, assurances (quality / compliance, environment and health & safety), operations and maintenance, testing and commissioning, contract administration, reporting and financial management.

Complexity came through the need to manage the multiple sources of project data, i.e. corporate, contractors, subcontractors and third party suppliers.

The Document Management System

Crossrail’s Electronic Document Management System (EDMS) was a secure data environment established to handle the enormous amount of information generated by Crossrail.  This was industry-standard information software offering centralised, structured and integrated data/content management (a “single source of truth”) alongside interoperation and collaboration functionality.

The  EDMS used was initially Documentum (developed and supported by EMC Inc.) then later eB (‘Enterprise Bridge web’, developed and supported by Bentley Systems Inc.).  The EDMS managed the document lifecycle stages of the programme and facilitated:

• Effective capture, storage, control and delivery of all formally issued technical information required throughout the lifecycle of the Crossrail project.
• That the right people were able to retrieve the correct information throughout the project lifecycle.
• High fluidity processes – capable of adapting to the ever changing information requirements of the project.
• Integration of key processes (e.g. contract administration and asset inventory management).
• Robust reporting.
• Reliable Quality Assurance – i.e. Compliance monitoring of transaction histories

The document lifecyle stages baselined in the EDMS are:

• Creation, Origin/Ownership
• Review and Approval
• Versioning
• Legibility and accuracy
• Controls for identification
• Document- and object-relationships (via metadata)
• Storage (security and confidentiality)
• Accessibility (retrieval and searchability via metadata)
• Distribution (workflows, permissions, communications)
• Archiving and Retention
• Disposition of Records

Document Compliance Baseline

The Information Compliance strategy was dictated by the legal and corporate governance requirements of the Crossrail programme and underpinned by requirements of the international quality standard BS EN ISO 9001.  Compliance for contractors, subcontractors and other third party suppliers meant working in accordance with established Crossrail processes, specifications and guidelines for document management and control.

In capturing document lifecycles, the Crossrail EDMS baselined a process and responsibilities landscape against which compliance could be tested and measured – and fed into the compliance principles of ‘Measurement, Analysis and Communication’ (Figure 1).

Figure 1 – Information Compliance principles: ‘Measurement, Analysis and Communication’

The above described principles gave way to the four primary undertakings of the Compliance team:

1. Performance Assurance reviews of primary Tier 1 contractors.
2. The Employer’s Completion Process (ECP) – providing validation data for contract close out – as assurance to the Information Managers.
3. Contract Reporting – generation of Key Performance Indicator (KPI) metrics which fed into programme dashboards and document delivery reports to support the ECP (point 2) and Handover Readiness (see  Information Handover Principles learning legacy paper). The output of these reports were communicated to contractors and programme-wide – with the aim of not only correcting issues, but actively implementing preventive actions to drive continual business improvement.
4. Root Cause Analysis. This was a standard problem-solving tool; to aid in the creation of practical solutions (mostly process improvements), to prevent recurrence of persistent problems.

These undertakings are discussed fully in the following sections.

Performance Assurance reviews of primary Tier 1 contractors

Performance Assurance reviews of Tier 1 contractors were undertaken as twice-annual ‘rounds’ – in line with the wider Crossrail Performance Assurance Framework (PAF), which is the subject of another learning legacy paper.

The PAF was led by the Crossrail Performance Assurance team – comprising on-site contractor assessments of Community Relations, Social Sustainability, Health and Safety, Quality, Commercial and Environment Management.

Information Compliance fell under the Quality section of the PAF – and focused on the following Information Handover elements:

1. Deliverable documents – as per scope of contract and contractual agreement (i.e. civils, systemwide)
2. Drawings register – Register and Issue Record (RIR)
3. Materials register and materials compliance – Materials Proposal Schedule (MPS)
4. Document Control (Contractor’s Working Records):
   • Accuracy (data that is complete, accurate and fit-for-purpose).
   • Accessibility (the right people are able to retrieve the right information throughout the project’s lifecycle).
   • Searchability (indexing/packaging and formatting).
   • Quality (creation, archiving, retention, disposition, security, physical storage, internal controls according to risk).
5. Competence, Training and Awareness (documented evidence of competence on the basis of appropriate education, training or experience).

Assessment findings were logged on a questionnaire template (a typical audit checklist) during assessment, drafted onto a report template and verbally debriefed to the contractor afterwards

Non-conformances and suggestions for performance improvement were highlighted to the contractor in the form of a set of improvement actions.  These were agreed with the expectation that actions would be incorporated into the contractor’s Quality Improvement Plan, and resolved by commencement of the following assessment window.

Contracts were scored qualitatively and quantitatively against each of the above five Handover-critical elements, finally feeding into a combined score for the Quality section of the PAF.

The contractor review process is detailed more fully in the Crossrail Commercial Assurance Procedure and the Crossrail Technical Information Compliance Procedure.

Key Performance Indicators from Compliance Reviews

Actions from the compliance reviews were managed through a central tracker and proactively followed up with the contractor until resolution.  The tracker allowed for helpful trending analysis, which drove the above discussed business improvements.

The charts in Figures 2 and 3 illustrate trending data from Round 5 (R5) -year 2015- and Round 6 (R6) -year 2016- of the PAF, where lower ‘improvement actions’ percentage figures correspond to better compliance.  The data is anonymised to protect contractor (con) identity.  Of 299 improvement actions raised across both PAF assessment rounds 169 were raised at R5, versus 130 at R6. 23% fewer improvement actions within 12 months.

There were two areas of transitory decline in compliance at R6 – namely, the MPS and RIR classes at R6 (26% and 14% respectively) compared to R5 (12% and 8% respectively). However, the overall combined figures for all classes of action at R5 and R6 showed significant improvement in contractor compliance and this sufficiently validated the Crossrail compliance programme motivating further contractor improvements.

Figure 2 – Sample trending data from two rounds of Compliance reviews of primary Tier 1 contractors

(Contractor comparison)

Figure 3 – Sample trending data from two rounds of Compliance reviews of primary Tier 1 contractors

(comparison of Class of Actions)

The Employer’s Completion Process (ECP)

In line with Crossrail’s procedure for closing out the contracts, an Employer’s Completion Process (ECP) form was required to demonstrate that Crossrail had exercised due diligence prior to acceptance of the Works.  These were certified by the Delivery Team and Contractors as being complete and compliant with the requirements of the Programme, Works Information and relevant third party requirements.  The ECP also supported the formal issue of contract defects / outstanding works lists.

The ECP comprised Contract Closeout Checklists covering each of the business groups including technical Information as shown in Table 1.  However, it is important to note that the commercial close out of the Contracts, including any settlements, final accounts, reconciliation of Crossrail financial systems (SAP, PRISM etc.) was independent of the ECP and covered separately by Crossrail corporate (aligned to corporate – not contractor assurance).

Requirements for Contract Closeout Checklist review and sign off by FLs (Functional Leads)
Contract Closeout Checklist	Function	Functional Lead	Completion of the whole of the Works	Sectional Completion
C1	Health & Safety	Sector Health & Safety Manager	Required	Required
C2	Technical Compliance	Chief Engineer supported by HoDs	Required	At discretion of Functional Lead
C3	Planning, Heritage and Highways	Head of Sustainability & Consents	Required	At discretion of Functional Lead
C4	Environment	Head of Sustainability & Consents	Required	At discretion of Functional Lead
C5	Undertakings & Assurances	Head of Sustainability & Consents	Required	At discretion of Functional Lead
C6	Land and Property	Head of Estates	Required	At discretion of Functional Lead
C7	Rail Interfaces	Rail Interface Manager	Required	At discretion of Functional Lead
C8	Systemwide	Systemwide Director	Required	At discretion of Functional Lead
C9	Technical Information	Head of Technical Information	Required	At discretion of Functional Lead
C10	Third Party Agreements	Head of Agreements	Required	At discretion of Functional Lead
C20	Completion	Completion Manager	Required	Required
C30	Authorisation to Accept WCCC or SCCC	Completion Manager	Required	Required

Table 1 – ECP Contract Closeout Checklists

The ECP commenced 180 days prior to contract completion (T-180 days).  However challenges observed during the first few ECPs, highlighted the need for contract intervention far earlier.  At T-180 days contracts had largely reduced their information management and document control resource in readiness to vacate their worksite.  Consequently, many document delivery issues would surface, such as un-submitted or rejected deliverable files which had not been reviewed and were therefore not fit for purpose.  These would pose a major concern and risk to Handover if it affected critical deliverable records for certification, operation and maintenance, health & safety or environment.  Hence an additional review point was added earlier at T-270 days.

Review Points prior to contract completion were implemented to measure document delivery progress against the Contractor’s master deliverable documents and schedule; with status reported as below:

• Documents delivered by Contractor and accepted by Crossrail (either code 1 ‘accepted’; or code 4 ‘received for information only’).
• Documents delivered by Contractor but rejected by Crossrail (not accepted – either code 2 ‘Work may proceed subject to incorporation of changes’ or code 3 ‘Work may not proceed’).
• Documents not delivered by Contractor to Crossrail (uncoded or unassigned).

Delivery team Project Information Controllers (PICs) were enlisted to collaborate bi-weekly through information sharing of their own site-surveillance activities and findings, feeding back their actions in line with any issues identified by the DDC team.  A shared reporting tool was developed within the DDC team SharePoint site which also served as a document library for supporting documents and ongoing correspondence between the DDC and Delivery team PICs. Information shared was used to resolve issues and progress the ECP.

In order to facilitate ECP closure, a collaboration and shared resource with the Computer Aided Design (CAD) team and Geographical Information Systems (GIS) team was developed by the Information Compliance team by means of a separately dedicated SharePoint-based list tracker linked to the Quality team’s contract completion calendar.  This enabled active management of regular contract review points, assigning responsibilities and setting target dates for completion of items within the formal ECP ahead of final ECP sign off.

Contract Reporting

Programme dashboards (‘Contract Performance Summary’ illustrated in Figure 4 below) were produced periodically and published programme-wide.  The ultimate goal was to drive business improvements.

EDMS-generated reports allowed for the generation of data for programme dashboards – as well as document delivery reports to support the ECP (discussed earlier) and Handover Readiness (see ‘Information Handover Principles’ learning legacy paper).

The EDMS document lifecycle report itemised errors in detail at both overview and individual document level.  It was also possible to attribute these errors to a responsible party (i.e. Crossrail or the contractor).  The issues highlighted were related to:

• Document delivery (i.e. approval and acceptance status against the planned/submitted and effective dates).
• Document associations (i.e. relationships established between related documents).
• Metadata integrity against the physical file (i.e. index, class, name, revision and document status decal).

Errors could be expressed as a percentage value for each issue type against the total number of documents for each contract.  This allowed for weighting/equalization, and thus ensured fairness and unbias.  The percentage value was then translated to final headline scores for each contract.

Figure 4 –  excerpt of template headers from the ‘Contract Performance Summary’ dashboard

Highlighting issues periodically permitted contractors to address specific issues and prevent recurrence where possible. It also enabled the Crossrail functional groups to identify high-risk areas and accordingly communicate, apply remedies and fill training gaps (e.g. clarification of procedural requirements, setting standards, guidelines and best practice).

This, alongside the setting of targeted improvement actions and the sharing of best practice, helped contractors to exceed the minimum requirements – advancing from ‘basic’ compliance to ‘world class’ compliance).  In this respect, ‘world class’ Information Compliance was regarded as the behaviours listed below in Table 3.

‘Basic’ compliance	‘World Class’ compliance
Meeting requirements of documented procedures	Demonstrating continued compliance with documented procedures (i.e. information transfer via training programmes, transparent data sources etc.)
Reacting rapidly and positively to changes in requirements.	Demonstrating Resilience: in adapting to and documenting processes to manage changes in requirements.
Reactivity: highlighting issues, proposing solutions and applying Corrective Action.	Proactivity: applying Preventive Action (and error-proofing).
Record unsuccessful events as ‘Lessons Learned’.	Recognise and record  success as ‘Lessons Learned’.  Then celebrating the success (i.e. rewarding excellence).
Counsel and guidance to Document Control.	Identification of knowledge gaps and addressing where possible.
Project collaboration.	Engage, communicate and build relationships through knowledge-sharing activities (e.g. focus groups, forums and printed media such as newsletters).  Rejecting the ‘silo’ mentality.
Reporting SMART Key Performance Indicators: – specific, measureable, achievable, realistic, time-bound).	Develop ‘SMART’er Key Performance Indicators – e.g. automated reporting and integration of data sources.
Report informative, clear and easy-to-understand Key Performance Indicators.	Using those Key Performance Indicators, to in turn, drive improvements in metrics – and verifying through further metric analysis.
Individual accountability for tasks.	Skill sharing – extending externally and integrating skills across teams.
Meeting target dates and deadlines.	Completing tasks ahead of target dates and deadlines – through management of priorities.

Table 3 – Defining ‘World Class’ for Information Compliance

To accurately reflect the progressive stages of the project from information creation, population and data migration, revisions and review & acceptance it was important to continuously re-define compliance success.  Thus key performance indicators were frequently reviewed for usefulness and newer more dynamic success criteria applied according to the project stage.

Also as basic compliance was generally established across all contracts (i.e. similar scoring and flat-lining performance curves) the success bar would be occasionally raised, incrementally setting more challenging scores to attain.  This thus motivated contractors to continually improve and progress to a world-class position.

Applying the same constant key performance indicators across the lifetime of the project, would have yielded inaccurate progress reporting.  This would have ultimately hampered project progression and limited opportunity for improvement.

At the commencement of the Crossrail project success criteria were focused on establishing a ‘right-first-time’ culture within the EDMS i.e. reducing errors such as metadata, code mis-matches, correctly identifying deliverables and assigning structures. At mid-point, project success criteria were focused on document delivery such as meeting target/submission dates, work order completion and communications closure.  At latter-point, project success criteria had evolved to addressing and closing out actions, document suitability, EDMS approval, and final Handover and Assurance coding.

Monitoring Data Compliance for Handover to Operations

Advancing towards final Handover the emphasis for information compliance has shifted from inward contract-facing document delivery assurance tasks to outward facing and interfacing whole-data merging and association activities with provision of assurance for all deliverable technical data, integrated where specified for Asset Information, Computer-Aided Design (CAD) and Geographic Information Systems (GIS).

Horizontal auditing against final acceptance criteria of the Crossrail Handover Information Plan (CHIP) and Information Review Points (CHIRPS) will take prominence over vertical auditing (itemised lifecycles).  These will also incorporate validation of review and acceptances of Testing and Commissioning, Operation and Maintenance, Health, Safety and Environmental packages, aligned as far as possible to final requirements.

To achieve this, higher level automated reporting will replace the eB / QlikView Lifecycle Tracking, utilising a traffic-light status reporting i.e. RAG ‘red-amber-green’ indicators for document package completions against the Crossrail Handover conditions.

Root Cause Analysis

Information risks were identified and corrected using Root Cause Analysis and the tool-of-choice was the ‘Spot-On’ methodology. bespoke to Crossrail.  This was a problem-solving tool to aid in the creation of practical solutions (mostly process improvements) and to prevent recurrence of persistent problems.

The methodology was based on the well-known and recognised ‘Fishbone (Ishikawa) Cause and Effect Analysis’ of Identification – Correction – Prevention.  This involved performing a formally documented and timed investigation with defined roles and responsibilities, examining:

1. What went wrong? – a detailed but concise and accurate definition of the circumstances with evidence and source.
2. Why? – identifying the fundamental and causative factors linked to knowledge- , rule- and situational- based issues (e.g. training gaps, system errors, operator errors etc.).
3. How to prevent recurrence (with rationale / justification) – implementing and following up actions with assigned and tracked tasks (e.g. training, implementation of controls, procedural modifications etc.).

Each process was documented in a ‘Spot-On Template’ defining the problem/risk area, process mapping, implementation of corrective and/or preventive action and measurement of effectiveness (validating action closure, checking for recurrence and ongoing monitoring of contractor performance through trend analysis).

A SharePoint-based list tracker was also developed to facilitate active logging, action tracking, trend reporting and visibility / knowledge-sharing between the contractors and delivery teams (illustrated in Figure 7).

Figure 7 – Screen-shot of SharePoint-based Root Cause Analysis (‘Spot-On’) tracker

Highly informative and practical data was generated from Root Cause Analysis (‘Spot-On’).  The intelligence was employed in prioritising remedial actions and applying the following corrective and long-term preventive actions:

• Ensuring user understanding of policies and procedures.
• Encouraging individual accountability for documents and data created (document owners and reviewers).
• Motivating users through scoring and benchmarking of key performance indicators.
• Highlighting training gaps to the user and the programme development team.
• Feeding back trends to the core business areas – specifically the Delivery and Audit teams. Then working together to drive improvements and efficiencies to the project
• Regularly assessing the effectiveness of the programme through periodic reviews and reporting

The benefits of Root Cause Analysis (‘Spot-On’) were demonstrated in the results and scores from performance assurance assessments.  Overall, steady improvement was observed period to period and year on year.

Conclusion

The output of the Document Data and Compliance activities allowed Crossrail to drive business improvements through:

• Solution finding through trend-based Root Cause Analysis (recommending, implementing and driving Corrective and Preventive actions ‘CAPA’).
• Bridging gaps across the business and programme-wide through collaborative knowledge-sharing and training to improve understanding of processes and modify behaviours.
• Setting and defining standards and guidelines and adaptation of best practices.
• Anticipation of problems in preparation for programme completion i.e. Handover readiness.

Transparency, cohesion and strong programme-wide relationships were essential in delivery of the Information Compliance strategy.  Knowledge-sharing and shared reporting resources provided intelligence which was used to prioritise and focus compliance monitoring activities.

Wasteful duplication in repeatedly highlighting similar areas of contractor non conformances was avoided by referring to Delivery team surveillance observations, Audit team evaluations and Quality team management review data.  This revealed valuable information such as contractor organisation / reorganisations, roles and responsibilities, resource issues, outstanding documentation actions e.g. absent or inadequate procedures, training gaps and quality improvement plans.

The project information compliance learning legacy imparted by Crossrail is one of ensuring adaptability of a compliance monitoring programme.  This must be achieved through committing resource for frequent assessment at each project phase and planned information sharing across the business.  Data sources must then be reviewed at regular intervals to assess their suitability for measuring compliance at any particular phase.

Further, importance must be placed on moving the bar higher and higher in performance monitoring activities as the project progresses. Avoiding the temptation to remain statically focussed on the same indicators of performance or success or completion.  This means replacing basic monitoring of non-conformances with realistically achievable and permanent behavioural indicators of compliance which drive cost and time efficiencies.  Examples include time taken to close critical actions, change control, communication/feedback of change and opportunities for learning.

There will always be demanding challenges in information compliance whether quality-checking at project start up ongoing or progressive monitoring of records, or validating completion and handover data sets.

The confidence is that, for information compliance, it will always be possible to evolve more advanced and multiplexed tools and reports to identify the ‘WHAT’ ‘WHO’, ‘WHERE’ and ‘WHY’.

However, the fundamental and principal challenge will be in the ‘HOW’. That is, in permanently influencing positive behavioural (cultural) compliance.  This is because, whatever sophisticated EDMS is in use, it will only ever be as exceptional as the humans using it.