Leased Lines Replacement Project (2020)

Crummock Water (by Ian Greenwood) - Courtesy of United Utilities

In April 2015, BT announced they would be withdrawing support of all their very low bandwidth (VLB) leased lines from a fixed deadline of the 31 March 2020. Leased lines are dedicated lines, used as connections to carry communications or electrical signals between two locations. They are often used in conjunction with telemetry outstations, with built in modems to carry numerous signals, or as part of an electrical ‘loop’ carrying a single signal. Leased lines are used by a wide variety of organisations, including operators of Critical National Infrastructure (CNI) like United Utilities (UU) in order to support essential public services. The network upon which BT provided a large portion of VLB services was ageing and approaching the end of its life.

Background

UU relied heavily on leased lines and had over 190 in place across the north-west providing business critical control and monitoring of key activities on over 380 sites. The loss of communications due to the leased lines being disconnected presented a number of risks to UU, including issues such as the loss of supplies to customers as a result of reservoirs running dry, or flooding of properties due to sewers overflowing. These risks had the potential to cause significant reputational and financial damage by adversely impacting the achievement of UU regulatory performance commitments.

The scale of the challenge

The majority of leased lines within UU had been in place for over 20 years and were originally installed on the instruction of local operational teams. They had locally agreed support arrangements and were not centrally managed or monitored by UU. The majority of the data held by BT referenced BT exchange locations rather than UU site details.

The first challenge was to establish the project scope by locating which UU sites were connected by a leased line, the specific function of each line and if it was still required. Another challenge presented to the team was the likelihood that most equipment communicating via the leased lines would be obsolete and unsupported. If equipment was to fail following disconnection to test the function of the line, there was a risk that it may not reconnect. This would make the proving the functionality ahead of the swap out a challenge and increased the risk of operational impacts during commissioning and design changes being required.

With over 380 sites affected from the Peaks to the Lakes and beyond, logistics were challenging. Equipment was housed in variable locations from huge treatment works serving Manchester to confined spaces in steep remote hillsides.

With no ‘like for like’ wired replacement being offered by BT, UU needed to identify a selection of suitable modern solutions that could interface with their existing control and telemetry systems. A training programme and new support model would also need to be established for the replacement communications method.

The team

The project team comprised of a project manager, supported by two coordinators, a lead engineer, three engineers and a dedicated testing resource. Operational teams supported in locating and identifying the functionality of the leased lines in addition to providing site access and commissioning support. A ‘core team’ comprising of senior leaders representing key stakeholders was established to provide strategic guidance and to act as an escalation route for issue resolution.

Due to the criticality of the leased line communications and their often ambiguous operational functionality, the decision was taken to retain accountability for the design and commissioning activities within UU, and utilise the support of a framework contractor, Ramptec. Their extensive experience of UU’s operations and procedures coupled with their track record of efficient project delivery and excellent working relationship made them an ideal fit to undertake the installations and support the engineers with commissioning.

Stakeholder liaison

Key operational stakeholders were identified and split into groups based on the locations of their sites. Stakeholder liaison meetings were held with each group to identify the leased lines within each area. These meetings facilitated the early identification of lines that had either previously been replaced with alternative communications methods or were no longer required.

Site surveys & development of standard solutions

A tight programme of surveys, beginning with a batch of 10 pilot sites was initiated over 18 months, and sites were batched geographically and prioritised based on volume. To ensure consistency of approach, quality of information and standardisation of data collection, a method statement and recording template were designed for engineers to collate their findings and document their proposed design solution. This document also contained requirements for panel design and builds, power arrangements, equipment housing, installation and panel and antenna locations, which would later form Ramptec’s scope of work.

Upon completion of the initial 10 pilot sites, a set of standard solutions were developed that were reliable and secure for both simple point to point (site to site) system designs and multiple point to point designs. The solutions had to be capable of marshalling and conditioning small amounts of analogue and digital signals between sites, replacing in the most part systems based on legacy outstations by Churchill (Micro Link, NANO Link, MEGA Link) and PACS. Solutions would also need to be suitable for replacing lines that were used as part of hard-wired instrument loops between sites.

These solutions were based on an ethernet enabled PLC and various DSL and 3G/4G routers employing technology that was understood and widely supported throughout UU. The WAN routers also provided capabilities for data encryption in line with UU IPSec strategies and policies. The team were confident that these solutions would be applicable to most set ups, however bespoke solutions would be implemented to ensure continued functionality and compliance if required.

Churchill outstation leased line example – Courtesy of United Utilities

Churchill outstation replacement solution example – Courtesy of United Utilities

The survey programme continued; now using knowledge gleaned from the pilot sites. Whilst on site, engineers tested and recorded each signal to ascertain if the lines were operational and still required by the business. The 3G/4G survey was critical, both to choosing the correct solution and to the performance of the completed migration. The survey utilised a setup of a standard 3G or 4G router and a mobile antenna.

This setup was used in conjunction with a bespoke Java application, which scans all available networks and produces a report detailing signal strength and quality for each network found. The area surrounding the leased line equipment was surveyed until a location with suitable signal strength and quality was found.

The antenna position often required the router to be installed in a dedicated remote panel adjacent to the antenna location, extending the antenna cabling was avoided due to the negative impact on signal strength and quality.

Where mobile network coverage was found to be inadequate, the use of existing DSL services was considered. If an existing DSL service was also unavailable, then existing PSTN services were considered for upgrade, or a new DSL service was sought. The survey checked any existing telemetry installations for spare UPS backed 24VDC power supply capacity, so that these could be used to feed the small amount of power needed for the replacement solution. This resulted in reliable performance, reduced installation costs and fewer UPS/battery installations.

As more surveys were undertaken it became clear that the majority of leased lines were either connected to telemetry outstations or to a service reservoir level transmitter at one end and a pumping station at the other (a loop that is powered by a pump control circuit).

Design review & approval process

Upon completion of each survey report, the proposed design solution would require endorsement from the lead and principal engineer at a weekly design assurance meeting. This vigorous process was implemented to ensure that the right solution was selected and to minimise rework at a later date. It served as an opportunity for engineers to discuss any issues and share best practice and lessons learnt throughout the project.

Installation & commissioning

In order to maintain progress against the fixed deadline put in place by BT, a rolling installation programme was implemented in parallel with the on-going survey programme. This enabled the project to work as a well-oiled production line, removing the need for all surveys to be complete before installations could commence.

The approved survey report provided a bill of materials to be procured, including hardware which was required to be configured and tested by a dedicated resource.

Workshop testing

The next step on the production line was for the solution to be configured and tested in a workshop environment before being passed to Ramptec for deployment on site. As all installation and commissioning work would be done on operational sites, it was critical that a thorough workshop test was carried out to identify and resolve any issues that could cause an operational impact and/or potential outages.

At this point, the router was set up on a bespoke telemetry dashboard in order for the health of the specific site communications to be monitored in real-time, a new, beneficial feature not available as part of the BT offering.

Standard installations

The completed survey report formed the basis of Ramptec’s scope document, incorporating details of the existing installation and a scope of the works for the new solution. This enabled Ramptec to accurately cost the installation without the need for an additional survey, providing time and cost efficiencies.

The installations followed a standard approach, where the first task was to establish communications at both sites by the end of day one. An overnight ‘soak test’ of the communications stability was carried out before transferring signal wiring to the new equipment on day two. As a final check, all signals were end to end tested against their values prior to migration.

Bespoke solutions

Whilst the majority of the leased line systems were replaced with PLCs linked by 3G mobile or ADSL communications, some more unique and complex links required bespoke solutions.

Where sites employed specialist ‘Sinaut’ configuration, the existing PLC system was modified to replace leased line communications between the two PLCs with TCP/IP communications via new ethernet modules added to the PLC’s, with the code modified to direct communications via the new, encrypted TCP/IP path.

Bespoke Sinaut system replacement: Before – Courtesy of United Utilities

Bespoke Sinaut system replacement: Solution – Courtesy of United Utilities

Where legacy GEM80 PLC systems were in place, these were modified to replace leased line communications between two PLCs with TCP/IP communications via new protocol convertors, using encrypted TCP/IP between the sites.

GEM80 serial communications: Before – Courtesy of United Utilities

GEM80 serial communications: Solution – Courtesy of United Utilities

Support

As a result of BT’s withdrawal, it was critical that a new support model was implemented. An early decision was made with key stakeholders and the core team to keep support in-house to give UU control and avoid reliance on a third-party provider. Discussions then took place with the support team’s senior management to understand and agree what information should be provided for a successful handover from the project team. The majority of this information was already available as a result of the detailed site surveys which avoided the need for any additional information gathering.

Summary

The collaborative and systematic approach adopted by a skilled project team resulted in all of the leased lines being successfully replaced ahead of BT’s withdrawal deadline. In addition to meeting performance commitments, increasing security of supply and reducing the risk of failing water quality compliance, additional benefits included reduced operational costs associated with BT line rental and the removal of operational costs for redundant lines.

The project completed significantly under budget. A factor in this was the early decision for UU to undertake site surveys rather than appointing a third-party contractor. This proved invaluable as it facilitated better risk mitigation, enabled design changes to be implemented with ease and in turn resulted in the majority of installations being successfully implemented first time.

The project received very positive feedback from key stakeholders. This was gratefully received as the project team had invested great time and effort in engagement from an early stage. An additional benefit of this was the stakeholders highlighting a number of lines which were redundant, removing the need to undertake a site survey and saving the project time and effort.

The support model put in place on the back of this project, and the process followed to achieve it, is now held as an example of best practice in UU which other projects are encouraged to follow. The support team was engaged early and at handover was provided with key information about each solution. The bespoke telemetry dashboard commissioned by the project has also proved an invaluable tool in monitoring the health of communications between sites remotely.

In summary, whilst working to a fixed deadline this project managed to successfully achieve its objective to replace the BT Leased Line service with an equally robust service which could be supported in-house, reducing the reliance on a third-party supplier. The project was completed under its capital budget whilst also providing operational savings to United Utilities. The output was a supportable, future-proofed solution which received high praise from a number of key stakeholders.