The major IT outage which caused significant disruption across the Southern Health & Social Care (HSC) Trust on September 17, and in the following days, was high on the agenda at a Southern HSC Trust board meeting held on Thursday, September 25.
Most of the Trust’s planned surgery and out-patient hospital appointments were cancelled on September 17 and 18 as a result of the outage, which has now been resolved.
Speaking at the board meeting, the Trust’s new Director of Planning, Performance and Informatics, Elaine Wilson, pledged that the matter would be fully looked into.
Detailing the steps that took place since the IT outing occured in the morning, she said: “At 8 o’clock on Wednesday, September 17, the Southern HSC Trust experienced a full network outage.
“It is important to be clear that while we are referring to an issue within our data centres, and we have two data centres on our Craigavon Area Hospital site, those are local data centres.
“The access to the regional data centre to provide information to the other Trusts was still there, so all of the other Trusts were still functioning as normal at that stage.
“After 9.15, it was clear that this was not a short outage issue. It was certainly not expected and [our network provider] felt it couldn’t have been foreseen.
“At 9.15, the major incident protocols were stepped in at that stage, and the decision was made to request from the region some support in relation to our emergency services.
“We also had some discussions in relation to the planned activity on the Wednesday, September 17, and [for] the safety of our patients we stepped down the elective activity on the Wednesday.
“At that stage, we didn’t know what had caused the outage, and it could have [lasted] a number of days.
“As the day progressed and the system remained down, we made the decision later on to cancel the elective procedures for September 18.
“While the network outage did impact on all of our sites, it was quite quickly decided that we could reroute, we had a data centre at Daisy Hill as well, and it was connected to the data centres in Craigavon.
“[Steps were taken] at around 11.30, and that allowed some connectivity back into the Daisy Hill Hospital site, and also to our community services.
“At that stage we rerouted our business continuity arrangements. Part of that continuity is linked to the encompass system.
“We now have business continuity devices that were provided as part of that changeover to that shared patients record, which is a really positive position for us to be in, in that we have PCs around our hospital sites and other community sites that allow us access at the time that the network goes down.
“We have an encompass app on little phones that some of our staff have in our hospitals, and there was a possibility to hotspot onto those and get some information downloaded.
“Although it was the data centre that caused the network outage, [some] people were working at home and they had their own Wi-Fi phones.
“They were able to get on some of these applications at home as well, so that helped in terms of managing the incident.”
The senior Trust officer pointed out that the new encompass IT care record, which cost £10million to roll out last May, is still fairly new to staff, and she asked for some understanding in that regard.
She continued: “While our business continuity arrangements were enacted, I think we do have to reflect on the fact that we are only four months post ‘going live’, in terms of encompass.
“Our staff are still learning how to use that system within a live environment, never mind within an environment where it isn’t live.
“While this was a really regrettable event, I think there will be massive learning for us, in terms of how we managed it, not just as a major incident, but in particular around those business continuity arrangements.
“This is a learning opportunity, and we will learn as quickly as we can in terms of how we managed the event.
“A lot of the community services were able to deliver fully on the Thursday. The biggest impact was on the Wednesday, with some [additional impact] on the Thursday, mostly on our hospital sites.
“[Rebooking appointments] has had to be a manual exercise. It is certainly a time-consuming piece of work, but as of Monday there were 1,192 of our 1,600 booked, so about 73 per cent have already been provided with new appointments, which I think is a really positive position.
“This was definitely based on clinical need, and our clinicians were involved in deciding who needed to be booked quicker. The hope is that we will get everybody else booked as quickly as possible.
“A significant amount of these appointments are not for surgery, they are for outpatient appointments.
“Of the 111 red flag appointments, four are remaining to be booked, and those will be booked very quickly.
“In relation to trying to investigate what happened, we’ve set up two groups.
“The first group has been set up which is called the Root Cause Analysis Working Group, and it’s chaired by myself.
“It’s extremely complex, and I know that people are frustrated that we haven’t said what happened, and given a guarantee that this won’t happen again.
“This was a very unusual event that was not anticipated. We really do need to get to the bottom of what exactly caused it, before we could say that 100 per cent it could not happen again.”
Meanwhile, Executive Medical Director at the Southern Health & Social Care (HSC) Trust, Dr. Stephen Austin, has revealed that resorting to pen and paper proved to be a handy ‘Plan B’ for IT-deprived medical staff during the recent outage.
Such was the scale of the IT incident that approximately 1,600 appointments had to be postponed on Wednesday and Thursday, September 17 and 18 respectively.
Addressing the board panel of the statutory body at a meeting held last Thursday, September 25, the senior Trust representative said: “The clinical impact of this for our doctors, nurses, AHDs [Allied Health Professionals] and other colleagues, [is that] there was very slow access to the Epic system [used to run the new encompass electronic care record].
“When I did speak to a number of staff, they were working with patients, but there was an air of controlled calmness.
“They were busy, they were challenged, having to go back to paper, but there was a sense of control, and it was very evident that there was no panic.
“I spoke to a number of consultants, and while they recognised it was challenging, they worked through that and had other staff assisting them to get the information needed.
“They also had a heightened sense of vision to look out for issues that might have arisen, so they were being alert to that.
“One of the things is to thank our staff; they did very well.
“They’d got used to electronics, and had to go back to paper, and it was a shock going back to paper again, because you get used to electronics more than you think.
“I think staff did exceptionally well going back to the old pen and paper, and it worked really well.”
