'We weren't ready' — Inside St. Michael Medical Center during October cyberattack outages

SILVERDALE – In early October, a fresh batch of troubles arrived at St. Michael Medical Center. They would first be linked to an “IT security incident.”

It would eventually be acknowledged by the hospital’s parent organization, CommonSpirit Health – one of the largest nonprofit health care systems in the country – that the group had been hit by a cyberattack. A number of facilities under the CommonSpirit umbrella across the country would report disruptions. The extent of patient information accessed in that breach remains unclear, though the organization said in an update on Thursday that some personal information of patients of St. Michael Medical Center and other Virginia Mason Franciscan Health facilities in the state was exposed when an unauthorized third party accessed its network.

In an initial statement provided to the Kitsap Sun on Oct. 3, Virginia Mason Franciscan Health noted that CommonSpirit was managing an “IT security incident” and said it had taken certain systems offline, including electronic health records and other systems. It would be about two weeks before VMFH would say that it was bringing systems back online.

In a statement on Thursday, CommonSpirit said it detected activity on its network on Oct. 2 that was later determined to be ransomware and said that the third party had access to portions of its systems between Sept. 16 and Oct. 3.

More:St. Michael Medical Center patient information at risk in ransomware data attack

The outages would compound problems caused by the short staffing issues that have plagued the Silverdale hospital for months, and the reality on the ground was far from rosy during that period in October. Interviews with hospital staff and a Kitsap Sun review of complaints made to the state’s Department of Health paint a concerning picture of the facility’s operations during that period.

“Patients continue to receive the highest quality of care,” CommonSpirit would claim to the public in an update on the situation on Oct. 17, “and we are providing relevant updates on the ongoing situation to our patients, employees, and caregivers. Patient care remains our utmost priority and we apologize for any inconvenience this matter has created.”

During the outage period, providers at the Silverdale hospital could no longer use Epic, a platform widely used in health care settings, to access and record patient information, and staff said backup information wasn’t available to work from. Without Epic, the facility was also operating without a central communication system and the patient safety checks that the platform provides. A switch to paper records slowed processes.

Delays in the emergency department stretched out, and during the chaos, some patients slipped through the cracks in the waiting room, only to be discovered during roll calls. Ambulances backed up as crews waited to hand off patients to the emergency department for care.

More:'Shouting into the void': Records outage, short staffing cause dire scene in Kitsap's ER

Lab results that should have come back in short order took hours, slowing patient care. Safeguards in the medication delivery process were removed. Communication between departments slowed. Even a timekeeping system for employees went down, causing pay headaches that are still being figured out, staff said in interviews this week.

“You couldn’t look up any patient history,” said an emergency department nurse at the hospital who requested to not be identified by name. “A lot of patients don’t know what medications they take or even what their allergies are. We had no access to that.”

That nurse added: “There are a lot of people in the community, a lot of medical information is over their head, they just don’t grasp what is told to them or at least the details. When you can’t look that up, that puts you at a huge disadvantage with what might be going on with this patient, especially if they had a recent procedure, looking at their medical history.”

That nurse noted challenges like being unable to look up what pacemaker a patient might have or what surgery they might have undergone recently. One patient that nurse treated had a cognitive deficiency and couldn’t report where they lived.

“We had no way to look up in a computer a contact information for that person, a family member,” that nurse said. “We didn’t know where they were going home to because we didn’t have a phone number of anyone to contact or an address to send them home to. You couldn’t look that up.”

WE WANT TO HEAR FROM YOU:Do you work at St. Michael Medical Center? Had a recent visit to the hospital? Fill out our form.

Was the Silverdale hospital prepared for such an outage?

“We weren’t ready,” said Cindy Franck, a nurse who works on the eighth floor of the hospital.

She added: “Especially when it comes to patient care, we have to have a backup plan. I don’t think we were prepared, and we have to be prepared for that. We saw how unprepared we really were.”

The anonymous nurse said that the hospital did have processes in place for outages but said that they are generally geared toward short periods, such as when system maintenance is required.

“I don’t feel like there were good processes in place for an extended downtime such as what we had,” that nurse said.

That nurse didn't have a specific example in which a patient was affected by the delays, but said, “I can’t imagine it not happening. If labs are delayed for hours, and patients are sitting for hours longer than they need to, I can’t imagine there not being an outcome that was worse than it needed to be.”

The Kitsap Sun requested an interview with St. Michael Medical Center president Chad Melton for this story, but VMFH spokesperson Susan Callahan referred a reporter to CommonSpirit, saying that the outage period was a “national issue and anything about the downtime needs to go through the company.” CommonSpirit spokesperson Chad Burns referred the reporter to a Nov. 9 statement on the organization’s website.

In its update released on Thursday, CommonSpirit said that as part of the ransomware attack, the third party gained access to portions of the group's network and may have gotten access to files that contained personal information of patients of facilities now under the VMFH umbrella. The organization didn't specifically refer to patients outside Washington in that update but said a review was ongoing.

"Upon discovering the ransomware attack, CommonSpirit quickly mobilized to protect its systems, contain the incident, begin an investigation, and maintain continuity of care," the latest update said. "In addition, CommonSpirit notified law enforcement and is supporting their ongoing investigation. Once secured, systems were returned to the network with additional security and monitoring tools."

A state Department of Health spokesperson confirmed that DOH had received reports about the situation and said those were reviewed through the department’s normal processes.

“An investigation is being conducted; should we find any violations a statement of deficiencies will be issued to the facility with an expectation that they’ll provide a plan to correct those deficiencies,” DOH’s Frank Ameduri wrote in an email on Nov. 21. “Hospitals are expected to have plans in place to respond to situations of this kind; each hospital develops its own plans, but they must be sufficient to adequately protect patient safety. We are unable to share additional details at this time.”

'It was a mess really'

Two emergency department nurses told the Kitsap Sun that they were part of a group of ED staff who spoke with Department of Health representatives on a Zoom call in October, and in that call, they spoke about a range of questions and concerns, including the records outage period, they said. Topics included management presence during the downtime, communication with employees, protocols and procedures, staffing and more, they said.

Kelsay Irby, one of those nurses, said she raised a number of concerns about the hospital’s handling of the outage period, including the lack of pharmacist oversight in the process of providing medications for patients, finding patients in the emergency department’s waiting room during roll calls who weren’t on its list to be seen, a particular night in which staffing was low to the point that mental health patients didn't have a dedicated nurse supervisor as they were supposed to.

“None of them have good poker faces, so some of the things that we said to them, you could tell that they were shocked, like, ‘How is this even happening in a hospital right now?’” Irby said.

Irby noted another critical piece of information staff didn’t have access to during the outage period: Patient advance directives and next-of-kin contact information.

“If you have somebody that’s brought in from an assisted living facility or something and they were found down and they were just kind of a ‘scoop and run’ and nobody really knows much about that person, I don’t know what their code status is, do they want me to do CPR? Do they want me to intubate them? Do they want me to call their family? What do they want?” Irby said. “It makes me emotional to think about it, because that is everybody’s really incredibly personal decision, do you want to have CPR or not? I can’t stress how important that is to be able to respect somebody’s … literally their dying wish, do or do not do all of the things. For us to not even know that, I felt like that was a major disservice. You can do harm by doing too much, just like you can do harm by doing not enough, too little.”

In the emergency department, staff used a spreadsheet to keep track of patients in the waiting room, though in the chaos, some might get lost in the shuffle, nurses said. Keeping track of where patients were in the hospital became a challenge. Patient safety checks that exist when Epic is up and running normally didn’t exist during the outage. Even interpreting handwriting could be an issue.

Franck, the eighth-floor nurse, said the lack of pharmaceutical oversight during the outage period was frightening.

“You really had to pay attention to what you were doing, and you had to know your meds,” she said.

A shift to paper charting, she said, slows everything down. “There are nurses that have never charted on paper, and so they don’t really train you how to do that on the job, because you rely on the computer. We should have been a lot more prepared than we were. … It was a mess really.”

Franck described the outage period as feeling like it was a decade long.

“In a hospital that’s already short-staffed, it wasn’t fun at all,” she said. “I just have to say this for all of the people that work there, really stepped up and did a lot of extra hours to make sure that we provided the safest care we could.”

'You’re dealing with criminals'

Health care organizations offer an attractive target to bad actors: They store sensitive data, and when it’s held hostage and patient safety hangs in the balance, there’s added pressure to pay up to get it back. (It’s unclear how CommonSpirit resolved the attack it sustained.)

Even with an array of protective technologies in place, people are still the first line of defense, said Michael Berry, chief information security officer for Kettering Health, a medical group with 14 hospitals in Ohio. All it takes is one wrong click to circumvent security measures, he noted.

Berry pointed to examples of organizations operating on downtime, or back up, procedures for months at a time.

“Imagine not only the patient care portion, but the rest of the business functions being impacted,” he said. “How do you pay people? How do you draw bills or claims? How do you receive money? How do you keep the business going?”

“The dependency upon the electronic medical record is obvious nowadays,” he added. “We’re so accustomed to having that information right at our fingertips. While those systems do have downtime computers, which are basically computers that provide a backup of historical data for patients in case of downtimes, if those additional systems were to be infected or unavailable, basically organizations that are impacted like this are going back to pen and paper.”

In a ransomware attack, an infection can reach an entire network before revealing itself, Berry said. Even backups could be compromised. An attack would encrypt patient data, sealing it off from access, and a bad actor would demand a ransom payment in exchange for handing over the keys to unlock it.

Even if an organization were to pay up, “You’re dealing with criminals,” Berry said. “There’s no guarantee, first of all, that you’re going to get your data back. Second of all, with the proper keys there’s no guarantee that you’re going to be able to unencrypt all the data successfully, and finally, you’ve marked yourself as a target as someone who’s going to pay.” Cryptocurrencies, the usual payment method, and the anonymity they can provide, have made cybercriminal activities "prolific," Berry said, but he noted that there have been examples of federal authorities clawing back such payments.

Attackers have shifted from stealing and selling private data to restricting access to data and demanding payment from organizations and now, to demanding payments directly from patients as organizations have refused to pay, Berry said.

“The bad actors are continually growing and changing their tactics,” he said. “Organizations have to be prepared and agile and be able to shift and protect against all facets of these threats.”

'Concerned for patient safety'

Department of Health complaint records obtained by the Kitsap Sun through the state’s Public Records Act show that five cases connected to the outage issues at St. Michael Medical Center during the month of October were under investigation at the time records were provided on Wednesday.

Of those five cases marked for investigation, two complainants were described as “unknown,” one was listed as “anonymous,” one had their name redacted and was described as a “whistleblower,” and a fifth was listed as the Department of Health itself. Department case summaries describe a range of problems connected to the outages.

According to a department summary of the whistleblower complaint, “Hospital administration and leadership has allegedly failed to provide a process for necessary safety checks for safe medication administration, after the electronic medical record (EPIC) went down and offline for providers to utilize for patient care. It is also alleged that hospital staff is allowing patients to be treated above the ED room capacity and not allowing any patient divert. Additionally, lab and imaging results are not provided in a timely manner with a wait time of two hours or longer.”

According to a summary of another complaint, lab results at one point were taking longer than 13 hours for some patients in critical care and STAT information, such as labs or imaging needed quickly, took up to five hours to be returned. That complaint alleged that history and physical examination information was not available for many patients, medication administration records were being transcribed by nurses with no double checks by pharmacists, and that some orders for new medications, discontinued medications or medication changes were not being updated in paper records for patients.

“Concerned for patient safety,” the department noted in its summary of that complaint.

“Staff have allegedly been forced to track down hard copy files to get patient history. Facility leadership allegedly have no training, guidelines, or safeguards in place for this scenario,” another complaint summary said, referring to the outages.

Another alleged “compromised and inferior patient care,” according to another summary: “Patient safety concerns due to hospital service outage issues and unsafe patient to nurse staffing ratios, causing patient harm to occur daily. Entire electronic system is unavailable causing communication and care issues at all levels.”

The complaint attributed to the Department of Health is connected to a widely publicized incident during the outage period in which Irby, one of the ER nurses, called Kitsap 911 to ask if fire department support could be dispatched to support the facility when the department became overwhelmed with patients. Two Central Kitsap Fire and Rescue responders helped in the emergency room for a period.

Cybersecurity as a patient safety issue

Experts told the Kitsap Sun that it’s important for health care organizations to have plans in place to be ready for the eventuality of an attack like the one that hit CommonSpirit.

“The reality is that ransomware attacks are becoming more common, and it is no longer a question of if but when,” said Dr. Michael Ramsay, CEO of the nonprofit Patient Safety Movement Foundation, in a statement. One vulnerability Ramsay noted: Some hospitals use older devices that aren’t hardened against today’s cyberattacks.

Ramsay said that clinicians need access to patient records to make decisions on care and that hospitals rely on technology for everything from setting appointments to doing tests. During an attack, care can be delayed, and hospitals are forced to prioritize patients based on urgency, which creates a backlog for care, he said.

It’s critical for organizations to have detailed cybersecurity plans in place, with training for staff on what their roles are in those situations, Ramsay said. Patients, he said, can be advocates for themselves by keeping copies of their own medical records available.

Organizations should focus in on security as early as possible during the process of procuring technology, seek to control security risks as best they can and have plans in place that are actually reviewed periodically, updated and put into practice through tabletop exercises, said Juuso Leinonen, a cybersecurity expert who focuses on medical device security for ECRI, a nonprofit organization that evaluates health care technology and safety.

Many health care organizations are used to various kinds of disruptions and have plans in place for outages, but it’s especially challenging when those disruptions last into days and weeks, Leinonen said.

“Not every incident response plan is going to account for that level of disruption or that length of disruption in those plans,” he said.

“Everyone’s going to back up,” said Berry, the information security officer with Kettering Health in Ohio. “The real question is, do you test the ability to restore from that backup? And that is where most people fail, because they don’t realize they can’t restore that backup until they absolutely need it.”

Organizations have become more and more reliant on modern, interconnected electronic systems to deliver patient care, and that interconnectedness can bring many benefits, Leinonen said, but he noted that the reliance on those systems makes disruption a particular challenge. Organizations can lose access to devices and patient records, safety systems that providers might be accustomed to can go away, and access to inventory, supply and appointment scheduling systems can disappear, Leinonen said.

“Cybersecurity absolutely should be thought of as a patient safety issue in a health care organization,” he said. “While in most cases a lot of these disruptions can be handled in various different ways, but truly it can cause disruption and if that disruption leads to delays in patient care, the worst-case scenario is patient harm and obviously no one wants that.”

This article originally appeared on Kitsap Sun: Inside St. Michael Medical Center during October cyberattack outages