Advocate Healthcare Moves 32 Terabytes of Patient Imaging Data to Safe, Easily Accessible Storage in Days with Hyper-Migration

Advocate Health Care recently utilized DataFirst’s Hyper-Migration to move patient data and reports out of a DVD platform into a single, universally accessible spinning disk archival system. Cath Lab Digest talks to PACS Program Manager Tuan Bui about his experience.

Can you tell us about your role?

I’m currently the manager of picture archiving and communication system (PACS) at Advocate Healthcare. Advocate Health Care is the largest fully integrated health care delivery system in the state of Illinois. It operates more than 200 sites of care, including:

• 10 adult and 2 children’s hospitals with more than 3,300 beds
• The region’s largest medical group with more than 80 locations across metropolitan Chicago 
• 3.1 million annual patient visits 
• 350,000 annual emergency department visits 
• 30,000+ employees 

We have a multitude of PACS systems in both our acute and outpatient settings.  From radiology to cardiology and other departmental imaging systems, you name it, and I’ll probably have it somewhere in the system. As the Advocate PACS Program Manager, I serve a number of roles within the organization: (a) providing strategic planning and direction where PACS is treated as a core integral technology, (b) managing the operations of a central PACS support organization, (c) architecting consistent standards-based, interoperable systems in a highly available (HA) environment that accommodates business continuance (BC) and disaster-recovery (DR) across the Advocate enterprise, and (d) continual assessment of technology and software applications that would hopefully result in improved performance, efficiency, end-user satisfaction or lower operating costs. 

What did you do when Advocate’s growth caused you to re-evaluate data storage and access?

There is a continuing trend in the industry where imaging studies are getting larger as technology continues to evolve.  The same imaging study today can easily be two or three times larger than one performed a few years ago.  However, the patient experience and expectations do not allow for delays, and in many cases, the end user’s expectation is that the information is “ready and available” at their fingertips.

We recognized that we had data stored in many different formats and in many different locations.  Given the changes in health care delivery, the impending establishment of accountable care organizations (ACOs) and the need for more integration of care, one area we recognized we could improve upon was making prior images more available to clinicians. 

In the past, we commonly used DVDs to archive imaging studies onto jukebox archives.  The challenge here was that once the jukebox reached its media capacity, sites were required to “expel” DVD media from its casing.  The expelled media then required a resource to manage and maintain the physical storage and cataloguing of the media, more commonly known as “shelf management.” For us, this literally meant thousands of DVDs that needed to be catalogued, stored in boxes, and then managed. To further complicate matters, often times the storage was remotely located from clinical areas needing the information. This also meant that when clinical staff needed to view a prior, someone had to leave the lab, go to the shelf archive, find the right box, find the right media(s), return to the lab and “import” the study for viewing. Operationally, this process often required a resource that needed more than twenty minutes to provide the requested historicals back to the clinicians. 

We recognized that although adequate at one time, the DVD technology could not meet our performance expectations as we have defined them for today’s workflow.  Current demands placed upon us by clinicians to retrieve and view historicals (some dating as far back as 10-15 years) became a clinical and operational issue. 

When the opportunity presented itself during one of our cardiology upgrades, the decision was made to transition off our DVD technology to a spinning disk archive, as spinning disk is now considered the norm for archive, is relatively cheap compared to media, is fast, and can be easily replaced and managed on storage systems. For Advocate, this was not just a process improvement opportunity; it was an opportunity to make our associates more productive in other more clinical areas. 

What did you want to accomplish with the data migration?

We were looking for few things. Data integrity was primary. We wanted to make sure that we were not going to lose anything during the migration process. Secondly, after the data is moved, it had to be readily available to our end users. End users shouldn’t have to run back and wait for studies or patient historicals to become available. The volume of data we manage has dramatically increased over time. The same procedure that was performed back in 2005 would, today, based on technology advances and the ability to do a more detailed diagnosis, be as much as doubled or tripled in size. What was inherently a 75-135 megabyte (MB) study a few years ago has probably increased to about 130-225MBs per study today.  As file sizes increased, retrieval times increased and end-user satisfaction decreased. Our third requirement was the time required to migrate all our current data. By using normal migration methodologies, we estimated it could take us as long as two years to migrate 32 terabytes of data archived. The time to migrate is important, as it also dictated the time the clinical staff would have to work in a hybrid state (some spinning disk, some DVD). And, like all other projects, we had to give serious consideration to cost, operational downtime, and the technical details of managing the migration over such a length of time.

What did the DataFirst offer that was different?

We went through a few different migration considerations, including migration of data using the normal migration methodologies through the vendor and the original equipment manufacturer (OEM). The time it would take for each of these options was significant. DataFirst was appealing for a number of reasons. First, we concluded a “short” migration was preferable to a long migration, as we wanted to get off the old technology as fast as possible. It also would minimize any clinical “confusion” that may arise during the transition period. Secondly, we liked how DataFirst approached the migration of large volume data. DataFirst’s methodology involved bringing in spinning disk archives to mass migrate DVDs in a multiple, concurrent manner. In doing so, they could ingest multiple media simultaneously. Ultimately, DataFirst’s Hyper-Migration moved about 32 terabytes of data within 12 days. Finally, DataFirst was willing to work with us to define a process for checking the integrity of the data and then working with us to remedy any corrupt files. In the end, we were able to identify corrupt data down to the patient, study, and date level, as well as identify physically damaged media in whole or in part. We then replicated the spinning disk archive.

What was the timeframe it would have taken otherwise?

For 32 terabytes, using normal migration methodologies would probably have taken the better part of 18 to 24 months, assuming you have someone to manage it every day. We replicated the spinning disk archive to another volume within 7 days.

Were there any concerns with the safety of your data as it went through this process?

Absolutely — this was one of our highest priorities. We put some parameters in place to make sure that the data that we moved was clean, and that the patient data was not corrupt. Part of the process involved evaluating the actual physical integrity of the disks. DataFirst went in and allotted resources not only to prep the existing DVD data, but to prep all our historical data. We had 4,100+ 9GB DVD media. That’s quite a bit to go through, and we staged our efforts. First, we sought to physically fix the disks as much as possible. Second, part of the data migration process involved some checks and balances done by DataFirst. Reading the DICOM headers meant they could recognize how many images should be associated with a prior study and make sure all were present.

We did have some inherent database issues that were considered outside the scope of the migration process; you can’t migrate data unless the database is first considered to be accurate. So we had to do some database cleanup as our process evolved. The process of moving the data into the spinning disk archive was checked and rechecked by DataFirst, and they provided us with a count of studies that passed, a count of studies that didn’t pass, studies that partially passed, and studies with corruption or some other incorrect mechanism. DataFirst has a process as part of their migration methodology that identifies these problems. Very valuable.  One challenge we found was that people had used a Sharpie or some other permanent ink on the DVDs, and over time, that ink actually permeates into the DVD media, affecting our ability to retrieve the data. Other physical issues with DVDs that can give you problems include scratches, gouges, and storage environment — these aspects can all have an adverse effect on how data is stored on the DVD, how it is retrieved, and if there is any corruption to the media itself. When we did our first run, we went through 4,100+ DVDs and came up with 122 that could not be read at all due to the physical integrity of the disk itself.

As DataFirst ingested information on the media, part of their process involved letting us know which media had problems. This meant either they didn’t know exactly what was on it or they couldn’t read the whole DVD, but frequently they could identify, based on the document header, what should be there. Advocate Health Care had a very, very small percentage of disks we were unable to read, about 122 out of 4,100+. We have re-shelved these 122 disks, since they have patient data. They are kept in case there is something new in the future that can help move the information off to the spinning disk archive as well.

To further ensure the integrity of the data migration, Advocate Health Care used Six Sigma methodologies to identify the sample size necessary to achieve a certain confidence level in the migration process. We wanted to achieve 99.999% validity that we could get the migrated data back in a safe and usable manner. The total number of studies migrated was determined to be about 130,000+ studies, and Six Sigma methodology indicated that the sample size needed in order to determine validity was approximately 4,000. Four thousand studies needed to pass not only the retrieval process, but also a test of our ability to manipulate the data that was pulled back. Advocate Health Care spent more time on the validation and retrieval of the data than the migration itself, in order to make sure that we were able to quantitatively and qualitatively address any issues about retrieval timing as well as data integrity. As it turns out, out of all of the 4,000+ studies, we only failed to pull back one study, meaning we had a 99.95% or 99.96% confidence level. From a data migration standpoint, those numbers exceeded our expectations and certainly no one can argue the efficacy of the DataFirst data migration process.

How has the data migration affected Advocate Health Care?

Today, from an end user perspective, what took about 20-25 minutes for a DVD or a historical retrieval from jukebox archives is now down to seconds. And, when you take in the cost of not having a resource running down to the jukebox to do the shelf management, as well as both the import and export of the media in and out of the jukebox, we have saved on productivity time for staff, patients, and physicians, as well as on our internal resource costs to manage the jukebox and this type of data.

Can you share more about the logistics behind the DataFirst Hyper-Migration?

I thought the process was very, very smooth. DataFirst did a fantastic job of staging the equipment, bringing in the proper resources, and staffing their resources correctly to support the migration. The team was 24/7 for 12 days, and they actually had people staffed throughout that process. The DataFirst migration was extremely successful and I would not hesitate to use them again.

Tuan Bui can be contacted at tuan.bui@advocatehealth.com.