Can Adding Laboratory Values Improve Risk-Adjustment Mortality Models Using Clinical Percutaneous Cardiac Intervention Registry Data?

Abstract: Background. Registry data for percutaneous coronary intervention (PCI) are being used in New York and Massachusetts and by the American College of Cardiology to risk-adjust provider mortality rates. These registries contain very few numerical laboratory data for risk adjustment. Methods. For 20 hospitals, New York’s PCI registry data from 2008-2010 were used to develop statistic models for predicting in-hospital/30-day mortality with and without appended laboratory data. Discrimination, calibration, correlation in hospital’s risk-adjusted mortality rates, and differences in hospital quality outlier status were compared for the two models. Results. The discrimination of the risk-adjustment models was very similar (C-statistic = 0.898 from the registry model vs C-statistic = 0.908 from the registry/laboratory model; P=.40). Most of the non-laboratory variables in the two models were identical, except that the registry model contained malignant ventricular arrhythmia and the registry/laboratory model contained previous coronary artery bypass surgery. The registry/laboratory model also contained albumin ≤3.3 g/dL, creatine kinase ≥ 600 U/L, glucose ≥270 mg/dL, platelet count >350 k/µL, potassium >51 mmol/L, and partial thromboplastin time >40 seconds. The addition of laboratory data did not affect outlier status for better-performing hospitals, but there were differences in identifying the hospitals with significantly higher risk-adjusted mortality rates. Conclusions. Adding laboratory data did not significantly improve the risk-adjustment mortality models’ performance and did not dramatically change the quality assessment of hospitals. The pros and cons of adding key laboratory variables to PCI registries require further evaluation.  

J INVASIVE CARDIOL 2015;27(7):E117-E124

Key words: percutaneous coronary intervention (PCI), risk-adjusted mortality, laboratory data

________________________________________

Percutaneous coronary intervention (PCI) is one of the most commonly performed procedures in the United States. It is estimated that about 492,000 patients underwent PCI in 2010.¹ During the last two decades, public reporting of risk-adjusted outcomes has become prevalent, with the goal of building public trust and improving patient outcomes. Public reporting of risk-adjusted mortality after PCI has been mandated in New York State for more than 15 years.² In 2003, Massachusetts became the second state to require case-level reporting of PCI outcomes.³ Also, the American College of Cardiology’s CathPCI registry reports hospital-specific outcomes for diagnostic catheterizations and/or PCI procedures.⁴

Clinical registry data generally used for reporting and risk-adjusting PCI outcomes contain numerous risk factors, such as patient demographics, extent of cardiac disease, ventricular function, and a variety of comorbidities. However, there are very few laboratory measurements in the current New York PCI registry data. Therefore, details about the severity of certain comorbidities are lacking. The goal of this study is to determine whether the addition of laboratory data to New York’s PCI registry database will yield laboratory variables that are significant independent predictors of in-hospital/30-day mortality, and if so, whether including laboratory variables in the risk-adjustment model will significantly alter risk-adjusted mortality rates and mortality outlier status of New York hospitals performing PCIs. 

Methods

Endpoint. The endpoint in this study was short-term mortality, defined as mortality occurring during the index admission for PCI or within 30 days of the index procedure. Deaths were confirmed using New York’s vital statistics data. 

Databases. The primary database used for the study was New York State’s clinical registry for PCI, the Percutaneous Coronary Interventions Reporting System (PCIRS). This registry contains detailed information on patient demographics, preoperative surgical risk factors, hemodynamic state, left ventricular function, coronary vessels diseased and attempted, complications, procedure choices, provider identifiers, discharge status, and in-hospital adverse outcomes. 

Completeness of data reporting is monitored by matching PCIRS to New York’s acute care and ambulatory care administrative reporting system, the Statewide Planning and Research Cooperative System (SPARCS) to identify cases reported in SPARCS that were not reported in the PCI registry. SPARCS contains patient demographics, diagnoses, and procedures, admission and discharge dates, and discharge disposition. PCIRS records were matched with SPARCS records using unique facility identifiers along with patient identifiers and admission, surgery, and discharge dates. In-hospital outcomes also are matched to SPARCS to ensure accuracy. Finally, the New York State Department of Health (NYSDOH)’s utilization review agent audits samples of records from hospitals to ensure the accuracy of risk factor reporting. 

Patient identifiers in PCIRS were used to link the index procedure to New York State’s vital statistics data to identify deaths that occurred after discharge. Laboratory data were obtained from hospitals’ computerized laboratory databases and were appended to SPARCS data using patient identifiers. The appended SPARCS data were then matched to registry data in the same manner as the data completeness monitoring described above. 

Patients and hospitals. All New York hospitals that performed a minimum of 50 PCIs per year in 2008-2010 were invited to participate in the study. Twenty of 59 eligible hospitals agreed to participate and send their electronic laboratory data to the NYSDOH to be appended on a patient-specific basis to PCI registry data. 

Patients who were diagnosed as being in shock prior to PCI (n = 165) were excluded because these patients are not contained in public reports. 

Laboratory data. Thirty laboratory test results were initially examined. Only laboratory values from the first day of hospitalization were incorporated into risk factors to ensure that they reflected the condition of the patient at admission rather than complications occurring during the hospitalization. 

Admission numerical results for each laboratory test were used to assign PCI cases to categories based on ranges that were internally homogeneous with regard to mortality. Cases with missing data were identified and their relative frequency and mortality rates were examined for each laboratory test. Laboratory tests for which more than 90% of patients had a missing laboratory value were eliminated (n = 8). Variables excluded were amylase, neutrophils bands, base units excess, brain natriuretic peptide (BNP), lactate dehydrogenase, arterial pCO₂, arterial pH, and ProBNP. Four other variables (bilirubin, bicarbonate, blood urea nitrogen [BUN], and white blood cell count) were excluded because there was no discernable pattern in mortality rates across ranges of values. Values for creatinine already documented in the PCI registry were used instead of corresponding test results from hospitals’ electronic data repositories. For all other tests, potential risk factors were selected based on clinical judgment and empirical relationships between test results and short-term mortality rates. Cases with missing laboratory test results were combined with the categorical range corresponding to normal test results. 

Statistical analysis. In-hospital/30-day mortality for PCI was examined on a bivariate basis for numerous patient characteristics, including demographics, comorbidities, ventricular function, preprocedural myocardial infarction (MI), and hemodynamic state. Mortality rates associated with abnormal ranges were computed for laboratory tests, including albumin, alkaline phosphatase, serum aspartate aminotransferase (AST), calcium, creatine kinase (CK), CK-MB, creatine MB fraction, glucose, hemoglobin, international normalized ratio (INR), platelet count, potassium, partial thromboplastin time (PTT), sodium, troponin I, prothrombin time, and troponin T. BNP at least 3x normal and creatinine also were included, based on information from the PCI registry. 

The mortality rate was determined for each categorical range for each laboratory variable. Variables that were significant at the 0.10 level with a chi-square test were considered as candidate variables in two stepwise logistic regression models (one including all registry variables and one including registry and laboratory variables). The models were cross-validated by splitting the patients in half, fitting a model on the first half, and then testing it on the second half. The laboratory categorical variables identified using the first half of the data were then refit to the entire database. Variables with P-values <.05 in the logistic regression models fit on the entire database were retained for the final models.

Significant variables in the two logistic regression models were compared, and laboratory variables appearing in the second model were noted. C-statistics of the two models were computed to compare model discrimination.⁵ Calibrations of the two models were compared using the Hosmer-Lemeshow goodness of fit statistics.⁶ 

Each model was used to calculate risk-adjusted mortality rates and outlier status for each of the 20 hospitals in the database using the same methods used in New York’s annual PCI reports.⁷ Risk-adjusted mortality rates (RAMRs) based on the two models were compared by examining the outlier status of hospitals. Hospitals were designated as high outliers (low outliers) if their RAMR was significantly higher (lower) than the rate for all 20 hospitals combined. A three-by-three matrix was developed that compared tertiles of RAMRs for the two models. Finally, the correlation between hospital RAMRs obtained from the two models was computed.

All tests were two-sided and conducted at the .05 level; all analyses were performed using SAS 9.1 (SAS Institute). 

Results

The study included 51,817 patients who underwent PCI at 20 participating hospitals between January 1, 2008 and December 31, 2010. The overall in-hospital/30-day mortality rate was 0.64%.

The bivariate relationship between patient-level variables (registry variables and laboratory variables) and in-hospital/30-day mortality is shown in Table 1. As indicated, older age, female sex, body mass index (BMI) ≤30, lower ejection fraction, different variations of previous MI with regard to onset time and type (ST-elevation and non-ST elevation), cerebrovascular disease, peripheral vascular disease, hemodynamic instability, congestive heart failure, malignant ventricular arrhythmia, chronic obstructive pulmonary disease (COPD), renal failure (dialysis), stent thrombosis, contraindication to aspirin and clopidogrel, BNP >3x normal, and creatinine ≥1.2 mg/dL were registry variables that were significantly bivariately associated with increased short-term mortality. In addition, lower albumin, higher alkaline phosphatase, higher serum AST, higher BUN, lower calcium, higher CK-MB fraction, higher CK, higher glucose, lower hemoglobin, higher INR, higher platelet count, longer PTT, lower or higher sodium, higher troponin I, higher white blood cell count, higher CK-MB, longer prothrombin time, and higher troponin-T were laboratory variables that were significantly bivariately related to higher in-hospital/30-day mortality. 

Significant independent predictors of short-term mortality when only PCI registry data were used are presented in Table 2. As shown, age >60 years, female sex, BMI ≤30, ejection fraction ≤29%, different variations of MI with regard to onset time and type, hemodynamic instability, BNP at least 3x normal, congestive heart failure, malignant ventricular arrhythmia, COPD, higher creatinine levels (mg/dL) (1.2-1.5, 1.6-2.0, and ≥2.1 vs <1.2), and renal failure were significantly associated with higher in-hospital/30-day mortality rates. The registry model demonstrated excellent discrimination (C-statistic: 0.898) and good calibration (Hosmer-Lemeshow statistic: 6.07; P=.64). 

Table 3 presents the significant independent predictors of in-hospital/30-day mortality when the laboratory variables were added to registry data. Compared with the significant predictors from the registry model (Table 2), all aforementioned significant independent predictors except malignant ventricular arrhythmia remained significant factors in predicting short-term mortality for the registry/laboratory model (Table 3). Moreover, previous CABG from the registry data entered the registry/laboratory model along with the following laboratory variables: albumin, CK, glucose, platelet count, potassium, and PTT. The registry/laboratory model had excellent discrimination (C-statistic: 0.908) and acceptable calibration (Hosmer-Lemeshow statistic: 6.42; P=.60). The C-statistics from the risk adjustment models with and without the laboratory data were not statistically different (difference = 0.010; P=.40). Furthermore, the patient-level predicted probabilities of mortality from these two models were significantly strongly correlated (Pearson correlation coefficient: r = 0.89; P<.001). 

Figure 1 shows the hospital-level correlation of risk-adjusted in-hospital/30-day mortality rates from the registry and registry/laboratory models. As illustrated, risk-adjusted hospital mortality rates were strongly correlated (Pearson correlation coefficient: r = 0.90; R² = 0.85). Table 4 presents the correspondence of tertiles of hospital risk-adjusted in-hospital/30-day mortality rates based on the two models. Most hospitals (14 of 20) were rated in the same tertile for both models. One hospital was in tertile 1 for the registry model and tertile 2 for the registry/laboratory model, and one hospital was in tertile 2 for the registry model and tertile 1 for the registry/laboratory model. Two hospitals were in tertile 2 for the registry model and tertile 3 for the registry/laboratory model, and two hospitals were in tertile 3 for the registry model and tertile 2 for the registry/laboratory model. Regarding outlier status, the same hospital was the only low outlier (significantly lower risk-adjusted rate than the mortality rate for all 20 hospitals combined) for both models. High outliers (significantly higher risk-adjusted rate than the mortality rate for all 20 hospitals combined) differed; the registry model identified one hospital as a high outlier, which was different from the two high outliers identified by the registry/laboratory model. 

Discussion

Beginning in 1992, the PCI reporting system (PCIRS) was established in New York State. Annual and 3-year PCIRS reports have been publicly released since 1997.² In these reports, risk-adjustment models are used to estimate patients’ risks of in-hospital/30-day mortality for PCI procedure along with hospital and operator risk-adjusted mortality rates using the high-quality PCIRS registry database. The patient-level risk factors that are used as candidates for the statistical models include demographics, clinical parameters of ventricular function and extent of coronary artery disease, ST-elevation MI or non-ST elevation MI type, and a wide variety of preoperative risk factors. According to the most recent 2008-2010 New York PCI report, older age, female sex, hemodynamic instability, lower ejection fraction, variations of acute MI type and onset time, current congestive heart failure, COPD, malignant ventricular arrhythmia, renal failure requiring dialysis, higher creatinine, left main disease, and multiple-vessel disease were significantly associated with higher short-term mortality.⁷ 

With the exception of creatinine levels, laboratory values usually are not contained in the registry models. In the current PCIRS registry database, only two laboratory values are available: creatinine and BNP at least 3x normal. Since increasing evidence suggests that the addition of laboratory values can markedly improve risk-adjustment models for various conditions using administrative data,^8-10 an important question naturally arises as to whether the addition of laboratory data to other clinical registry data can enhance the performance of risk adjustment models. That is, when clinical registry models are used to assess hospitals’ quality of care, can laboratory variables significantly increase the predictive power of the models, and do these laboratory-enhanced clinical models alter assessments of provider quality? If the addition of laboratory data significantly improves risk adjustment of hospital mortality, this would support the addition of these significant independent laboratory values to the current clinical registry database. If there are notable differences in hospital mortality assessments identified by these two models, policymakers will have to make a decision which model should be used to identify risk-adjusted mortality rates and outlier hospitals. 

In 2007, Pine et al used Pennsylvania data to show that for five medical conditions (acute MI, congestive heart failure, acute cerebrovascular accident, gastrointestinal tract hemorrhage, or pneumonia) and three surgical procedures (abdominal aortic aneurysm repair, CABG surgery, or craniotomy), adding a “present on admission” indicator and selected numerical laboratory data collected at the time of admission substantially improved the C-statistic of logistic regression models predicting hospital mortality from 0.79 to 0.86 (P=.01).⁸ More recently, Tabak et al demonstrated that laboratory and vital sign variables greatly improved the power of administrative variables in predicting mortality for hospitalized pediatric patients.¹⁰ However, it remains unknown whether adding laboratory values to a clinical registry database will enhance the performance of risk-adjustment models in predicting hospital mortality. 

What we currently know is that statistical models using clinical registry database have been shown to predict mortality more accurately than models using administrative databases. For example, Hannan et al showed that the New York CABG registry database was considerably better in terms of predicting hospital mortality for CABG patients than the New York SPARCS hospital claims database.¹¹ In another study, Hannan reported that after removing complications of care (that had been mistakenly identified as comorbidities) from administrative models, there was a large difference in the discriminatory powers of a registry model and an administrative model for CABG patients (C-statistic: 0.789 vs C statistic: 0.732).¹² The main reasons why registry models perform better than administrative models are that registry databases are more accurate because of auditing and the use of more precise variable definitions. 

To the best of our knowledge, this study is the first one to investigate the effect of adding laboratory values to a PCI registry database for predicting hospital mortality. We found that the registry model and the registry/laboratory model had very similar discriminatory power (C-statistics: 0.898 for the registry model and 0.908 for the registry/laboratory model); this difference in model discrimination was not statistically significant (P=.40). In addition, the two models had comparable calibration (P=.64 for registry model vs P=.60 for registry/laboratory model). Almost all of the significant independent predictors from the registry data were identical in the two models, except that the registry model contained malignant ventricular arrhythmia and the registry/laboratory model included previous CABG. Furthermore, the registry/laboratory model contained six significant laboratory variables: albumin ≤3.3 g/dL, CK ≥600 U/L, glucose ≥270 mg/dL, platelet count >350 k/µL, potassium >51 mmol/L, and PTT >40 seconds.

Albumin is a protein produced by the liver; the serum albumin level can help determine whether a patient suffers from liver disease or kidney disease, or the body is not absorbing enough protein. The normal range is 3.4-5.4 g/dL, and lower values (≤3.3 g/dL) may be a sign of kidney or liver disease. In this study, 3.8% of PCI patients had an albumin ≤3.3 g/dL, and the crude in-hospital/30-day mortality rate for this group was statistically higher than the normal value group (3.18% vs 0.54%; P<.001). 

Creatine kinase is a measure of muscle damage; the serum CK levels may vary depending on patient demographics and other personal characteristics. The normal range for adults is 40-400 U/L; higher values may indicate muscle damage. In this study, 3.50% of PCI patients in the present study had CK ≥600 U/L; these patients had an increased crude mortality rate compared with patients who had lower values (3.26% vs 0.54%; P<.001). 

An abnormal glucose can be diagnostic of diabetes, or can indicate how well diabetes is controlled in those already diagnosed. The normal range of blood glucose usually is below 125 mg/dL. Higher levels may indicate that a patient has diabetes or that already-diagnosed diabetic patients are not well controlled. Although the PCIRS data do contain a variable for “diabetes requiring medication” (defined as a patient receiving oral hypoglycemic or insulin), this measure cannot capture how well the blood glucose level is being controlled. In this study, 2.32% of PCI patients had an extremely high glucose level (>270 mg/dL); the crude mortality rate for this group was much higher than for patients with normal blood glucose values (ie, glucose level ≤170 mg/dL) (3.50% vs 0.50%; P<.001). 

Platelet counts measure how many platelets are in the blood. Platelets are important in the clotting of blood. Platelets also are acute-phase reactants that increase in response to various noxious stimuli, including systemic infections, inflammatory conditions, bleeding, and tumors. The normal range for platelet count is 150-350 k/µL. In the present study, 2.37% of PCI patients had a platelet count >350 k/µL; the crude mortality rate for this group was significantly higher than for other patients (2.53% vs 0.59%; P<.001).

A potassium test measures the amount of potassium in the serum. Potassium plays an important role in nerve and muscle communication and helps move nutrients into cells and waste products out of cells. A normal range is 3.7-5.1 mmol/L. High levels of potassium may be caused by a variety of conditions, including renal failure, metabolic or respiratory acidosis, transfusion, and a high-potassium diet. Abnormally high potassium levels can cause severe cardiac arrhythmias. In this study, 1.69% of PCI patients had a potassium level >5.1 mmol/L; the crude mortality rate for this group was much higher than for other patients (3.08% vs 0.59%; P<.001). 

Abnormal PTT can indicate bleeding or clotting problems. In general, the normal PTT is between 25-35 seconds. Abnormally high PTT may be due to conditions such as disseminated intravascular coagulation, factor XII or factor XI deficiency, hemophilia A, hemophilia B, and liver disease. In this study, 6.12% of PCI patients had PTT >40 seconds; the crude mortality rate for this group was significantly higher than for other patients (1.86% vs 0.56%; P<.001). 

Study limitations. There are several limitations to our study. First, although there were a lot of laboratory variables considered for the registry/laboratory risk adjustment model, ten of them could not be considered for the statistical models because they were missing >90% of patients (n = 8) or because the pattern of mortality rates for different ranges had no discernable pattern (n = 2). In addition, it is very likely that there is hospital variability with regard to the measurement accuracy of laboratory measurements. Another caveat is that this study was limited to 20 of the 59 hospitals in New York where PCI was performed during the study period (2008-2010). These 20 participating hospitals may not be representative either of New York or of United States hospitals that perform PCI. 

Conclusion

The addition of laboratory data did not significantly enhance the discriminatory power or the calibration of New York’s PCI registry risk-adjustment models for in-hospital/30-day mortality, did not affect the outlier status for better performing hospitals, and had minimal effect on the tertile of risk-adjusted mortality of participating New York PCI hospitals. There were, however, moderate differences in identifying the hospitals with significantly higher risk-adjusted mortality rates. Moreover, several laboratory variables not contained in the PCI registry database (albumin, CK, glucose, platelet count, potassium, and PTT) were found to be significant predictors of mortality in the laboratory enhanced model. Additional studies to further explore the benefits of adding these laboratory variables to the PCI registry data are warranted. The challenges of collecting and adding these laboratory variables include acquisition costs, reporting accuracy, effects on inter-hospital comparisons of measurement quality, and the burden of auditing these laboratory variables for accuracy. 

Acknowledgments. This work was funded by a grant from the Agency for Healthcare Research and Quality (#R01 HS19965-01). The authors would like to thank Mary Lou Caprara, Kathryn J. Schmit, and John Piddock for their invaluable help in securing the data required for this study. The research team would like to thank all the participating hospitals on this funded project and hospital staff who submitted the laboratory data to NYSDOH. 

References

1. Go AS, Mozaffarian D, Roger VL, et al. Heart disease and stroke statistics — 2013 update: a report from the American Heart Association. Circulation. 2013;127:e6-e245.
2. Hannan EL, Cozzens K, King SB 3rd, Walford G, Shah NR. The New York State cardiac registries: history, contributions, limitations, and lessons for future efforts to assess and publicly report healthcare outcomes. J Am Coll Cardiol. 2012;59:2309-2316.
3. Resnic FS, Welt FG. The public health hazards of risk avoidance associated with public reporting of risk-adjusted outcomes in coronary intervention. J Am Coll Cardiol. 2009;53:825-830.
4. Brindis RG, Fitzgerald S, Anderson HV, Shaw RE, Weintraub WS, Williams JF. The American College of Cardiology-National Cardiovascular Data Registry (ACC-NCDR): building a national clinical data repository. J Am Coll Cardiol. 2001;37:2240-2245.
5. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29-36.
6. Hosmer DW, Lemeshow S. Applied Logistic Regression. 2nd ed. New York: John Wiley and Sons, Inc; 2000.
7. NYSDoH. Percutaneous Coronary Interventions in New York State 2008-2010. 2012.
8. Pine M, Jordan HS, Elixhauser A, et al. Enhancement of claims data to improve risk adjustment of hospital mortality. JAMA. 2007;297:71-76.
9. Tabak YP, Sun X, Derby KG, Kurtz SG, Johannes RS. Development and validation of a disease-specific risk adjustment system using automated clinical data. Health Serv Res. 2010;45:1815-1835.
10. Tabak YP, Sun X, Hyde L, Yaitanes A, Derby K, Johannes RS. Using enriched observational data to develop and validate age-specific mortality risk adjustment models for hospitalized pediatric patients. Med Care. 2013;51:437-445.
11. Hannan EL, Kilburn H Jr, Lindsey ML, Lewis R. Clinical versus administrative data bases for CABG surgery. Does it matter? Med Care. 1992;30:892-907.
12. Hannan EL, Racz MJ, Jollis JG, Peterson ED. Using Medicare claims data to assess provider quality for CABG surgery: does it work well enough? Health Serv Res. 1997;31:659-678.

________________________________

From the ¹Department of Health Policy, Management, and Behavior, University at Albany, State University of New York, Albany, New York; ²Michael Pine Associates, Chicago, Illinois; and ³New York State Department of Health, Albany, New York.

Funding; This work was funded by a grant from the Agency for Healthcare Research and Quality (#R01 HS19965-01).

Disclosure: The authors have completed and returned the ICMJE Form for Disclosure of Potential Conflicts of Interest. The authors report no conflicts of interest regarding the content herein.

Manuscript submitted October 1, 2014 and accepted October 6, 2014.

Address for correspondence: Feng Qian, MD, PhD, Assistant Professor, Department of Health Policy, Management, and Behavior, School of Public Health, University at Albany – State University of New York, One University Place, Room 169, Rensselaer, NY 12144. Email: fqian@albany.edu