Predictive Validity and Reliability of the Turkish Version of the Risk Assessment Pressure Sore Scale in Intensive Care Patients: Results of a Prospective Study

Multiple pressure ulcer (PU) risk assessment instruments have been developed and tested, but there is no general consensus on which instrument to use for specific patient populations and care settings.   The purpose of this study was to determine the reliability and predictive validity of the Turkish version of the Risk Assessment Pressure Sore (RAPS) instrument, which includes 12 variables — 5 from the modified Norton Scale, 3 from the Braden Scale, and 3 from other research results — for use in intensive care unit (ICU) patients. The English version of the RAPS instrument was translated into Turkish and tested for internal consistency and predictive validity (sensitivity, specificity, positive predictive value, and negative predictive value) using a convenience sample of 122 patients consecutively admitted to an ICU unit in Turkey. The patients were assessed within 24 hours of admission, and after that, once a week until the development of a PU or discharge from the unit. The incidence of PUs in this population was 23%. The majority of ulcers that developed were Stage I. Internal consistency of the RAPS tool was adequate (Cronbach’s a = 0.81). The best balance between sensitivity and specificity for ICU patients was reached at a cut-off point of £27 (ie, sensitivity = 74.2%, specificity = 31.8%, positive predictive value = 38.7%, and negative predictive value 91.3%). This is lower than the cut-off point reported in other studies of the RAPS scale. In this population of ICU patients, the RAPS scale was found to have acceptable reliability and poor validity. Additional studies to evaluate the predictive validity and reliability of the RAPS scale in other patient populations and care settings are needed.

Pressure ulcers (PUs) have a profound impact on patient quality of life, cause pain and discomfort, and may result in prolonged hospital stays.^1,2 Prevention of PUs should be a priority in nursing care.

Although a prospective study³ has shown most PUs can be prevented by accurate prediction and appropriate nursing interventions, PU incidence is reported to range from 1% to 56% in intensive care units (ICUs).^4,5 Only limited prevalence and incidence studies have been conducted in Turkey: a descriptive study⁶ conducted on 922 patients found the PU prevalence at a university hospital in Turkey was 7.2%; a prospective, analytic, descriptive study⁷ conducted among 84 postoperative surgical patients found the incidence rates to be 54.8%; and in a descriptive, prospective study⁸ involving 140 ICU patients, incidence was 14.3%.

Treating PUs can be extremely costly.⁹ A comparative study has estimated the cost of treatment to be 2.5 times higher than the cost of prevention.¹⁰ Thus, all clinicians should focus on preventing PUs.

Low PU incidence rate is considered an indicator of quality nursing care.¹¹ In order to prevent PUs, individuals at risk should be identified accurately and preventive measures should be initiated. The prompt and accurate identification of risk factors associated with PU development is essential for effective prevention^12,13; accurately determining PU risk using a valid PU risk assessment tool is crucial.

A PU risk tool should be sufficiently competent to distinguish between patients who are and those who are not at risk to appropriately target preventive nursing interventions. Multiple tools have been developed to help nurses identify PU risk. Use of validated risk assessment tools (eg, Norton, Braden, Waterlow, Gosnell) is recommended, but which tool is best suited to a particular care setting has yet to be determined.¹⁴ A descriptive study by Defloor and Grypdonk¹⁵ conducted among 1772 elderly people in long-term care facilities suggested the use of the Braden and Norton scales leads to 80% of patients receiving unnecessary preventative measures but that both scales were better at predicting PU than the clinical judgment of nurses. Such results infer more validity studies of the tools should be conducted.

Lindgren et al¹⁶ developed and tested the Risk Assessment Pressure Sore (RAPS) scale for an English-speaking Swedish population. It is composed of 12 variables based on risk factors in the Norton Scale, the re-Modified Norton Scale, and the Braden Scale: general physical condition, activity, mobility, food intake, fluid intake, moisture, sensory perception, friction and shear, skin type, body constitution, body temperature, and serum albumin. The authors tested the instrument’s ability to predict PUs among 530 patients in medical wards, wards for the treatment of infection, and nursing home residents. Results showed the Swedish and Norwegian version of the RAPS tool are reliable and valid for predicting PU risk.^16,17

A direct translation of scales into other languages does not guarantee maintenance of validity.¹⁸ If a scale is to be used across cultures, it should be translated to be linguistically accurate and adapted to preserve content validity in different populations.¹⁹ Moreover, in order to decide which tool is the most appropriate for a particular clinical area, the predictive validity of a tool must be considered when piloting use of risk assessment tools or reviewing the literature. The predictive validity assesses a predictive tool’s ability to differentiate individuals who are at risk from those who are not by measuring the tool’s sensitivity and specificity.²⁰ The validity and reliability of the RAPS scale was tested in a heterogeneous population¹⁶ but did not include ICU patients. The purpose of this study was to determine the reliability and predictive validity of the Turkish version of the RAPS scale in intensive care patients.

Design and sample. This methodological study was carried out in the anesthesia and reanimation clinic of a university hospital in İzmir, Turkey from October 2013 to November 2014. All patients newly admitted to this unit at the study start were enrolled. Eligibility criteria for the study stipulated 1) being 18 years or older, 2) having an expected hospital stay of at least 7 days, and 3) no PU on admission to the unit. Patients with existing PUs on admission were excluded; comorbidities were not a consideration other than as they affected items in the scale. The Ethics Committee of Ege University Faculty of Nursing and the participating hospital approved the study. All participants were informed about the research and the fact they could withdraw from the study at any time.

Instruments. Three tools were used to collect data: a patient information form, the RAPS scale, and the National Pressure Ulcer Advisory Panel (NPUAP) Pressure Ulcer Classification System.

Patient information form. This form included 3 questions about the patient’s gender, age, and primary diagnosis. It was completed by the researcher using patients’ records.

RAPS scale. The RAPS scale consists of risk factors included in the Norton Scale,²¹ the re-modified Norton Scale,^22-24 and the Braden Scale.^25,26 Developed by Lindgren et al,¹⁶ it incorporates 10 variables. Nine were scored from 1 to 4, where 1 = the worst situation and 4 = the best situation: general physical condition, physical activity, mobility, moisture, food intake, fluid intake, sensory perception, body temperature, and serum albumin level. The remaining question addresses friction and shear and was scored from 1 to 3, where 1 = large problem and 3 = no problem. Skin was inspected and PUs classified from Stage I to Stage IV, with responses incorporated as a part of the scale — ie, PU status was assessed as part of the study, but that variable is not included in the total score, which is 39. Lower scores indicate greater risk for PU development. An optimal cut-off point of 31 for determining when a resident is at risk for PU was found for the Swedish version of the RAPS scale for medical patients and persons with infectious diseases. Cronbach’s a of 0.82–0.83 was reported by Lindgren et al.¹⁶ For the Norwegian version, Fossum et al²⁷ reported the intraclass correlation coefficient (ICC) of the RAPS was 0.95.

NPUAP Pressure Ulcer Classification System.²⁸ Since 1989, the NPUAP’s staging system has been one of the most widely used PU classification systems. Stage I refers to an area of intact skin with nonblanchable erythema. Stage II refers to partial-thickness loss of dermis presenting as a shallow open ulcer with a red-pink wound bed without slough. Stage III refers to full-thickness tissue loss; subcutaneous fat may be visible, but bone, tendon, or muscles are not exposed. Stage IV refers to full-thickness tissue loss with exposed bone, tendon, or muscle.²⁸ A recent systematic review by Kottner et al²⁹ of 24 studies on NPUAP staging reports Spearman P values ranging between 0.39 and 1.00 and k values between 0.12 and 0.97 (95% confidence interval [CI]).

Procedures.
Translation of the scale into Turkish. Lindgren (the tool’s author) provided permission for use of the RAPS instrument. First, the language validity of the RAPS scale was studied. The scale was translated from English to Turkish by 4 nursing instructors and 1 English language specialist. The authors compared 5 translated versions to develop the text in Turkish. Then, the initial translation into Turkish was translated back into English by the English language specialist who had not seen the original English text. The scale’s statements, translated from Turkish into English, were compared with the original statements and necessary revisions were made.

Content validity. A panel of 10 content experts with professional expertise in PUs (5 intensive care nurses and 5 academician nurses) evaluated clarity and face content validity of the translated version, mindful of the items’ understandability and appropriateness for the purpose. No changes were necessary.

Pilot test. To identify potential problems in the Turkish version of the RAPS, a pilot study was undertaken using 30 nurses selected from the target population. In this preliminary testing of the scale, the nurses used the instrument and suggested wording changes, and these changes were incorporated into the final version of the tool. To test interrater reliability, 3 ICU nurses who were trained to use the scale assessed 30 patients concurrently and independently.

Data collection. One of the researchers who was familiar with the RAPS scale collected the data in the ICU by assessing the patient and using information from the patients’ records.

The patients were assessed within 24 hours of admission, and after that, once a week until the development of a PU or discharge from the unit. The patients’ skin condition was inspected on admission and once a week during hospitalization. The research data were stored securely by using a USB drive. Patient anonymity was maintained using a code name.

Data analysis. Statistical analysis was performed using SPSS 16.0 (SPSS Inc, Chicago, IL, USA). Descriptive statistics were utilized to examine the distributions of demographic and patient data. The area under the curve (AUC) is a common criterion for determining the prognostic validity of a scale (the higher the AUC, the better the scale). To determine the association between PU susceptibility and the occurrence of PUs, the AUC was computed for the scale. In a next step, the sensitivity and specificity were calculated for best cut-off point.

Reliability. To test internal consistency, Cronbach’s α coefficient was used. ICC was used to determine equivalence between 2 measurements for 30 patients.

Validity. In order to estimate predictive validity, sensitivity, specificity, positive predictive value, and negative predictive value were tested. Sensitivity refers to the proportion of actual positives correctly identified as patients at risk who developed PUs. Specificity is the proportion of negatives correctly identified as not being at risk that did not develop PUs. Positive predictive value is the proportion of PUs that developed among persons classified as at risk. Negative predictive value refers to the proportion of patients not having PUs among persons classified as not being at risk.³⁰

Sample characteristics. A total of 146 patients were screened for participation; 24 were excluded because they did not meet eligibility requirements. Of the 122 participants who provided written consent, 52 were women and 70 were men, mean age 56.5 ± 18.6 years. Mean length of stay in the ICU was 10.4 (SD 5.3) days. Mean length of patient follow-up was 18.2 days (SD 4.9). Among the 122 patients, 59 (48.6%) had respiratory failure and 38 (31.1%) had postoperative complications. A total of 31 PUs developed in 28 of the 122 patients. The average RAPS score of the patients who developed PUs was 19.06 (SD 2.1). The incidence of PU was 23%. Of the 31 ulcers, 21 (67.7%) were Stage I, 9 (29%) were Stage II, and 1 (3.3%) was Stage III. Fifty percent of ulcers were observed between 2 and 5 days after admission to the unit (see Table 1).

Reliability analysis. Internal consistency was found to be adequate (Cronbach’s a = 0.81). ICC between observers for total RAPS scores was 0.92 (95% CI 0.82–0.96, P <0.001, n = 30). The ICC values regarding the item level are listed in Table 2.

Validity analysis. As shown in Table 3, AUC value was 0.50 for the RAPS scale. The best balanced cut-off point for the scale was £27.

All patients in ICUs should be provided appropriate nursing care to prevent PUs. The incidence of PUs in the total sample was 23%. Although this is not in accordance with incidence found in a study performed in Turkey, where the incidence was 14.3% in ICU patients, these results underscore the problem of patients developing PUs remains a significant one in nursing care. In addition, clinical guidelines recommend using a valid and reliable risk assessment instrument to accurately determine patients at risk for PU and to provide appropriate care to these patients to prevent development of a PU. In this context, various relevant assessment tools are available to identify patients at risk.³¹ The Braden Scale, one of the most widely used risk assessment tools, has been tested in various settings. However, few validity evaluations have been conducted among ICU patients; in addition, results of validation studies conducted in ICUs were limited or inconclusive because of inadequate sample sizes.³² A recent systematic review³³ of risk assessment scales found the Braden Scale had the optimal validation and the best sensitivity/specificity balance (57.1%/67.5%) when compared to the Norton Scale (46.8% /61.8%) and the Waterlow Scale (82.4% /27.4%).

Generally, to verify diagnostic accuracy, the assessment tool is compared with a well-proven method. However, it is difficult to establish such a method because the diagnosis is not present at the time of assessment. Therefore, the occurrence of the incident is used as an external criterion to prove the predictive validity of risk assessment tools.³¹ In this study, the validity of the tool refers to criterion validity that describes the relationship between a test score and the occurrence of an event, and it is generally stated as sensitivity, specificity, and a positive predictive value and a negative predictive value. The first criterion for the best cut-off point was acceptable values for sensitivity and specificity of at least 70%; the second was to choose the cut-off point with the highest sensitivity. If the first criterion could not be reached, the cut-off point for an acceptable sensitivity of at least 70% was chosen, and the corresponding specificity was determined. The same was performed separately for a specificity of at least 70%.³⁴ In this study with this population, the best balance between sensitivity and specificity for ICU patients was reached at a cut-off point of £27 — that is, sensitivity 74.2%, specificity 31.8%, positive predictive value 38.7%, and negative predictive value 91.3%. In terms of specificity and positive predictive value, the RAPS scale showed low values. The AUC was 0.50, demonstrating low discriminatory power. However, the findings of the current study do not support previous recommended cut-off levels. In addition, a higher AUC value was reported in a cross-sectional descriptive study conducted on 346 patients in a general hospital setting by Källman and Lindgren.¹⁷ Lindgren et al¹⁶ reported predictive validity of RAPS at a cut of point of 31; the best balance for medical patients achieved was sensitivity 75%, specificity 70%, positive predictive value 19.2%, and negative predictive value 96.7%. Similarly, another methodological study³⁵ conducted in Ludhiana on 100 medical and surgical patients suggested a cut-off point of >35 for critical care patients.

Reliability has been defined as the consistency between independent measurements of the same thing.¹⁸ Cronbach’s avalue, calculated to determine the internal consistency of the Turkish version of the RAPS, indicates a high level of reliability (Cronbach’s a = 0.81). Interrater reliability was explored using ICCs for 30 patients. The ICCs are interpreted as strong when the value is >0.75, modest 0.40–0.74, or poor 0.00–0.39.³⁶ The levels of agreement on each item using ICCs in the RAPS scale varied between 0.58 and 0.92 and indicated satisfactory values between 2 nurses. Some ICC values were found to be moderate regarding certain factors (general physical condition, moisture, food intake, fluid intake and sensory perception). Similarly, lower values were demonstrated by Fossum et al²⁷ among 481 Norwegian nursing home residents. A possible explanation may be difficulties in the rating of these items between nurses.

This study was performed in a single ICU in order to standardize nursing care. Patient characteristics and PU prevalence in this unit, as well as the small sample size, may have affected the predictive validity of the tool. Further research on a larger sample and in different units is needed to generalize findings. Also, further study could compare the predictive validity of the tool among other risk assessment tools in ICUs.

In this study, the RAPS scale was found to have acceptable reliability and poor validity, achieving either good sensitivity with a low specificity or vice versa to detect ICU patients at risk for PU development among a patient sample with a PU incidence of 23%. For the practical use of the RAPS scale, the clinician will need to decide which property should be given priority: either to identify most of the patients at risk (with a low cut-off point) with many false-positive cases or to determine most of the patients who are not at risk (with a high cut-off point), potentially leaving many patients who are at high risk undetected. Additional research is needed to evaluate if the RAPS scale can, or should, be used in lieu of other risk assessment scales and in which patient population.

Dr. Günes is an Associate Professor and Ms. Efteli is a Research Assistant, Ege University Faculty of Nursing, İzmir, Bornova, Turkey.

Please address correspondence to: Ülkü Yapucu Günes, PhD, Ege University Faculty of Nursing, Kazımdirik Street, 35040 İzmir, Bornova, Turkey; email: ulkugunes@yahoo.com.