ADVERTISEMENT
Clinical Quality Indicators of Venous Leg Ulcers: Development, Feasibility, and Reliability
Studies on venous leg ulcer patients show large variations in the use of proven efficacious diagnostic and therapeutic modalities.1-4
This suggests that the quality of medical technical care — eg, provision of appropriate prevention, diagnosis, treatment, and rehabilitation5 — is far from optimal. Paradoxically, even though the medical technical care is the core product of any healthcare organization treating venous leg ulcer patients, as well as the goal of healthcare professionals and the most important concern of venous leg ulcer patients6 — quality in relation to medical technical care is seldom measured.
To ensure the provision of high quality venous leg ulcer medical technical care, most healthcare institutions are periodically evaluated/accredited to verify that the structure and organization to deliver that care are in place. However, this does not provide information about the actual diagnosis and treatment patients receive in everyday clinical life (ie, to what extent are patients provided the diagnosis and treatment to which they are entitled). For instance, internationally, it appears most facilities are not able to document their own healing or recurrence rates and therefore are not able to determine how well they actually are performing. Clinical quality indicators of medical technical care are needed.
Clinical quality indicators are quantitative measures used to monitor and evaluate the quality of important clinical functions that affect patient outcomes of care.7 Documenting quality of care using clinical indicators provides a statistically valid and data-driven mechanism that generates a continuous stream of performance information8,9 provided the clinical indicators are developed and tested using rigorous scientific principles.10,11 This enables clinicians to determine current patient status, set appropriate goals, and evaluate progress toward set goals in a valid and reliable fashion, an ability that is sorely needed in the art of healing venous leg ulcers. However, no published reports of rigorous quality indicator development for venous leg ulceration appear to exist. The purpose of this paper is to describe the scientific development of evidence-based clinical quality indicators of medical technical care for patients with venous leg ulceration and to show that quality of care can be measured reliably using a few meaningful clinical indicators.
Materials and Methods
Literature search and evaluation. A quality indicator development process was developed. A medical scientific literature search was conducted for potential clinical quality indicators related to venous leg ulceration using the following terms: venous, varicose, stasis, ulcer, venous insufficiency, leg, and limb. The search was a combination of database and hand searches conducted in cooperation with a documentalist from the Danish National Library of Science and Medicine. Fifteen medical databases and six indicator databases (first entries August 2001) were searched for studies using indicators related to venous leg ulcers (see Table 1).
Study eligibility was determined based on whether information on the following issues was included: 1) confirmed venous leg ulceration (eg, venous diagnosis was based on an objective physiological assessment method); 2) clinically relevant outcomes or efficacy measurements concerning diagnosis, treatment, or quality of care; and 3) English or Scandinavian language.
The included studies were evidence graded12 and the validity of the published results were evaluated according to the National Health Service Center for Review and Dissemination Guidelines13 and The International Task Force On Chronic Venous Disorders of the Leg.14
Indicator rankings. Potential indicators found in the literature were summarized on spreadsheets to enable indicator rankings. The spreadsheets were organized into 14 areas of care relevant to venous leg ulcers and specified the type, expression, and use of the potential indicators in terms of scientific evidence for the individual indicator and timeframes employed in the research studies. The spreadsheets were mailed to a panel of experts in chronic wounds who were asked to rank the clinical indicators according to importance, validity, relevance, and feasibility.
Panel participants. The panel comprised nine Scandinavian experts in chronic wound healing and included two specialists in dermatology, two specialists in vascular surgery, one expert in wound healing research, one general practitioner, one general surgeon, one plastic surgeon, and one nurse specialized in chronic wound healing . The panel was international, multidisciplinary, inter-professional, and cross-sectional to obtain the views of different types of professional healthcare workers involved in the diagnosis, treatment, and care of venous leg ulcer patients.
Indicator development process. The objective of the development process was to identify practical, feasible, clinically relevant quality indicators of the medical technical quality of venous leg ulcer care. In addition, the panel aimed to determine standards of good clinical practice in relation to the clinical quality indicators. Prognostic factors were chosen in order to support the interpretation and comparison of the indicator results by enabling adjustment for case mix.
At the first meeting, the panel selected preliminary clinical indicators based on the rankings of outcome measurements found in the literature and consensus among members of the expert panel. Consensus was achieved using the nominal group consensus method15 and experienced audit moderators.
The meeting enabled the panel to select additional clinically relevant indicators of quality not represented in the ranking material. Further, relevant prognostic factors of outcome were selected. The meeting was audiotaped and attendees were surveyed to confirm equal participation of the experts in the discussion and selection process.
Following the expert conference, an evidence report was generated that detailed the scientific research evidence base of the indicators and prognostic factors. This was mailed to the expert panel before a second conference took place.
At the second conference, each clinical quality indicator was discussed in detail in relation to the evidence base, current clinical results, relevance, feasibility, and practicability of measurement in the clinical setting. On the basis of these discussions, the panel defined and specified the clinical quality indicators and prognostic factors in detail and recommended data source and data collection strategies to foster implementation in practice. Finally, the panel determined standards of good clinical practice for each of the indicators on the basis of evidence in the literature.
Pilot study. A pilot feasibility study was conducted by the first author and an independent medical doctor to test the indicators in clinical practice. The quality of the medical technical care was assessed on 100 consecutive patients with chronic venous leg ulceration according to the clinical indicators and standards of good clinical practice selected by the expert panel. The patients were diagnosed and treated between September 2000 and May 2001 at the Copenhagen Wound Healing Center (CWHC), an independent, multidisciplinary, inter-professional facility with a large patient flow that specializes in the diagnosis, treatment, and care of chronic wounds.16
Among data sources were patient interviews and medical records (including notes from referring doctors), medical records from the CWHC and case record summaries from facilities involved in the treatment before reference to the CWHC, nursing records, blood tests, physiological tests, and discharge summary letters. The case records were blinded and assessed retrospectively in random order. Information not systematically registered in the medical records was collected through telephone interviews with the patients. The interviews were performed 2 to 4 months after discharge.
Indicator assessments were dichotomous (ie, fulfillment or non-fulfillment of the clinical indicator). Healing was defined as “restoration of sustained function and anatomic continuity”17 and recurrence as occurrence of any ulcer on the specific leg after healing.18
Statistics. An overall assessment indicator score was calculated by adding the number of indicators that met the standards. The maximum indicator score was 9 points. The results were calculated in three ways: per ulcer, per leg, and per patient (first ulcer that appeared was entered into the calculations) in order to investigate whether the assessment results varied according to definition of a treatment entity (unit of analysis).
Clinical indicator compliance was investigated in a two-step procedure — once by the first author as a test-retest and once by an independent medical doctor. Inter- and intra-rater variability were examined using the kappa statistic. The interpretation of the kappa coefficients followed the guidelines of Landis and Koch.19 The data were otherwise analyzed descriptively using the SPSS 8.0, P-value where 5% was considered significant.
Results
Clinical indicator identification. The literary search generated 785 clinical studies, of which 178 met the inclusion criteria. Sixty-eight were randomized, clinically controlled trials and 110 were quasi-experimental studies.
The literary search identified 143 individual potential clinical indicators classified in 14 quality topics (see Table 2). The panel selected three outcome indicators and identified four additional process indicators of presumed correlation to outcome from the ranking material. Additionally, 11 prognostic factors with a potential effect on results of care were selected.
Pilot study. The quality assessment included 98 of the 100 patients (118 legs, 127 ulcer episodes) (see Table 3). Two patients were excluded due to insufficient data in the medical records. Of the 98 participating patients, 20 (20%) had bilateral wounds, nine (9%) had an ulcer recur during follow-up at the CWHC, 26 (26.5%) suffered from superficial venous insufficiency, and 48 (49%) had mixed superficial and deep venous insufficiency. The type of venous insufficiency was unknown in 24 (24.5%) patients.
Information regarding the indicators “recurrence within 3 months” and “ulcer related pain” was obtained by patient interview because data were not systematically registered in the case records. “Recurrence within 12 months” data were not obtainable because the timeframe exceeded the study. Data collection relevant to other indicators proved unproblematic.
Quality assessment according to the clinical indicators found that within 3 months of the initial assessment date, 31 patients (32%) were assessed for venous surgery and 61 patients (62%) had a Duplex-verified venous diagnosis. Arterial pressure measurement was prescribed for 33 patients (34%); compression therapy was prescribed for all patients. Ulcers were healed in 12 months in 72 of the 98 patients (73%). No statistical differences in healing of subgroups were found. Leg ulcers recurred in 11 (11%) patients within 3 months of initial healing. Further, ulcer-related pain was treated appropriately in 57 patients (58%) (see Table 2).
The definition of treatment entity (unit of analysis) did not alter the assessment results. The overall assessment scores calculated per patient, per leg, or per ulcer were the same — per patient: 0.59, per leg: 0.60 and per ulcer: 0.60. Similar findings were disclosed with respect to the individual indicator results.
The overall inter- and intra-rater reliability was high (kappa = 0.79 P <0.01 and kappa = 0.89, P <0.01, respectively) (see Table 4 and Table 5). Apart from the indicators “healing within 12 months” and “vascular assessment,” which showed a fair and moderate inter-rater reliability, the intra- and inter-rater reliability was good or very good for all other indicators.
Discussion
In the present investigation, the indicator strategy to measure the quality of venous leg ulcer care was tested and proved to be feasible and reliable.
Seven quality indicators of venous leg ulcer care were developed using acknowledged scientific methodology.20, 21 This method combines a systematic review and synthesis of the literature with rankings and consensus development in an expert panel based on scientific evidence and, where gaps in the evidence are present, on the expert’s own experiences. The approach has previously been shown to be externally valid22-25 and has generated face validity and reliable clinical quality indicators in other medical fields.26 However, one drawback seems to be the lack of robust evidence of validity and reliability of consensus methods.27,28
The reliability of clinical quality indicators depends on 1) agreement on interpretation by the indicators, 2) the training of the data abstractors, 3) the time available, and 4) the professional background of the abstractors.29 In this investigation, the selected indicators proved feasible and reliable to measure because the majority of the indicators showed a fully satisfactory inter- and intra-rater reliability.
The approach to data collection, however, has some limitations. Some aspects of the indicators depend on the level of documentation by the health professionals in the medical records. However, the level of documentation is, in itself, an indicator of performance; poor documentation may represent poor quality patient care.30 In addition, survey data among patients are influenced by overstatement, recall bias, telescoping, and social acquiescence, resulting in bias of misclassification.31
Some factors may have affected the level of quality negatively. The case note review of the patient records revealed that in many situations a procedure was provided but not within the timeframe set in the indicator. Thus, the measured standard not only reflected the provision of appropriate care alone but also the timeliness of the delivered care. The ability to measure both the appropriateness and the timeliness of care is unique for clinical quality indicators.
Information in the medical literature regarding the provision of specific processes in clinical settings is scarce and consequently provides a weak basis for comparison. Although a number of quality problems were identified and the CWHC did not meet all standards of good clinical practice, the results from the CWHC are comparable to or surpass results from other clinical settings.32-34
A key issue in quality measurement is to ensure that standards, against which actual care is compared, represent normative medical care — ie, that the standards are valid. The standards of good clinical practice in this investigation were based on best scientific evidence when present; otherwise, they were based on consensus of a panel of experts. However, the quality of the studies was generally low and the majority was seriously methodologically flawed. The studies were based on small patient numbers and small ulcer sizes. The research results varied considerably and the variability grew more obvious with declining evidence grade of the studies. Therefore, relying heavily on the evidence of trials may create unrealistic expectations of performance, resulting in standards of good clinical practice that are unrealistically high and which no clinical unit can attain.35 These problems can be ameliorated using the quality indicators in practice by determining benchmarks based on comparisons of quality measured consecutively and in several units. Only when other clinical settings commence to document their quality of care can realistic standards be set and meaningful comparisons performed. Without performance measurements, determining how well jobs are performed, what areas could be improved, or if improvement activities yield the desired results is not possible.
Conclusion
Evidence-based, clinically relevant, reliable, and feasible quality indicators of venous leg ulcer care were identified. The quality of care provided in a multidisciplinary, specialized wound healing center was assessed according to the quality indicators and found to be in accordance with or surpassing results from other clinical settings. However, not all standards of good clinical practice were met, which raises questions regarding the current mode of standard determination and calls for benchmarks based on consecutive quality measurements in several other clinical settings. Documenting the quality of venous leg ulcer care using clinical quality indicators is a feasible and reliable strategy to ensure that desired outcomes of care are attained.
Acknowledgments
The authors would like to express their sincere gratitude to Christina Lindholm, Rolf Jelnes, Lasse Strand, Mats Bjellerup, Olle Nelzén, Mogens Thomsen, Henrik Kjaerullf, Tonny Karlsmark, and Finn Gottrup for their work as part of the expert panel and also Ditte Saunté for her help in indicator retesting. This study was supported financially by Copenhagen Wound Healing Center and the Dagmar Foundation, for which the authors are grateful.
1. Moffatt CJ, Franks PJ, Oldroyd M, Bosanquet N, Brown P, Greenhalgh RM, et al. Community clinics for leg ulcers and impact on healing. BMJ. 1992; 305:1389–1392.
2. Cornwall JV, Dore CJ, Lewis JD. Leg ulcers: epidemiology and aetiology. Br J Surg. 1986;73:693–696.
3. Bale S. Community nurses’ awareness of dressing materials. Care. 1989;7:90–92.
4. Lindholm C, Bjellerup M, Christensen OB, Zederfeldt B. Leg and foot ulcers. Nursing care in Malmo, Sweden. Acta Derm Venereol. 1992;72(3):224–226.
5. Donabedian A. The quality of care. How can it be assessed? JAMA. 1988;260(12):1743–1748.
6. Kjaer ML, Mainz J, Soerensen LT, Karlsmark T, Gottrup F. Venous leg ulcer patient priorities and quality of care: results of a survey. Ostomy Wound Manage. 2004;50(1):48–55.
7. Joint Commission on Accreditation of Health Organizations (JCAHO). Primer on Indicator Development and Application. Measuring Quality in Healthcare. Chicago, Ill.: JCAHO;1990.
8. Brook RH, Davies-Avery A, et al. Assessing the quality of medical care using outcome measures: an overview of the method. Med Care. 1977;15(suppl):165.
9. McGlynn EA. The outcomes utility index: will outcomes data tell us what we want to know? Int J Qual Health Care. 1998;10:485–490.
10. Rubin HR, Pronovost P, Diette GB. From a process of care to a measure: the development and testing of a quality indicator. Int J Qual Health Care. 2001;13(6):489–496.
11. McGlynn EA, Asch SM. Developing a clinical performance measure. Am J Prev Med. 1998;14(S3):14–21.
12. Eccles M, Freemantle N, Mason J. North of England evidence based guidelines development project: methods of developing guidelines for efficient drug use in primary care. BMJ. 1998;316:1232–1235.
13. Undertaking Systematic Reviews of Research on Effectiveness. NHS for Review and Dissemination Guidelines. 2001.
14. Abenhaim L, Clement D, Norgren L, et al. The management of chronic venous disorders of the leg: an evidence-based report of an international task force. Phlebology. 1999;14(S1):1–126.
15. Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2(1):53–63.
16. Gottrup F, Holstein P, Jorgensen B, Lohmann M, Karlsmark T. A new concept of a multidisciplinary wound healing center and a national expert function of wound healing. Arch Surg 2001; 136(7):765–772.
17. Lazarus GS, Cooper DM, Knighton DR, et al. Definitions and guidelines for assessment of wounds and evaluation of healing. Arch Dermatol. 1994;130:489–493.
18. Kjaer ML, Jorgensen B, Karlsmark T, Holstein P, Simonsen L, Gottrup F. Does the pattern of venous insufficiency influence healing of recalcitrant venous leg ulcers after skin transplantation? J Euro Vasc Endovasc Surg. 2003;25(6):562–567.
19. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174.
20. McGlynn EA, Asch SM. Developing a clinical performance measure. Am J Prev Med. 1998;14(S3):14–21.
21. Rubin HR, Pronovost P, Diette GB. From a process of care to a measure: the development and testing of a quality indicator. Int J Qual Health Care. 2001;13(6):489496.
22. Jorn SD, Williamson JW. Statistical methods for reliability and validity testing: an application to nominal group judgments in health care. Med Care. 1977;15(11):922–928.
23. Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med. 1998;338:1888–1895.
24. Duffield C. The Delphi technique: a comparison of results obtained using two expert panels. Int J Nurs Stud. 1993;30:227–237.
25. Ayanian JZ, Landrum MB, Normand SL, Guadagnoli E, McNeil BJ. Rating the appropriateness of coronary angiography — do practicing physicians agree with an expert panel and with each other? N Engl J Med. 1998;338:1896–1904.
26. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311(7001):376–380.
27. Cantrill JA, Sibbald B, Buetow S. Indicators of the appropriateness of long-term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility, and reliability. Qual Health Care. 1998;7(3):130–135.
28. Sackman H. Delphi Critique. USA;1975.
29. Hopkins. Clinical audit: time for a reappraisal? J Royal Coll Phys Lond. 1996;30:415–425.
30. Liesenfeld B, Heekeren H, Schade G, Hepp KD. Quality of documentation in medical reports of diabetic patients. Int J Qual Health Care. 1996;8(6):537–542.
31. Fowles JB, Rosheim K, Fowler EJ, Craft C, Arrichiello L. The validity of self-reported diabetes quality of care measures. Int J Qual Health Care. 1999;11(5):407–412.
32. Moffatt CJ, Dorman MC. Recurrence of leg ulcers within a community ulcer service. J Wound Care. 1995;4:57–61.
33. Stevens J, Franks PJ, Harrington M. A community/hospital leg ulcer service. J Wound Care. 1997;6:62–68.
34. Elliot E, Russell B, Jaffrey G. Setting a standard for leg ulcer assessment. J Wound Care. 1996; 5:173–175.
35. Lauritzen T, Mainz J, Lassen JF. From science to everyday clinical practice. Need for systematic evaluation of research findings. Scand J Prim Health Care. 1999;17:6–10.
