Systematic Reviews and Meta-Analyses — A Literature-based Recommendations for Evaluating Strengths, Weaknesses, and Clinical Value

Good quality systematic reviews (SRs) summarizing best available evidence can help inform clinical decisions, improving patient and wound outcomes. Weak SRs can misinform readers, undermining care decisions and evidence-based practice. To examine the strengths and weaknesses of SRs and meta-analyses and the role of SRs in contemporary evidence-based wound care practice, and using the search terms systematic review, meta-analysis, and evidence-based practice, the authors searched Medline and the Cumulative Index to Nursing and Allied Health Literature (CINAHL) for important terminology and recommendations to help clinicians evaluate SRs with meta-analysis. Reputable websites, recent textbooks, and synthesized available literature also were reviewed to describe and summarize SR strengths and weaknesses. After developing a checklist for critically evaluating SR objectives, inclusion/exclusion criteria, study quality, data extraction and synthesis methods, meta-analysis homogeneity, accuracy of results, interpretation, and consistency between significant findings and abstract or conclusions, the checklist was applied to topical wound care SRs identified in Cochrane and MEDLINE searches. Best available evidence included in the SRs from 169 randomized controlled trials on 11,571 patients supporting topical intervention healing effects on burns, surgical sites, and diabetic, venous, or pressure ulcers was summarized and showed SRs and clinical trials can demonstrate different outcomes because the information/data are compiled differently. The results illustrate how evidence insufficient to support firm conclusions may still meet immediate needs to guide carefully considered clinical wound and patient care decisions while encouraging better future science.

As new knowledge in health sciences (including wound care) continues to expand, the need for rigorous syntheses of the best available evidence also has increased. As the number of published studies increases, so does the ability to stay current. In the course of making sense of these multiple studies, the number of synthesized reviews also has increased. These purposely combined or synthesized studies are called systematic reviews (SRs). Sometimes SRs include an accompanying quantitative technique called meta-analysis.¹

Combining available information into an integrated result can save considerable time and provide evidence to modify clinical practice.² The publication of SRs and meta-analyses will continue to grow; however, this involves many critical caveats in performing and interpreting them. Poorly developed and written SRs and meta-analyses can yield misleading information.¹ Meta-analysis can be particularly challenged to quantify a treatment effect size across clinical trials due to the diverse clinical variables involved, such as varied wound types and clinical settings and myriad of potential patient differences across studies.³

The purpose of this scholarly review is to examine the strengths and weaknesses of SRs and meta-analyses and the role of SRs in contemporary, evidence-based wound care practice. Discussion of selected problematic issues is offered with clinical exemplars.

Epidemiology of Wound Care: the High Stakes

The economic burden of chronic wound care for the United States is sobering. Currently, more than 70 million acute surgical wounds and approximately 7 million chronic wound patients require care every year.⁴ For pressure ulcer care alone, the cost is estimated to approach $11 billion annually.⁵

The population is aging. By 2030, 19% of Americans will be >65 years old, a group that includes 9 million frail elders.⁶ Age is a risk factor for chronic wounds. Diabetes care will challenge the American health care system. Currently, 29.1 million Americans are diabetic (mostly type 2); 86 million are prediabetics. Diabetes is a risk factor for wounds and the seventh leading cause of death in the United States.⁷

Chronic wound prevalence is not just an American issue. A recent market forecast report⁸ described global predictions: chronic wounds (pressure, venous, and diabetic ulcers) are on course to increase globally from 40 million to >60 million by 2017. The need for good evidence on which to base wound care is imperative, but available high-level evidence is rare. Calls for quality conduct and reporting of randomized controlled trials (RCTs) are occurring in the wound literature.⁹

Evidence-Based Practice (EBP) and How/Why it is Used

EBP is a term describing a problem-solving approach to health care delivery that crosses all disciplines¹⁰; it involves the conscientious, explicit, and judicious use of current best evidence in making decisions about clinical patient care.¹¹ The literature supports that best evidence should be integrated with patient/family preferences and values, individual clinical expertise, and the patient’s clinical context.^10,12,13

EBP is critical to current and future clinical care, especially in light of the Affordable Care Act. According to expert researchers and clinicians,^10-12 EBP can lead to the highest quality of care and the best patient outcomes. Interventions for patients with clinical conditions have to be selected with effectiveness in mind. Scarce resources will become more so as the patient burden in the nation increases.^4,10 In addition, in contemporary care reimbursement for interventions is provided based on clinical outcomes. The legal community also is aware of recommended approaches. Clinicians will be expected to keep current with effective techniques and interventions and provide legally defensible care. Patients seek advice from many sources, too. Clinicians will be expected to explain why they do or do not select a treatment. The specialty of wound care will be heavily impacted by these factors because extensive explanations are time-consuming, and time is money in this costly arena.^10,12

The process of EBP is first enacted by obtaining the best research evidence available. In the hierarchy of EBP, where clinical evidence is ranked according to strength of freedom from biases, meta-analyses and SRs are ranked at the top of describing quantitative effectiveness.¹⁴ If feasibility, appropriateness, and meaningfulness are the goals, qualitative meta-syntheses provide the strongest evidence.¹⁵ Researchers will have synthesized numerous high-quality studies (RCTs) or clinical trial (CTs) results from the literature.¹² Then EBP researchers will approach the literature review based on the PICO or PICOT format — that is, the heart of the SR is a focused question based on a population or group of participants regarding an intervention of interest. The intervention is compared to a control or standard treatment. The outcomes of the intervention are specified. Some researchers include timing issues as well. A sample PICOT question for a SR may read: “In patients with chronic wounds (eg, diabetic foot ulcers), how does use of negative pressure wound therapy (NPWT) affect rate and quality of wound healing in the first 2 weeks of use?” Researchers then perform a systematic search of the literature using clearly specified search terms (Medical Subject Headings or MESH) in a variety of databases (eg, Medline, EMBASE, CINAHL), and RCTs and CTs that match the search processes are retrieved and reviewed.

Using a mnemonic should not infer this is a simplistic activity. The protocol process used in SRs is highly rigorous. Inclusion/exclusion criteria and terminology are thoroughly discussed and explicitly defined. The entire process must be clearly and succinctly detailed in any SR.^16,17

The EBP approach is not without critics. In a narrative review, Berguer¹⁸ argues the evidence-based medicine (EBM) approach, for example, implies only one right way. Proponents suggest EBM embodies scientific truth, and those who critique it must be opting for “non-evidential unscientific data.”¹⁸ Berguer¹⁸ cautions the need to examine how “best evidence” is selected, how pertinent best evidence recommendations are to individual patients, and how the epistemic limitations of RCTs relate to the mechanics of disease.

The Language of Research: Helpful Terminology

Research and EBP use terminology with which clinicians may not be familiar. Just as wound care providers have to learn the correct terms to document wound assessment, they will have to learn new words to comprehend SRs. Table 1 contains a list of selected commonly used terms and definitions for easy reference.

SR. A SR is a structured comprehensive synthesis of the research literature to determine the best research evidence to address a health care question (frequently an intervention). Most often groups of researchers (at least 2) conduct the quantitative SR using meta-analysis (if appropriate).^19,20 A qualitative SR summarizes primary quantitative studies but does not combine the study results using statistical methods.²¹ This latter form should not be confused with synthesis of qualitative research studies, a process called meta-synthesis.

Also known as a research synthesis, a SR has well-defined characteristics^17,21:

•  Clearly describes objectives and focused questions
•  Specifies explicit inclusion/exclusion criteria before work begins
•  Involves an exhaustive search to identify published and unpublished relevant studies
•  Appraises the validity and quality of studies and reporting of inclusion/exclusion choices
•  Includes a data analysis of included studies
•  Presents synthesized extracted findings
•  Provides transparent explicit reporting of methodology used to conduct the review.¹⁹

To promote rigor and transparency and to reduce potential error, a protocol for the SR is developed a priori. The finished SR publication must address explicitly all the multiple steps and decisions. At least 2 reviewers conduct the data extraction.²²

Two international groups commonly conduct SRs related to health care. The Cochrane Collaboration usually involves clinical effectiveness questions utilizing CTs and RCTs.²³ Another group, Joanna Briggs Institute, is more focused on nursing issues that include a greater focus on qualitative research as well as the publication of meta-syntheses.²⁴

The plethora of SRs across health care disciplines is so substantial some researchers are doing SRs of SRs to identify trends, strengths, and gaps in areas of health care.^25,26 The literature supports that SRs have the power to change practice positively when done well. For example, most clinical practice guidelines are a combination of clinical experience, expert opinion, and research evidence. Narrative reviews²⁷ note many guidelines rely on SRs to bolster the evidence base of the guideline.

Meta-analysis. Meta-analysis refers to statistical techniques for combining results from distinct clinical studies. In a sense, a meta-analysis is conducting research on existing research. Meta-analysis is usually conducted to answer a clinical question and should employ research studies that closely match the topic of interest. Gene Glass, who coined the term, called meta-analysis the analysis of analyses.²⁸

The purpose of a meta-analysis is to gather and combine information from research studies to gain higher statistical power for some common metric (a single numerical value of overall treatment effect) across the studies.²⁹ Many wound care RCTs sample too few patients to tell if clinically important outcome differences are statistically significant. If P is the probability of incorrectly rejecting the null hypothesis (the conclusion there is no effect of the treatment tested), the recognized acceptable P value for statistical significance is P <0.05 — ie, differences this large would be found by chance alone only 5% of the time (1 time in 20 replications of the RCT). Combining small studies all measuring the same outcome in the same way adds to the statistical power: the probability the test correctly rejects the null hypothesis when it is false. This allows reviewers to report significant trends not observable in any one of the individual RCTs. An essential belief in meta-analysis is all trials measure a common treatment effect; any observed differences between the trials are due primarily to chance. For example, this common treatment effect can be estimated as a weighted average of the treatment effects in the individual trials.³⁰

Critical to meta-analysis is the inclusion of all relevant primary studies (significant/not significant, published/not published). Narrative descriptions³¹ note unpublished studies, doctoral dissertations, and, if possible, data from primary researchers should be included to enhance the quality of the meta-analysis; therefore, searching for all available studies can be quite complex.²²

Notably, investigators have to make critical choices in conducting a meta-analysis. Studies have to be selected on a set of well-defined objective criteria. Incomplete data issues have to be addressed, data have to be analyzed appropriately, and correct data entered (eg, no typographical errors). Researchers must choose to address publication bias (or not), including the impact of retracted studies.

Because of the ability of SRs and meta-analyses to support best patient management decisions (if done well) or to deleteriously erode quality care (if done poorly), standards for the conduct and reporting of SRs and meta-analyses have been published.³² This statement — Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA)^{33 —} and the Institute of Medicine³⁴ recommended standards for developing SRs provide guidance for research authors and clinician users. Synthesized meta-analyses are being conducted to allow other researchers to identify trends and weaknesses inherent in published studies when meta-analysis is the study focus.³⁵

Strengths and Weaknesses of SR and Meta-analysis

Narrative literature reviews^{3,14,32,36,37} support that the strengths of SRs and meta-analyses are substantive. Combining studies in SRs and meta-analysis increases sample size and gives more precise estimates of effect size (increasing power and precision); clinicians can evaluate a body of evidence pertinent to a clinical question in a facilitated way by encapsulating information into a single result³⁶ that includes all eligible studies related to the clearly defined topic. Statistics can be applied to assess clinical variation or heterogeneity and include Cochran’s Q Statistic or the Inconsistency Index (I²); users must understand how to interpret, but they provide help with understanding the big picture.¹⁴

When done well, SRs and meta-analysis can result in conclusions more informative than any individual trials they combine³⁰; meta-analysis should act to decrease bias because all included trials will likely not be affected equally by a source of bias. Meta-analysis can include studies (trials) that do and do not feature statistically significant results; the latter may have not been published because of a lack of statistical power or small sample size.¹³ As such, using both kinds of studies increases external validity (real world applicability).

Because meta-analysis uses objective statistical techniques, researcher bias should be much less than what would be present in a traditional narrative review.³⁸ In summary, a well-designed and well-conducted meta-analysis can provide valuable information for clinicians, researchers, and policy-makers.¹

Narrative reviews^29,30,39-43 also note SRs and meta-analysis have potential flaws or criticisms. The problematic issues raised about SRs and meta-analyses are serious given their power to positively inform or seriously degrade good clinical treatment. The reality is even good SRs with meta-analysis cannot summarize a research field. By definition, the literature suggests meta-analysis should synthesize effect sizes and not report just a summary effect. A reported summary effect that ignores heterogeneity is “missing the point of the synthesis.”³⁹

Another issue is the file drawer effect. Because unpublished studies may be missed in a meta-analysis, the use of primarily published studies may overestimate true effect size. Meta-analysis has the goal of accurately synthesizing all existing data, so experts suggest attempts to obtain unpublished trials have to be described. Otherwise, publication bias may occur.^44-46

Publication bias is defined as a bias against publishing negative findings (eg, those that do not achieve statistical significance). Several ways exist to identify publication bias in SRs. The most common approach is the funnel plot, which visually identifies possible bias. Statistical tests can be used, including Rosenthal’s Fail-Safe-N and the “trim and fill” analysis.^45,46

The “apples and oranges” problem (also called clinical heterogeneity) is another potential flaw. If dissimilar clinical trials are combined in the SR and meta-analysis, the ultimate meaning of results is potentially threatened. Clinical trials in a meta-analysis will differ somewhat in their characteristics. The challenge for researchers is to decide just how similar they need to be. Data from chronic and parasitic wounds may appear statistically homogeneous, but it may not be clinically relevant to combine such data to inform clinical decisions about one of these wound types. If chosen comparisons are not logical, the validity of the results is eroded. The literature supports that sources of heterogeneity have to be investigated and identified.⁴⁷

Quality of clinical trials used in SR with meta-analysis matters — the flaw is called garbage in-garbage out. Inclusion criteria of selection must be clear and applied fastidiously. Close attention to bias and individual study quality is paramount. One approach is to set a quality threshold (ie, explicit selection criteria established in advance that is logically and systematically applied).² A meta-analysis of poor quality studies cannot yield good results in most circumstances.⁴⁰

Another potential flaw is important studies may be left out of the SR. Good researcher judgment, transparently applied, must be used to make sure all studies meeting the inclusion criteria to address the SR objective are included and studies subjected to combined analysis are similar enough to yield interpretable results.

Potential exists for meta-analyses to disagree with large-scale randomized trials. Clinician users need to beware of discounting the results of the meta-analysis or individual clinical trials. This is not a scenario of the large clinical trial or meta-analysis being right or wrong. Upon closer review, it is likely something inherent in the 2 publications, such as patient risk factors or co-interventions, differ significantly.³⁹

Another problem or flaw with SR and meta-analysis is poorly performed work. Meta-analysis is very complex; researcher mistakes are inevitable. Reviewers and clinician users need to consider the impact of these errors on the validity of the SR and meta-analysis. Research suggests the methodological qualities of SRs vary considerably.⁴² A large potential for bias exists in the selection and interpretation of data in retrospective research. Narrative reviews note meta-analysis can be influenced by biases inherent in data-derived analyses.⁴⁰

Another problem is variation of standards of treatment over time. “Usual care” controls add 2 sources of error variability to a SR. First, the actual interventions applied in the study are omitted from the SR. Second, interventions representing or concomitant to “usual care” change over time and across settings, with no consistent meaning to readers. The influence of concomitant treatments in standard treatment protocols may influence meta-analysis interpretation.⁴⁰

SRs also can differ substantially from real world practice. For example, clinical practice guidelines are usually consistent with clinical practices in disciplines and are an advancement of best practices. SRs potentially reflect the biases and philosophies of the authors who control the choice of included studies,⁴¹ which can lead to stark differences in practical usage and clinical relevance.

Due to limited availability of relevant RCTs, the recognized gold standard for reducing bias in subject treatment assignment, researchers may resort to other experimental studies that are not RCTs. Researchers must acknowledge biases inherent in the chosen designs and make definitive recommendation about practice with great caution.²² Because publication bias occurs more frequently in small trials, meta-analyses based on only small studies cannot be trusted.³⁵

Meta-analyses on the same topic even can have discrepant results. This discrepancy is likely a multifactorial problem and can be due to differences in inclusion/exclusion criteria for study design, outcomes, populations, interventions, settings, definitions, and other factors.⁴⁸

To avoid some of these concerns, researchers conducting meta-analysis should adhere to a few basic requirements. Given the need for a conclusion about a treatment, the minimum number of studies needed is 2.³⁷ In addition, the usefulness of small meta-analyses (<200 to 300 events) to guide practice is very limited.⁴⁰ Also, because of the limitations of meta-analyses and when adequately powered RCTs exist on a topic, the meta-analysis should not be given preferential treatment.⁴⁹

In summary, with all the limitations or flaws inherent in SRs and meta-analysis noted in the literature, it is important to remember to “not forget to critique the critique.”⁵⁰

Critically Evaluating SRs

Although utilization of SR by busy clinicians seems challenging, key strategies for critical assessment, described in the literature, can help with evaluation and critique.^{13,16,22,26,32,36} The clinician needs to ask 8 yes-no questions:

1. Objective(s) of the SR: Do the included studies meet the SR objective(s)?
2. Inclusion/exclusion criteria: Do the studies in the SR match inclusion/exclusion intervention criteria?
3. Quality of studies: Is there critical appraisal of included studies for “quality indicators” such as adequate randomization, allocation concealment, dropout rates, reporting accuracy, blinding, and appropriate statistical analyses?³²
4. Data extraction/synthesis: Were data from the included studies extracted correctly and synthesized appropriately? Are inconsistencies and problems explained?
5. Homogeneity: Do the foci of interest (eg, types of wounds) pass the similarity test (comparing apples to apples)?
6. Accuracy of results: Has a clear and accurate summary of each included RCT been provided in the Results text and analyses?
7. Interpretation: Are descriptions or implications of the SR supported by the data provided? Are limitations acknowledged?
8. Consistency: Do the Abstract and Conclusions of the SR reflect the SR’s significant RCTs results appropriately?

Although this checklist does not guarantee appropriate differentiation of good quality versus erroneous SRs with meta-analyses, it does offer analytical clarity assistance to clinicians.

One of the challenges to clinician users of SRs with meta-analysis is to understand why certain statistics are reported. In addition, clinicians must be able to interpret graphical or visual displays of results. For selected statistical usage of effect size, the nature of the variable being targeted (eg, effect of a wound intervention) is the determining factor. For dichotomous (yes/no) variables (developed a disease; did not develop a disease), the literature suggests the most commonly used effect size estimate is the odds ratio (OR). Risk ratio or relative risk also can be used, but OR usually is utilized. For continuous variables (eg, decrease in wound size or pain intensity scales), the most commonly utilized presentation is the standardized mean difference (SMD) — ie, the mean difference between groups (trials) divided by the pooled standard deviation (SD) of the groups, also called a Cohen’s D statistic.¹³ Mean difference also can be used.

The visual (graphical) display of the meta-analysis is called a forest plot. A forest plot offers valuable information: the number of RCTs and subjects reporting a specific outcome favoring the subject intervention and whether each RCT result was statistically significant at P <0.05, as well as overall significance of the result and potential lack of data homogeneity. It is beyond the scope of this article to discuss interpretation, but excellent, easy-to-understand resources are available for clinicians.¹³

Although utilization of SRs and meta-analysis may seem daunting, the literature also contains articles targeting critical appraisal and clinical application. The reader is encouraged to read more about these processes. The best way to learn to appraise and apply SR with meta-analysis is to do it.^16,29,36,43

SRs and meta-analyses are valuable processes that, without additional resources, permit exploration of treatment benefits from previously completed studies. Clinicians reading their results should have a clear understanding of their strengths and weaknesses, as illustrated in the following sections of this article. Knowledge works only if used. The following provides examples of getting optimal results from SRs even when they have issues.

The Role of SRs in Informing Clinical Practice

Taking a critical thinking approach to reading relevant SRs can enhance their usefulness in improving wound care outcomes. The literature^51,52 provides guidance on using clinical expertise with a healthy dose of critical thinking gleaned from the authors’ experience as Cochrane²³ or Joanna Briggs reviewers.²⁴ Below are some ways to use the critical thinking checklist to improve benefits from a SR or meta-analysis to inform clinical practice decisions.

Look past the abstract. Ideally, best available RCT evidence supporting effects of the SR’s topic intervention on its subject outcome for each clinical indication addressed by the SR is detailed in the Results and Analysis sections. This is where clinical expertise is useful, sifting out results relevant to a challenging patient or wound. To get the most value from a SR, wound care providers need to find clinically relevant RCT content that meets the needs of current patients and wounds. Rather than accepting a SR abstract on faith, diligent wound care professionals will stay focused on finding RCTs that match their patient’s characteristics, capabilities, and wound care needs to glean relevant RCT evidence that may help improve wound and patient outcomes. If a SR abstract omits significant results from RCTs most relevant to the patient at hand, an astute decision maker will likely find them in its Results or Analyses.

Do not wait for perfection. One should not be discouraged from using evidence in a SR that concludes there is insufficient evidence to support an intervention. Many SRs teach clinicians how to aim for perfect evidence to inform clinical decisions. Patients and practitioners who cannot wait for perfect evidence can still find valuable evidence in such SRs to inform choices of care. Authors^51,52 suggest the scientific method will sort out SR flaws and improve evidence-based guidelines of care to optimize results. Today’s patient needs the best possible help now, informed with the best evidence available. One or more relevant RCTs in the Results section of a SR may better inform clinical decisions than 1,000 less-informed opinions or selected cases. Wise professionals can use the best quality of evidence found in SRs to inform clinical decisions instead of giving up on SRs that conclude evidence is insufficient to inform their decisions.

Watch for red flags. Common flaws, discrepancies, or errors such as those described can warn observant readers a SR may not be robust. Inaccurate or inconsistent elements of a SR diminish its credibility. Even if SR conclusions do not reflect its results, one still can check accuracy, then use its relevant, valid RCT results to optimize wound care outcomes for relevant patients.

Clinical objectives. The first purpose of this activity is to empower busy practitioners to critically evaluate a SR’s clinically useful information by illustrating use of the above tips and checklist. Example Cochrane SRs were reviewed to illustrate how one can critique even a meticulously conducted, high-quality SR using the checklist to discover potential flaws and to reveal practice gems to inform clinical wound management decisions. The second purpose is to provide an example of how to mine practice gems from example SR Results and Analysis sections by surveying them, in this instance, for evidence supporting topical therapy.

Search methods. To locate example SRs to review in order to accomplish these objectives, the Cochrane Library and MEDLINE reference databases were searched for the combined terms topical wound healing. First, the SR quality checklist was applied to each example SR to identify potential issues in each, summarized in Table 2a, 2b, 2c.^53-75

To accomplish the second purpose of this work, example SRs addressing common wounds including burns, surgical wounds, diabetic foot ulcers, pressure ulcers, and venous ulcers were reviewed for all significant (P <0.05) healing outcomes, including percent healed, healing time, or percent reduction from baseline wound area or dimensions. The rationale for this approach was these SRs were packed with RCTs summarizing less biased, more dependable evidence supporting topical wound care efficacy than nonrandomized clinical trials, which can only support an intervention’s safety or suggest RCTs needed to prove its efficacy. This is why RCTs are so valuable in informing professionals whether the intervention is likely to work on patients and wounds they manage. The more compelling a SR’s RCT support of efficacy and safety and the more relevant to one’s own patients and wounds, the more valuable it will be in informing clinical decisions.

Results. The Cochrane Library search returned 22 Cochrane SRs. Ten SRs with no RCTs reporting relevant wound healing outcomes were excluded. The MEDLINE search found 5 more relevant Cochrane SRs for a total of 17 qualifying Cochrane SRs ^{53,61,66,74,76-88} and 4 additional similar non-Cochrane SRs,^65,89-91 increasing the total to 21 unique, relevant SRs. The same SRs from this literature search provided examples of SR RCT healing outcomes to inform clinical decisions summarized in Figure 1. To develop Figure 1, one author (LB) tabulated in an EXCEL file all RCT results from RCTs included in the Results or Analysis sections of the included SRs. Studies were tabulated summarizing numbers of subjects and listing whether any healing difference was statistically significant (P <0.05) as reported in any included SR Results section for each major wound etiology pertinent to the literature search. The number of studies reporting statistically significant healing benefits for each intervention for each major wound etiology were counted and displayed as stacked bars with each color corresponding to one of the major wound etiologies in Figure 1. Interventions with only 1 RCT supporting the topical agent were not included in Figure 1, because 1 RCT is not generally considered the highest level of evidence.⁹²

Table 2 describes results of applying the SR readers’ checklist to guide critical thinking focused on meeting patient and wound needs. The second column shows clinical information that is still available despite flaws.

Using the checklist to critically analyze SRs. It is important to note most information in the Cochrane SRs was carefully searched and well documented. The few checklist issues reported in Table 2 most commonly involved inappropriate inclusion or exclusion of RCTs in the SR, omission of significant results from the SR Abstract, combining unlike wounds in meta-analyses, and misinterpretation of results leading to Abstract and Authors Conclusions that contradicted evidence summarized. Checklist flaws that could change SR conclusions were rare, found in only 3 of the 18 Cochrane SRs reviewed. In general, wound care professionals and patients are well served by these thorough reviews — they are laden with pearls of evidence capable of informing clinical decisions to improve outcomes.

In addition to the critical analysis of SR checklist flaws in Table 2, a pattern of 3 retractions was observed for honey interventions that may have potential to bias related literature and prevent access to RCTs, potentially biasing future SRs. Erasing from the literature a rigorously controlled RCT on 108 subjects reporting faster healing in large, sloughy venous ulcer dressed with foam dressings impregnated with honey compared to hydrogel⁶² presents a disservice to clinicians by rendering these clinically important data unavailable to inform practice or to include in future SRs. Without a clearer description of the rationale for retraction than “due to errors in the data analysis which affect the article’s findings,” it would seem more appropriate to have published an erratum statement clarifying and correcting the data analysis error(s). This would allow the valuable clinical information gleaned from this important study to continue to inform clinical decisions.

A second retraction⁹³ relating to the pH of wounds treated with honey may help explain findings of reduced antibiotic use^56,94 or earlier reduced bacterial burden^56,94-99 in burns, infected surgical sites, or diabetic foot ulcers receiving primary dressings impregnated with honey. Eradicating this research from the literature, rather than publishing a correction of any errors discovered, disserves clinical readers and researchers and interrupts the continuum of scientific discovery.

The third retraction¹⁰⁰ was a critical comment about inappropriate negative statements in the publication of a large RCT on small venous ulcers with varying “usual care” control interventions. A co-author on this RCT accurately clarified inappropriate statements that treating venous leg ulcers with honey dressings was “unlikely to help healing” stating, “The reality is that the results were inconclusive, not negative.” The retracted statement also clarified inappropriately negative interpretations of health economic and pain outcomes in this study. This disagreement between the 2 authors was retracted from the literature with no apparent explanation and no further effort to ensure a more accurate fair, balanced summary of the study results. Absence of this perspective may cause discrepancies in SRs of the topical honey literature.^61,65

Finding evidence in SRs to inform clinical decisions. Astute clinicians will view evidence from any SR through a lens focused on patient and wound needs, applying concepts such as those listed in the SR checklist to ensure a fair, balanced appraisal of relevant evidence. To illustrate how wound care professionals can search SRs to inform clinical decisions while compensating for checklist flaws such as those described in Table 2, Figure 1 displays numbers of RCTs reporting at least 1 significant (P <0.05) effect on any measured healing parameter in response to topical interventions for each of the major wounds listed. This is not a comprehensive SR meeting stringent definitions of what constitutes a healing benefit. Unlike a Cochrane SR, no judgment is made about individual study quality. The visual display illustrates a way to summarize best available RCT evidence, defined as RCTs meeting standards adequate for inclusion in at least 1 peer-reviewed SR, reporting recognized healing parameters relevant to informing clinical decisions about the common wounds represented. For a more stringent, comprehensive meta-review of all outcomes for local and systemic wound care, readers are referred to Brölmann et al.¹⁰¹

Figure 1 summarizes best available SR evidence from 169 RCTs involving 11,571 patients, favoring each specified topical intervention when compared to a control intervention on the indicated type of wound. Of these RCTs, the 78 that reported a statistically significant (P <0.05) healing benefit are summed for all clinical wounds reviewed. To avoid counting any study more than once, each RCT was counted only for the intervention reporting at least 1 statistically superior healing difference compared to at least 1 other control topical intervention in an RCT. For example, an RCT¹⁰² on 32 patients reporting faster healing (P <0.05) of skin graft donor sites dressed with foam dressings compared to nanocrystalline silver was counted only once in the visual display as supporting healing efficacy of foam dressings. Healing metrics qualifying for inclusion in any bar of Figure 1 included statistically significant improvement in time to complete epithelialization, percentage of wounds completely healed, and percent or absolute reduction in wound area or length, width, or depth. Including all healing parameters underscored the study time frame was too short for most chronic wounds to achieve complete epithelialization. This approach does inform readers of the most possible clinically relevant healing outcome information.

The top 8 interventions with the most evidence significantly supporting any healing parameter were in order of descending numbers of RCTs with significant supportive evidence:

1. Hydrocolloid dressings: 17 RCTs reporting a healing benefit (P <0.05) among 34 favorable healing RCTs on 2,487 patients, including 8 of 21 RCTs on 1,528 individuals with a venous ulcer, 4 of 5 RCTs on 364 pressure ulcers, 2 of 5 RCTs on 240 partial-thickness burns, 2 of 2 RCTs on 140 surgical wounds, and 1 of 2 RCTs on patients with 175 mixed chronic wounds;
2. Honey: 15 RCTs reported a healing effect (P <0.05) among 20 RCTs reporting any healing benefit on a total of 2,625 subjects. This included 9 out of 10 RCTs on 1,654 persons with partial-thickness burns, 3 of 5 RCTs on 375 patients with a mixed acute or chronic wound, 1 RCT each on 40 patients with an infected Caesarian section incision, 108 individuals with a large, 50% slough-covered venous ulcer, and 40 persons with a pressure ulcer;
3. Biosynthetic dressings: 10 RCTs reported a healing benefit (P <0.05) among 13 favorable RCTs on 498 patients with partial-thickness burns;
4. Iodine complexes: 9 RCTs reported a healing benefit (P <0.05) among 14 RCTs on 788 patients. All 9 significant RCTs studied people with a venous ulcer;
5. Silver compounds: 8 RCTs reported a healing benefit (P <0.05) among 20 RCTs on 2,267 patients, mainly on burns, with 1 RCT each on individuals with a pressure ulcer, venous ulcer, or mixed chronic ulcers;
6. Hydrogels: 5 RCTs reported a healing benefit (P <0.05) among 13 favorable RCTs on 993 patients. Significant effects were mainly on partial-thickness burns, diabetic foot ulcers, or venous ulcers;
7. Foam dressings: 3 RCTs reported a healing benefit (P <0.05) among 16 favorable RCTs on 807 patients. Significant effects were reported in 1 RCT each on individuals with a diabetic foot ulcer, venous ulcer, or a surgical wound;
8. NPWT: 3 RCTs reported a healing benefit (P <0.05) among 6 favorable RCTs on 184 patients with a diabetic foot ulcer or a venous ulcer or on persons with an acute or chronic wound of any origin.

Limitations. Only healing outcomes are summarized visually in Figure 1. Future summaries may address other measured outcomes, such as patient-reported pain, incidence of infections or antibiotic use, debridement, restoration of negative wound cultures, quality of life, or economic outcomes. The visual display does not summarize comparisons within any dressing category; for example, comparing one foam dressing with another would not inform a clinical choice to use a primary foam dressing compared to some other topical intervention.

The results’ visual display illustrates how much RCT information one can glean from SRs that can help inform wound care decisions about topical agent healing RCT-level evidence. Figure 1 is not intended to fulfill the higher purpose of judging study quality or sufficiency of evidence as many SRs do. It simply summarizes the volume of evidence supporting each topical intervention by counting RCTs and patients with the indicated wounds, focusing on RCTs supporting statistically significant (P <0.05) healing differences favoring each intervention listed.

Only healing outcomes such as increased percent of patients achieving a predefined healing outcome (eg, 100% epithelialization) or reductions in wound dimensions (length, width, depth) or area or time to heal are displayed in Figure 1. The figure does not summarize other important wound-related outcomes, such as time to achieve a bacteria-free wound culture, patient-reported pain reduction, improvement in health-related quality of life, or measured debridement parameters. RCTs with >30% of intent-to-treat patients withdrawn before the healing outcome was measured were not included, because they may unfairly represent the clinical population sampled. In addition, interventions with only 1 RCT significantly supporting healing on only 1 clinical indication found in the SR search were not included in the visual display. These were: amniotic membrane, benzoate complex, benzoyl peroxide, dextranomer beads, enzymes, ethacridine lactate, film dressings, mupirocin, phenytoin, or saline gauze.

A limitation of focusing on Cochrane SRs was that examples found in this limited search did not contain all healing RCTs on all the listed indications. For example, several recognized RCTs on growth factors or on hydrocolloid dressings did not appear in the SRs returned by the search. This review did not seek to perform a comprehensive search of all healing RCTs, only to identify enough SRs to illustrate 1) how to use the SR reader’s checklist to identify possible flaws and 2) ways to look past SR flaws to wield evidence of sufficient quality to meet Cochrane inclusion standards to inform clinical practice.

Wound care professionals have final responsibility for their patients. SRs or the RCTs they contain such as those summarized for healing (see Figure 1) can help inform decisions about patients and wound care, but they may not preempt clinicians’ decisions. Unlike content-validated guidelines, based on evidence plus expert opinion or, in some cases independently compiled content validation,⁶⁷ SRs should not recommend a course of action. A SR is designed to tell readers if evidence is sufficient to inform clinical decisions about comparative efficacy and/or safety of an intervention on a given type of wound. Only involved wound care professionals know the patient’s wound care needs and capabilities well enough to select the right care. The responsibility for knowing and applying best available evidence to inform care decisions that can improve patient outcomes belongs solely to responsible care providers. Limited time and resources make it essential to use the best quality evidence available.

Cochrane reviewers have provided important summaries of best available evidence on wound care options. Even if the compiled evidence is insufficient to support practice decisions by Cochrane standards, it represents a good source of the best available evidence, providing helpful perspective to supplement basic wound care knowledge. Authors of each SR have saved wound care providers time and effort of compiling and analyzing the RCT evidence. Implications for practice are substantial. Even if a SR contains potential issues such as those described in Table 2, wound care professionals can look past the flaws to examine evidence compiled for relevant statistically significant outcomes that may help inform decisions to meet patients’ needs. By using best available evidence found in a SR, a person stands on the shoulders of past researchers to see a bit further, better informing clinical decisions. If a wound care professional focuses on patient and wound challenges at hand while reading the Results section of a SR, its best available evidence, even if imperfect or “insufficient”, informs clinical decisions better than opinion alone. Future research will help determine how intermediate findings become compelling standards of care. 

The authors gratefully acknowledge this work was supported by an unrestricted educational grant from Derma Sciences, Inc, Princeton, NJ. Neither author has any financial interest in Derma Sciences, Inc. Dr. Bolton consults for Derma Sciences, Inc on the development of acleristide, a topical active agent to treat diabetic foot ulcers unrelated to the topic of systematic reviews. 

Dr. Beitz is a Professor of Nursing; and Director, Wound Ostomy Continence Nursing Education Program (WOCNEP), Rutgers University School of Nursing –Camden, Camden, NJ. Dr. Bolton is an Adjunct Associate Professor of Surgery, Robert Wood Johnson University Medical School, New Brunswick, NJ. Please
address correspondence to: Laura L. Bolton, PhD, 15 Franklyn Place, Metuchen, NJ 08840; email: llbolton@gmail.com.