The Complementary Relationship Between Quantitative and Qualitative Research Methods in Enhancing Understanding of Treatment Decisions, Outcomes, and Value Assessment

J Clin Pathways. 2021;7(6):20-23. doi:10.25270/jcp.2021.0708.5

In the July 2021 edition of the Journal of Clinical Pathways, May and colleagues argue that qualitative research can both complement and augment quantitative research methods by providing important, “on-the-ground” context and perspectives in the assessment of treatment value and outcomes.¹ We agree with this argument, and firmly believe that the best approach to addressing complexity in health care innovation and care delivery is evidence generated through a mix of both quantitative and qualitative research methods. 

Frequently, quantitative research methods are used to illuminate how detrimental a disease might be in health outcomes and associated costs, what effect a treatment might have on the disease in question, and how much we should be willing to pay for said treatment. 

However, quantitative methods are limited in that such approaches are often only able to illuminate the part of a problem that can be told numerically and cannot fully account for on-the-ground realities. Qualitative methods can help to fill this gap by providing valuable first-hand perspectives and by placing quantitative study findings into context. Similarly, developing research questions for quantitative studies requires a sound understanding of underlying issues, which can be augmented by qualitative approaches. 

Quantitative research methods answer a broad range of important questions in health research. Taking COVID-19 as a contemporary example, quantitative research methods have been used to address key epidemiologic concerns that include disease incidence, prevalence, and transmission² while also considering the overall economic burden of disease³ and examining questions related to pricing of vaccines and therapies.^4,5 Additionally, researchers have relied on quantitative methods to study the impact of policies aimed at limiting the spread of COVID-19, including testing, mask mandates, lockdowns, and other social distancing measures.^6-8

Among quantitative methods, retrospective database studies, economic/econometric approaches, and simulation models are most frequently used to generate insights. Retrospective database studies or claims analyses often are used to provide real-world evidence on cost and use of therapies, as well as efficacy and safety outside of the clinical trial setting.⁹ By deploying retrospective study methods, researchers can quickly collect and analyze large, diverse data sets that include longitudinal follow-up reporting.¹⁰ However, retrospective studies using administrative claims databases, such as the MarketScan Database,¹¹ or large, nationally representative survey data, such as the University of Michigan’s Health and Retirement Survey,¹² also pose various methodological challenges and limitations that include lack of control over quality and inconsistent collection or reporting of data.^13,14 

Similarly, economic studies have been used to assess the impacts of governmental policies on population health outcomes or to evaluate the effectiveness of different health programs and interventions.¹⁵ For instance, a recent study by Lyu and Wehby examined whether county-wide mandates for face masks decreased incidence of COVID-19 and found that by May, 2020, mask mandates had resulted in 200,000 fewer deaths from COVID-^19.16 Economic and econometric studies allow researchers to use formal analytic techniques to make evidence-based decisions.¹⁷ Nevertheless, these studies typically do not take subjective experiences into consideration, which can limit the overall evaluation of real-world effects.¹⁸

Lastly, simulation models have served as ways to estimate and project uncertain long-term effects of programs, treatments, and interventions at a population level.¹⁹ In the context of COVID-19, Jahn and colleagues used a dynamic agent-based population model to compare different vaccination strategies to maximize the impact of Austria’s limited vaccine supply.²⁰ The authors found that in order to minimize hospitalizations and deaths related to COVID-19, elderly and vulnerable populations should be prioritized for vaccination until further vaccines are available. These methods allow researchers to explore various scenarios in a structured way.²¹ Even so, simulation methods cannot be used to accurately evaluate all scenarios, especially in situations where key parameters are unknown or highly uncertain.²² 

Quantitative research methods are used frequently to guide decision-making in health policy and health care. For instance, for initial approval of a drug or therapy, regulatory agencies such as the US Food and Drug Administration (FDA) and the European Medicines Agency (EMA) consider analyses of clinical trial data on safety and efficacy end points. Post-approval, regulatory attention often shifts to analyses of real-world evidence to demonstrate how drug efficacy and safety measures hold up outside of the clinical trial setting. Lastly, insurance companies and payer organizations rely on cost-effectiveness and budget impact analyses to determine coverage and reimbursement practices for treatments, and to gauge whether a treatment’s benefits in averted health consequences outweigh its costs. 

Cost-effectiveness analysis often use quantitative research methods to evaluate the value and pricing of interventions.  In the United Kingdom, the National Institute for Health and Clinical Excellence (NICE) quickly released detailed guidelines for COVID-19.²³ In the United States, the Institute for Clinical Effectiveness Research (ICER) performed a cost-effectiveness analysis in 2020, determining that the cost-effective price for remdesivir as a treatment for COVID-19 would be between $4,580 and $5,080 per treatment course.²⁴ 

Similarly, government agencies such as the Canadian Agency for Drugs and Technologies in Health (CADTH) and NICE often base reimbursement decisions on comprehensive health technology assessments (HTA) that usually contain economic evaluations for the intervention.²⁵ 

Lastly, the FDA recently has begun considering modeling and simulation methodologies when reviewing and approving drugs and has advocated for their use in predicting clinical outcomes.²⁶ Among other things, the use of physiologically-based pharmacokinetic (PBPK) simulation models has increased in recent years to address uncertainties remaining around clinical trial data, though acceptance by the FDA has been varied.²⁷

Quantitative studies are the gold standard for establishing causality and drawing conclusions that can be generalized beyond the population studied. However, quantitative research methods also often lack contextual considerations and perspectives that can be further informed by qualitative approaches. To overcome potential limitations of either research approach, we strongly believe that qualitative and quantitative research methods are best used in concert. 

For example, as previously noted, HTA are commonly used to estimate whether or not a drug or therapy can be considered “value for money.”^28,29 However, within the context of rare diseases or rapidly emerging health crises such as the COVID-19 pandemic, reliance on empirical data alone is often insufficient, as data may be too sparse to inform population health decision or may not be collected quickly enough to guide decision-making in crisis. 

According to ICER, conducting cost-effectiveness analyses for rare diseases such as B-cell lymphoma, hemophilia A, and spinal muscular atrophy can be particularly challenging due to the small number of patients afflicted, a lack of available treatments or treatment alternatives, and limited long-term empirical data.^30,31 To overcome some of these challenges in the United Kingdom, NICE relied heavily on expert judgment to fill evidence gaps, though processes for eliciting expert assessments have varied across NICE’s guidance-making programs.³² In the context of COVID-19, quantitative research methods have provided highly valuable insights as outlined above, but key questions remain in areas where empirical data is not yet available, or where data collection has not kept pace with the demand for evidence to guide policy decisions. To reference a few examples, qualitative research designs have been used to study vaccine hesitancy among communities of color,^33,34 as well as the impact of COVID-19 on family caregivers.³⁵

True mixed-methods research that borrows from both qualitative and quantitative research methods can serve as a solution to this problem: qualitative methods can be used to generate hypotheses and develop research questions, while quantitative methods can be used to confirm hypotheses and further solidify qualitative research findings by using a larger, more diverse sample.

Expert elicitation methods such as the Delphi method or the Sheffield Expert Elicitation Framework (SHELF) are interesting ways to combine qualitative and quantitative approaches to address gaps in empirical knowledge.^36-38 Typically, both methods systematically elicit expert knowledge about uncertain parameters and quantities through interviews with, or questionnaires administered to, key subject matter experts. Results from individual assessments are then aggregated in the form of subjective probability distributions. Both methods follow strict protocols to avoid respondent bias and to ensure high levels of objectivity in eliciting key parameters in areas devoid of empirical evidence.³⁶ 

In a recent application of the Delphi method, Graf and colleagues systematically elicited how US payers evaluate trade-offs between costs, as well as both clinical and real-world evidence of efficacy and safety in the context of a rare disease, Hemophilia A.³⁹ The study concluded that US payers strongly preferred treatments with known efficacy and safety, and well-understood costs over newer treatments with uncertain long-term effects for patients with Hemophilia A, providing important insights into payer decision-making practices in areas with limited empirical evidence and high levels of uncertainty.

Using the SHELF method, Cope and colleagues systematically elicited survival rates and related uncertainty for children and young adults with relapsed or refractory acute lymphoblastic leukemia, 2 to 5 years after receiving chimeric antigen receptor T-cell (CAR-T) therapy.⁴⁰ Individual expert estimates were combined with observed data using time-to-event parametric models that accounted for experts’ uncertainty, producing an overall distribution of survival over time. The authors found that a combination of clinical trial data and expert judgments substantially improved the precision of extrapolated survival curves when compared with naïve extrapolation methods based solely on clinical trial data.  

However, while expert elicitation methods combine both qualitative and quantitative research methods to provide valuable information in areas that lack long-term, empirical evidence, some limitations persist. Most importantly, small sample sizes often prevent the generalizability of results beyond the expert population consulted.³⁷ Additionally, any assessments that rely heavily on individual judgments can be subject to various sources of bias, including anchoring and adjustment bias (eg, providing a specific point estimate or reference in the question, thereby influencing experts), the availability heuristic (eg, experts might overstate the likelihood of an event if they recently experienced it), range–frequency compromise (eg, when provided with a range of options, experts might apportion responses evenly), and overconfidence (eg, experts might provide confidence intervals that are too narrow).³⁶ 

In an example mixed-methods study, Yelverton and colleagues interviewed both HIV patients and physicians about the use of antiretroviral therapy (ART), and then conducted quantitative ranking exercises to develop priority areas for shared decision-making between patients and their providers.⁴¹ Similarly, Barber and colleagues conducted interviews with pediatric hypodontia patients and their parents, in order to inform the development of key treatment attributes for a discrete choice experiment.⁴² Specifically, interviews were used to assess what factors both patients and their caregivers value in potential treatment alternatives, and how they evaluate trade-offs between these attributes. 

Many health research challenges require a combination of quantitative and qualitative research approaches. The importance of quantitative approaches to generate empirical data and to detect patterns and trends cannot be understated.  Yet, a quantitative approach alone has its limitations. Quantitative approaches as we have described here can only be strengthened by a mixed-methods approach that serves as a continual feedback loop on the data generated and offers new avenues for exploration. A mixed-methods approach allows for a continued reflection on where there could be gaps in the data collected. 

Authors: Marlon Graf, PhD; Rifat Tuly, MPH; Suepattra G May, MPH, PhD

Affiliations: PRECISIONheor, Los Angeles, CA