Skip to main content

Advertisement

ADVERTISEMENT

Peer Review

Peer Reviewed

Original Research

Prioritizing Native Breast Skin Preservation or Scar Symmetry in Autologous Breast Reconstruction? Using Crowdsourcing to Assess Preference

 

Ashraf A. Patel, MD1; Carol L. Feng, MD2; Jessica Marquez, BA1; John P. Spaw, BS, MS3; Rebecca M. Garza, MD4; Gordon K. Lee, MD, FACS5; Rahim S. Nazerali, MD, MHS, FACS5

December 2023
1937-5719
ePlasty 2023;23:e75
© 2023 HMP Global. All Rights Reserved.
Any views and opinions expressed are those of the author(s) and/or participants and do not necessarily reflect the views, policy, or position of ePlasty or HMP Global, their employees, and affiliates. 

Abstract

Background. Recent literature on autologous breast reconstruction suggests that such factors as scar symmetry and skin paddle size impact patient preferences more than preservation of native breast skin. Since patient satisfaction with plastic surgery procedures can be largely influenced by beauty standards set by the general public, this study used a novel crowdsourcing method to evaluate laypeople’s aesthetic preferences for different bilateral autologous breast reconstructions to determine the relative importance of scar and skin paddle symmetry and preservation of native skin.

Methods. Using Amazon’s Mechanical Turk crowdsourcing marketplace, participants ranked images of reconstructions based on overall aesthetic appearance. Images were digitally modified to reflect 4 types of reconstruction: immediate (IR), delayed symmetric (DS), delayed asymmetric (DA), or mixed (MR). 

Results. DS was ranked most favorably (1.74), followed by IR (1.95), DA (2.93), and MR (3.34). Friedman rank sum and pairwise tests showed statistical significance for comparisons of all 4 reconstruction types. Likert ratings were higher for IR than for DA reconstructions for skin quality (P = .002), scar visibility (P < .001), scar position (P < .001), and breast symmetry, shape, and position (P < .001). Ratings increased for all aesthetic factors following nipple-areolar-complex reconstruction (P < .001). 

Conclusions. More symmetric breast scars were rated aesthetically higher than nonsymmetric scarring, and our participants preferred maintenance of scar symmetry over preservation of native breast skin. These findings are consistent with previous studies that surveyed non-crowdsourced participants, which demonstrates the potential for crowdsourcing to be used to better understand the general public’s preferences in plastic surgery.

Introduction

Many approaches to bilateral autologous breast reconstruction are possible with a wide variety of skin flaps and scar patterns. Another aspect affecting aesthetic outcome is the surgical decision to stage the procedure in an immediate, delayed, or delayed-immediate fashion.1 Reconstruction may also occur unilaterally or bilaterally, thereby resulting in an asymmetric or symmetric scar pattern and presence of the native or reconstructed nipple-areolar complex (NAC).2 Furthermore, the need for adjuvant radiation or chemotherapy affects the quantity and quality of healthy breast skin available for reconstruction.3,4 This creates a unique challenge for reconstructive surgeons as they work to reach the patient’s aesthetic preference and goal, which may be influenced by public perception of ideal breast aesthetics. 

Previous studies have highlighted the importance of providing the patient with realistic preoperative information, as this can better enable patients to make more satisfying long-term decisions.5,6 Therefore, surgeons should understand the factors that determine patient satisfaction in order to properly plan surgical approaches for breast reconstruction. Studies have found that such factors as scar symmetry and size of the skin paddle significantly impact patient preferences more than preservation of native breast skin, the latter of which had long been a focus of plastic surgeons.7 These results suggest a potential disconnect between the surgeon and patient’s definition of a successful reconstruction.8 However, prior survey studies have not been large enough to effectively establish a consensus regarding patient preference. 

This study aimed to explore how a patient may perceive breast reconstruction as aesthetically successful on a larger scale. Because patient satisfaction with plastic surgery procedures are largely influenced by conventional beauty standards determined by the aesthetic preferences of the general public, we utilized crowdsourcing to better understand the general public’s perceptions of breast reconstruction results. We then used these findings as a proxy to explore the patient’s preference among reconstructions prioritizing the use of native breast skin versus reconstructions prioritizing scar symmetry with non–breast skin.9

Table 1

Methods

This study was approved by Stanford University’s Institutional Review Board (IRB # 47054). A survey was created using Google forms (Palo Alto, CA) with images from 3 patients who had undergone bilateral autologous reconstructions by one of the authors (GKL). These images were then digitally modified using Adobe Photoshop (Adobe Systems Incorporated) to create 16 total images that reflected 4 different types of breast reconstruction. In cases where we did not have images of the exact reconstructive style, skin paddles were superimposed and integrated onto the patient image to create a similar depiction. These images were created using scars from our patients and modifying those digitally to better match each patient. Additional modification was used to ensure the superimposed reconstruction was well integrated onto the patient image, which included digitally creating the skin changes seen on the periphery of the superimposed reconstruction. 

We utilized Mechanical Turk (MTurk) (Amazon; Seattle, WA) crowdsourcing marketplace to obtain eligible workers to complete the survey. All survey participants were from the US and were at least 18 years of age; participant demographics are detailed in Table 1

Figure 1
Figure 1. (A) Example of photo set in Part A of the survey that includes the 4 categories of bilateral autologous breast reconstruction studied: immediate (top left); delayed symmetrical (bottom left); delayed asymmetrical (top right); and mixed (bottom right). (B) Example of delayed asymmetrical and immediate breast reconstruction studied in Part B of survey before (above) and after (below) nipple-areolar complex reconstruction/tattoo. 

Survey Design

The survey was designed to be completed within 10 minutes. Our survey began with collecting basic participant demographics including age, gender, and ethnicity as well as a signature to provide consent. Participants responses were only collected once. The remainder of the survey was split into 2 parts. In Part A, participants were shown 4 sets of images (Figure 1A) with each set containing 1 image for each of the following types of bilateral breast reconstruction and asked to rank each image based on aesthetic appeal: delayed symmetric (DS), delayed asymmetric (DA), immediate (IR), or mixed breast reconstruction (MR). Delayed asymmetric involved autologous reconstruction with large skin paddles to both breasts with scar asymmetry. Delayed symmetric involved delayed reconstruction with symmetrical scarring on both breasts. Immediate reconstructions maintained the maximum amount of native breast skin and included centrally located, symmetrical skin paddles. Mixed reconstruction involved delayed reconstruction to 1 breast and immediate reconstruction of the other, often leading to asymmetric scar variability throughout both breasts. Of the 4 sets of images participants were shown, 1 set included images following NAC reconstruction and tattooing, also performed by one of the authors (GKL). None of the patients depicted in this part of the survey had undergone radiation. 

In the second part of the survey, participants were shown 3 sets of images that included before and after NAC reconstruction and tattooing (Figure 1B). However, the participants were only shown 1 image at a time and asked to rate the image on a 5-point Likert scale to assess their perceptions of skin quality/color, scar formation (visibility), scar position, breast symmetry, breast shape, breast size, breast position, and overall aesthetic outcome. For both parts of the survey, the order in which the images were presented was randomized for each respondent. Patients depicted in this portion of the survey had undergone right breast radiation. 

The survey also included 2 control questions randomly dispersed throughout the survey to ensure accurate responses were being recorded. These questions asked participants to select a specific answer choice as instructed. Any respondents who gave incorrect answers to these questions were excluded from the study, which reduced the number of included respondents from 256 to 132. Each survey response was correlated to a unique Amazon MTurk Worker Identification number to ensure that the each MTurk worker could only take the survey once. The results were exported to Microsoft Excel (Seattle, WA) and analyzed statistically to determine differences in scores and preferences among the 4 types of reconstruction with and without nipple reconstruction. 

Figure 2
Figure 2. Distribution plot displaying the differences in mean rank among all 4 types of reconstruction: delayed symmetric (DS), immediate (IR), delayed asymmetric (DA), and mixed reconstruction (MR). 

Statistical Analysis

For Part 1 of the survey, mean rankings were computed for each type of reconstruction (DA, DS, IR, MR) from all 4 sets of images. The Friedman rank sum test was utilized as a nonparametric test for differences in ranked choice data. This provided strong evidence of a global difference among mean ranks of types of reconstruction, and a pairwise test was utilized to compare the mean rank of 1 type of reconstruction to the mean rank of all other methods (eg, IR vs [DA, DS, MR]). Distribution plots were then created to interpret preferences of types of reconstruction (Figure 2).

For Part 2 of the survey, response mean rankings for each characteristic were computed for delayed symmetric and immediate reconstructions before and after nipple tattooing. P values were computed utilizing the unpaired t test, and P values below .05 were considered to be statistically significant. 

Results

A total of 132 unique respondents were included in this study; of these, 50% were men, 47.7% were women, and 2.3% declined to indicate gender. White respondents made up a majority of our cohort (62.9%), followed by Asian respondents (17.4%). According to our self-reported data, 91.7% of our respondents had never been diagnosed with breast cancer whereas 7 respondents (5.3%) reported having been diagnosed with breast cancer in the past. Of these 7 respondents, 5 (71.4%) had undergone some form of breast reconstruction. Participant demographics are summarized in Table 1.

Table 2

Results from Part A of the survey are summarized in Table 2, which demonstrates participant ranking from 1 (highest rank) to 4 (lowest rank) for all 4 modes of reconstruction; these rankings are also depicted in Figure 1. Overall, DS was ranked most favorably with a mean ranking of 1.74, followed by IR (1.95), DA (2.93), and MR (3.34), which was ranked least favorably. The Friedman rank sum test concluded strong evidence of a global difference among mean ranks of reconstruction method, and the pairwise test confirmed statistical significance among all 4 mean rankings.            

Table 3

Part B compares various aesthetic factors between IR and DA reconstructions with and without nipple reconstruction, and these data are summarized in Table 3. Ratings were on a scale of 1 to 5, with 5 being the highest rating. When all IR images were compared with DA reconstructions, all but 1 aesthetic factor ranked higher for IR: skin quality/color (3.38 vs 3.14; P = 0.002), scar formation and visibility (3.23 vs 2.83; P < .001) , scar position (3.52 vs 2.81; P < .001), breast symmetry (2.89 vs 2.55; P < .001), breast shape (3.16 vs 2.74; P < .001), and breast position (3.29 vs 2.97; P < .001). The greatest difference among aesthetic ratings was seen for scar positioning (0.71). Breast size was rated equally between IR and DA reconstructions (2.94 vs 2.94; P = .17). The P values for each pairwise test were adjusted using a Bonferroni correction to conservatively account for multiple-testing and prevent against falsely flagging insignificant results. 

Table 4

To further investigate the effect nipple reconstruction had on aesthetic rating, the results from Part B were separated by nipple reconstruction for both IR and DA. These data are organized in Table 4. Ratings for each aesthetic factor were statistically significantly higher after nipple reconstruction for both IR and DA reconstructions.

Discussion

An array of options currently exist for patients choosing to undergo autologous breast reconstruction, but the results from each form of reconstruction may vary. A result may be viewed as an aesthetic success by the surgeon, but this may not align with a patient’s desired aesthetic result.7 With a greater emphasis on shared decision-making and patient satisfaction, plastic surgeons have shifted the reconstructive priority to be better tailored to the patient’s preference, which may be influenced by societal beauty standards determined by the aesthetic preferences of the general public.10,11 

To effectively measure public response, large sample sizes and diverse subject pools are essential, which is often a challenge for traditional prospective single-institution survey studies. To overcome this, our study utilized online crowdsourcing to survey the general public’s preference among 4 modes of bilateral autologous breast reconstruction. This study utilized the same questionnaire and images used in the 2016 study by Garza et al, which assessed patient and surgeon preference in breast reconstruction results by surveying in-person participants.7 By recreating a similar study design, we also analyzed the efficacy of using Mechanical Turk (MTurk) as an online crowdsourcing platform and evaluate the reproducibility of results by comparing our findings with those reported by Garza et al. 

Our study measured the mean ranking among 4 methods of bilateral autologous reconstruction: IR, DS, DA, and MR (reconstructions that incorporated both mixed and immediate reconstructions to each breast). IR seeks to preserve more skin as a larger mastectomy flap is available for manipulation, whereas delayed reconstructions utilize abdominal flaps to reconstruct larger areas of the breast, leaving behind less native breast skin. DS, as opposed to DA, goes a step further by removing additional native breast skin from the more benign breast in order to preserve scar symmetry between both breasts. MR preserves a greater amount of native breast skin than DA, but the degree of asymmetry is greater in MR. 

Garza et al reported that patients with breast cancer ranked DS most aesthetically appealing, whereas plastic surgeons ranked IR highest. DS reconstructions ranked the highest (1.74) in our study as well, which continues to challenge the notion that immediate or delayed-immediate breast reconstructive outcomes are superior to delayed reconstructive outcomes, particularly from the patient and layperson perspective.12-14 IR maintained the maximum amount of native breast skin, but given that DS was ranked higher, this indicates laypeople prefer symmetry to native breast skin preservation. This idea is further supported by the portion of our study focusing specifically on delayed reconstruction, where DS continued to be ranked higher than DA. Though MR is associated with a greater degree of asymmetry than DA, it does preserve more native skin; however, MR was still ranked lower than DA, which again reinforces the idea that laypeople do not prefer native skin preservation when sacrificing symmetry.

The ranking results of this study also suggest that skin paddle and scar placement is important for aesthetic outcome. Spear and Davisondeveloped the subunit principle of breast reconstruction, which outlined natural areas, or subunits, of the breast within which skin paddles and scars could be placed for optimal aesthetic outcome.15 Based on a retrospective analysis of 264 autologous breast reconstructions, they concluded that the best aesthetic subunits were (1) the areolar subunits—concentric circles around the NAC up to 6 cm in diameter, and (2) the inferior oblique subunits—lower outer crescent areas of the breast that use the horizontal nipple plane as the upper limit of scar/paddle placement. Violations of these subunits, especially scars/paddles involving the superomedial aspect of the breast, lead to less attractive breast aesthetics.15-17  Findings in our study generally support the subunit principle described by Spear and Davison such that the small, centrally placed skin paddles depicted in IR reconstructions demonstrated overall high aesthetic rankings. However, one caveat to the subunit principle that the present study may challenge is that symmetry can improve aesthetic outcomes for reconstructions that violate breast subunits. For example, our DS reconstructions, which often did place scars/paddles in the superomedial quadrant of the breast bilaterally and symmetrically, consistently ranked higher than DA reconstructions, which had “better” scar/paddle placement adhering to the subunit principle in at least 1 breast. The importance of symmetry has been previously demonstrated by Coutinho et al, who found that participants preferred not only bilateral and symmetric but also large transverse reconstructions over smaller unilateral reconstructions with poor scar placement.16 

To compare 2 methods that preserved greater amounts of native skin than DS, Table 3 illustrates a 5-point ranking comparison of aesthetic factors between IR and DA reconstructions. For all aesthetic variables, our participants ranked IR as more aesthetically appealing, which exemplifies the importance of symmetry in reconstruction. Although IR preserved more native breast skin than DA, we believe that rankings were higher for IR primarily due to the location of scar positioning, which is much more symmetric. This notion is supported by the difference in rating being greatest for breast scar (0.71). When dividing the aesthetic factor ratings by images before and after NAC reconstruction, we see that all rankings are higher following NAC reconstruction, which may reflect asymmetry being better tolerated after NAC reconstruction. This trend is consistent with well-established improvements in patient satisfaction seen with NAC reconstruction in the current literature.12,18

A secondary aim of this study was to assess the validity of MTurk as a survey tool. MTurk allows for rapid and diverse feedback from an on-demand human workforce from around the globe.19,20 This tool has been applied in the behavioral sciences, as well as in plastic surgery literature with a study characterizing important attributes of aesthetic surgeons.21,22 Through our experience, we found MTurk to be a useful research tool with several advantages. Our results for ranking the aesthetic outcome of the 4 types of reconstruction have the exact same ranking order as the patients in the Garza et al study, despite the study requiring participants to complete the survey in person rather than online. Although a large majority of our participants were not patients with breast cancer, we see that laypeople’s opinions of what is aesthetically appealing are more similar to those of patients than of plastic surgeons, which Garza et al found to be different.7 By re-creating the results for rankings as well as the improvement of aesthetics following NAC reconstruction, we find MTurk to be a reliable research tool that is particularly useful when survey recruitment is challenging or if a very large sample size is desired. 

MTurk also provides an affordable platform where researchers can set their own price, with a typical fee of $1 being charged for shorter, 10 to 15-minute surveys. This is particularly useful when cost is a limiting factor for a research study, and the cost can be further minimized if the survey requires a shorter time commitment. However, we do recommend survey compensations be increased to match at least minimum wage, to adequately reward participants and ensure quality responses. The MTurk setup allows for participants to be quickly recruited, as several responses can be collected within just a few minutes, which can tremendously speed up the research process. This study in particular was able to recruit 150 participants within 6 hours. 

Though MTurk offers several advantages, the limitations of both the platform and this study must be considered. Though large subject recruitment is easily attainable through MTurk, we found the quality of responses must be monitored to ensure participants are actively engaging and providing accurate responses. However, this can be controlled for by including control questions, and MTurk allows responses to be reviewed before providing payment to the participant to ensure the control question was answered correctly. Out of a total of 256 responses within hours, only 132 participants (51.5%) were included for analysis after assessing correct answers to control questions and responses  differing ranks. In other words, we had to exclude responses that did not answer control questions correctly or those in which respondents had ranked all reconstructions the same number. Another option to optimize response quality includes offering a higher monetary compensation for completing the survey or adding MTurk filters for users who have received >90% positive feedback on other MTurk tasks. 

Limitations

Limitations of this study include the aforementioned issues with the MTurk platform, which resulted in a smaller-than-intended sample size, and lack of a validated breast reconstruction photo assessment tool. The digitally edited images of the symmetric breast reconstruction may be an unrealistic result compared with most reconstructive results. Additionally, selection bias may be present. Future directions include increasing sample size with more funding, incorporating stronger quality assessment measures for MTurk responses (eg, instructing participants about exclusion criteria explicitly at onset of survey), and more directly examining symmetry and scar placement impacts on aesthetic outcome (ie, by directly comparing qualities of IR and DS reconstructions before and after NAC reconstruction and tattoo). Our study was also limited to participants in the US, and our participant demographics could have been more diverse to better represent the population; an element of self-selection may have been present in our sample. Further studies focused in different geographical areas are needed to determine whether breast reconstruction shape preferences varies based on culture. 

Conclusions

In bilateral autologous breast reconstruction, more symmetric breast scars are rated higher aesthetically than those reconstructions with nonsymmetric scarring. Our findings also suggest that subjects preferred maintaining scar symmetry over preservation of native breast skin. In this way, this study challenges traditional plastic and reconstructive surgical principles that aim to preserve maximal native tissue, and our findings suggest that scar symmetry should be optimized as a reconstructive goal with aesthetically pleasing outcomes. Therefore, when counseling patients who are seeking bilateral autologous breast reconstruction, surgeons should individualize the discussion and promote informed decision-making, which can include setting priorities between scar symmetry or preservation of native breast skin.

Additionally, individual breast aesthetic factors were consistently rated higher after NAC reconstruction than without NAC reconstruction. These findings are consistent with previous studies utilizing non-crowdsourced participants, which demonstrates the potential for online crowdsourcing to be used as a powerful and reliable tool to assess the general public’s preference and subsequently improve patient satisfaction in plastic surgery. 

Acknowledgments

Affiliations: 1Division of Plastic Surgery, University of Utah Hospitals & Clinics, Salt Lake City, Utah; 2Division of Urology, Rush University Medical Center, Chicago, Illinois; 3Civis Analytics, Chicago, Illinois; 4Garza Galante Plastic Surgery, Schererville, Indiana; 5Division of Plastic and Reconstructive Surgery, Stanford University Medical Center, Palo Alto, California

Correspondence: Ashraf A. Patel, MD; Ash.Patel@hsc.utah.edu

Disclosures: The authors disclose no financial interest in any of the products, devices, or drugs mentioned in this manuscript.

References

1.         Kronowitz SJ, Hunt KK, Kuerer HM, et al. Delayed-immediate breast reconstruction. Plast Reconstr Surg. 2004;113(6):1617-1628. doi:10.1097/01.prs.0000117192.54945.88

2.         Mcneese MD, Buchholz TA, Strom EA, Robb GL. Delayed-immediate breast reconstruction. 2003:1617-1628. doi:10.1097/01.PRS.0000117192.54945.88

3.         Kronowitz SJ, Robb GL. Radiation therapy and breast reconstruction: a critical review of the literature. Plast Reconstr Surg. 2009;124(2):395-408. doi:10.1097/PRS.0b013e3181aee987

4.         Schaverien MV, Macmillan RD, McCulley SJ. Is immediate autologous breast reconstruction with postoperative radiotherapy good practice?: a systematic review of the literature. J Plast Reconstr Aesthet Surg. 2013;66(12):1637-1651. doi:10.1016/j.bjps.2013.06.059

5.         Lee CN, Pignone MP, Deal AM, et al. Accuracy of predictions of patients with breast cancer of future well-being after immediate breast reconstruction. 2019;153(4):e176112. doi:10.1001/jamasurg.2017.6112

6.         Sheehan J, Sherman KA, Lam T, Boyages J. Association of information satisfaction, psychological distress and monitoring coping style with post-decision regret following breast reconstruction. Psychooncology. 2007;16(4):342-351. doi:10.1002/pon

7.         Garza RM, Chen TA, Lee GK. Should we excise native breast skin envelope to achieve symmetric scars in bilateral autologous breast reconstruction? A survey of surgeon and patient preference. 2016;76(Suppl 3):175-178. doi:10.1097/SAP.0000000000000775

8.         Wachter T, Edlinger M, Foerg C, et al. Differences between patients and medical professionals in the evaluation of aesthetic outcome following breast reconstruction with implants. J Plast Reconstr Aesthet Surg. 2014;67(8):1111-1117. doi:10.1016/j.bjps.2014.04.004

9.         Naran S, Wes AM, Mazzaferro DM, Bartlett SP, Taylor JA. More than meets the eye: the effect of intercanthal distance on perception of beauty and personality. 2018;29(1):40-44. doi:10.1097/SCS.0000000000004072

10.       Yip JM, Watson DI, Tiggemann M, Hsia S, Smallman AE, Dean NR. Determinants of breast reconstruction outcome: how important is volume symmetry? J Plast Reconstr Aesthet Surg. 2015;68(5):679-685. doi:10.1016/j.bjps.2014.12.038

11.       Edsander-Nord A  null, Brandberg Y, Wickman M. Quality of life, patients’ satisfaction, and aesthetic outcome after pedicled or free TRAM flap breast surgery. Plast Reconstr Surg. 2001;107(5):1142-1153; discussion 1154-1155. doi:10.1097/00006534-200104150-00007

12.       Shaikh-Naidu N, Preminger BA, Rogers K, Messina P, Gayle LB. Determinants of aesthetic satisfaction following TRAM and implant breast reconstruction. Ann Plast Surg. 2004;52(5):465-470; discussion 470. doi:10.1097/01.sap.0000123901.06133.b7

13.       Ter Louw RP, Patel KM, Sosin M, Weissler JM, Nahabedian MY. Patient-centred decision making in breast reconstruction utilising the delayed-immediate algorithm. J Plast Reconstr Aesthet Surg. 2014;67(4):477-482. doi:10.1016/j.bjps.2013.12.049

14.       Albino FP, Patel KM, Smith JR, Nahabedian MY. Delayed versus delayed-immediate autologous breast reconstruction: a blinded evaluation of aesthetic outcomes. Arch Plast Surg. 2014;41(3):264-270. doi:10.5999/aps.2014.41.3.264

15.       Spear SL, Davison SP. Aesthetic subunits of the breast. Plast Reconstr Surg. 2003;112(2):440-447. doi:10.1097/01.PRS.0000070486.35968.38

16.       Coutinho M, Southern S, Ramakrishnan V, Watt D, Fourie L, Sharpe DT. The aesthetic implication of scar position in breast reconstruction. Br J Plast Surg. 2001;54(4):326-330. doi:10.1054/bjps.2001.3563

17.       Bailey S, Saint-Cyr M, Zhang K, et al. Breast reconstruction with the latissimus dorsi flap: women’s preference for scar location. Plast Reconstr Surg. 2010;126(2):358-365. doi:10.1097/PRS.0b013e3181de1b41

18.       Momoh AO, Colakoglu S, de Blacam C, et al. The impact of nipple reconstruction on patient satisfaction in breast reconstruction. Ann Plast Surg. 2012;69(4):389-393. doi:10.1097/SAP.0b013e318246e572

19.       Deal SB, D M, Lendvay TS, et al. Crowd-sourced assessment of technical skills: an opportunity for improvement in the assessment of laparoscopic surgical skills. Am J Surg. 2018;211(2):398-404. doi:10.1016/j.amjsurg.2015.09.005

20.       Schmidt GB, Jettinghoff WM. Using Amazon Mechanical Turk and other compensated crowdsourcing sites. Business Horizons. 2016;59(4):391-400. doi:10.1016/j.bushor.2016.02.004

21.       Bates JA, Lanza BA. Conducting psychology student research via the Mechanical Turk crowdsourcing service. N Am J Psychol. 2013;15:385-394.

22.       Wu C, Scott Hultman C, Diegidio P, et al. What do our patients truly want? Conjoint analysis of an aesthetic plastic surgery practice using internet crowdsourcing. Aesthet Surg J. 2017;37(1):105-118. doi:10.1093/asj/sjw143

Advertisement

Advertisement

Advertisement