The Impact of a Vascular Learning Management System on Surgical Trainee Education

ABSTRACT: Introduction: This study aimed to determine the impact of an online, structured learning management system (LMS) on the learning of vascular surgical trainees by comparing qualifying examination performance with and without LMS utilization. Method: The vascular LMS course, inclusive of an online curriculum, peer-reviewed textbook, and 717 board-style vascular practice questions with explanations, was created in 2010. This course was made available to vascular trainees nationally as a preparation tool for the written vascular surgery board examination between 2011 and 2012. Following completion of the board exams, all participating trainees were invited to participate in a survey to determine the impact the vascular LMS had on their examination outcome. Statistical analysis was conducted via descriptive statistics and Pearson Chi-square with P<.05 deemed statistically significant. Results: Over a 2-year period (2011-2012), a total of 148 vascular surgeons participated in the online vascular surgery review course of which 32 (22%) participated in the survey to determine board score outcomes of the cohort. Of the 32 respondents, 18 (58%) completed the entire online review course, while 14 (45%) did not. 94% (17) of the surgeons who completed the course passed their written exam compared with only 61% (8) who did not complete the course (P<.05). Performance in the review course correlated with perceived difficulty of the written board exam (r²=.81). Conclusion: Vascular surgeons who complete a robust online program are over 10 times more likely to pass their exam compared to those who do not prepare in a similar manner.

VASCULAR DISEASE MANAGEMENT 2014;11(8):E172-E176

Key words: vascular training, vascular surgery education, surgical trainee, surgical education, learning management system, online learning curriculum

__________________________

The Vascular Surgery Qualifying Examination is a 6-hour examination composed of multiple-choice questions offered annually to surgery residents and fellows who have completed vascular surgery training.¹ Vascular surgery training programs utilize a variety of tools to help prepare their trainees for board certification, including didactic sessions, problem-based learning, practice questions, case-based discussions, and simulation-based training.^2-5 Trainees are increasingly using a structured curriculum in the form of the Vascular Education Self-Assessment Program (VESAP) and other third-party programs to augment their education.⁶

A learning management system (LMS) permits the delivery of an online, structured education program that permits interactive, accessible, and multimedia delivery of individualized content.^6,7 An LMS can provide a curriculum to organize studying, incorporating a textbook to describe key concepts, and reinforcing these concepts through practice questions.⁷ 

A surgeon-led LMS was created in 2010 to facilitate vascular fellowship learning comprised of an internally published textbook, online structured curriculum, and a large board-style question bank. This study aimed to determine the impact of an online, structured LMS on the learning of vascular surgical trainees by comparing qualifying examination performance with and without LMS utilization.

Materials and Methods 

Study Design

An internally created curriculum that included a peer-reviewed vascular textbook and 717 board-style practice questions was made available to junior and senior vascular surgery fellows in 2011-2012 academic year.⁸ Surveys were emailed to vascular fellows who completed the year-long LMS course to request their vascular surgery qualifying examination performance in order to compare outcomes of those trainees that utilized the course to those who used other methods. Perceived difficulty of the exam was included as a question within the survey because objective measures of test difficulty were not available for this particular exam.  

The number of trainees who completed the review course was determined by the online course management system, which tracks resident progress via reading material and practice examination performance. 

Course Materials

The review course includes online access to an internally published vascular surgery review textbook, an interactive question review program that divides 717 vascular surgery board-style questions into major subject areas, and a course architecture that provides immediate feedback on performance and progress. A published copy of the Clinical Review of Vascular Surgery was provided to all participants. The format of the course followed the content outline published by the American Board of Surgery on the vascular surgery written examination. 

 Survey Instrument

An online survey was sent to all participants in 2012 upon completion of the vascular surgery board exams (Table 1). Surveys were emailed to each participant requesting their qualifying examination performance from 2011 (second-year fellows) and 2012 (first- and second-year fellows) to compare outcomes in those trainees that utilized the course compared to those who did not complete the course.

Statistical Analysis  

Data was de-identified and placed into a Microsoft Excel spreadsheet. Statistical analysis was conducted via descriptive statistics and Pearson Chi-square with P<.05 deemed statistically significant. 

Results

Over a 2-year period (2011-2012), 148 vascular surgeons participated in an online vascular surgery review course utilizing the LMS. These surgeons were sent a brief survey inquiring about their exam experience and whether they passed their vascular surgery qualifying examination; 32 (22%) responded to the survey of which 18 (58%) participants completed the entire online review course and 14 (45%) did not.  

94% (17) of the surgeons who completed the course passed their written exam while only 8 (61%) who did not take the course passed their exam (P<.05, OR 10.6, 95% CI 1.06-106.60). The average score in the online review course for participants who completed the course was 68 +/- 12%. Overall performance in the review course (percent score) inversely correlated with the perceived difficulty of the qualifying exam (r2=.81).

Discussion

Structured learning tools have been used with success in other disciplines in medicine.^6,9 Utilization of educational programs has previously been shown to improve examination performance on general surgery examinations.¹⁰

Our study found a strong correlation between completing an educational review program and passing the vascular surgery qualifying examination. Furthermore, high scores on the educational tool were correlated with a lower perceived difficulty of the qualifying examination. The use of a multimodal tool for education appears to be effective among surgery trainees.¹¹ 

Other medical disciplines have utilized structured learning tools with success; radiology residents have employed web-based LMS systems to improved diagnosis in polytrauma while interactive algorithms and online teaching tools have been used with success by a Slovakian online education portal resulting in greater utilization of learning tools and student satisfaction.⁹ For trainees in the surgical community, any learning tool should aim at meeting three goals: (1) provide a comprehensive educational structure with a focus on important concepts in surgery, (2) encourage repetition, reinforcement, and feedback, and (3) provide accessibility and flexibility. Unfortunately, comprehensive reviews of surgery are difficult to complete in a purely didactic environment.^10,11 Thorough reading of major reference books in surgery is not only difficult to complete but has significant issues with retention, and reviewing practice questions without a sufficient fund of knowledge is also ineffective.^12,13 

The utilization of a multimodal LMS such as the one that we studied offers trainees ample opportunity for practice and reinforcement while addressing different learning styles by offering multimedia content. This learning program uses a faculty and peer-developed review textbook in vascular surgery to provide key concepts in surgery while rigorously supplementing active learning through a database of practice questions that reinforce major topics and concepts in vascular surgery. Slide presentations and audio/video lectures cover important topics in vascular surgery and the LMS is geared to provide immediate feedback to trainees about their own performance and performance compared nationally to their peers. This type of approach to comprehensive learning has previously been shown to be effective when assisting residents with poor performance on surgery training examinations.¹⁴ While our data is promising, the low survey response rate makes it challenging to conclude that the LMS alone was the primary contributor to academic success. It is possible that students who completed the LMS were more motivated than those who did not and perhaps studied other material to supplement their learning. Alternatively, it is possible that any type of studying from an organized system results in better outcomes. Regardless, our study did find that those who utilized an LMS performed better on their exam than those who did not. 

Limitations

This study was limited by the fact that a survey collection was used as a data tool. We were unable to control for the variation in educational tools utilized by the trainees in the limb of the study that did not utilize the structured LMS. 

Conclusion

Vascular surgeons who complete a robust online program that includes reading material and board-style practice questions in a timely manner before their vascular surgery qualifying examination are over 10 times more likely to pass their exam compared to those who do not prepare in a similar manner. A properly constructed online education curriculum may positively impact vascular surgery learning and could provide an additional learning resource for the busy vascular surgeon. n

Editor’s Note: Disclosure: The authors have completed and returned the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr. Dua reports no disclosures related to the content of this manuscript. Dr. Desai reports that he is CEO of Surgisphere, which creates learning management systems for surgical trainees, including the one studied in this paper.

Manuscript received March 6, 2014; provisional acceptance given April 16, 2014; final version accepted May 6, 2014. 

Address for correspondence: Anahita Dua, MD, University of Texas-Houston, Center for Translational Injury Research, Department of Surgery, MSB 5.030, 6413 Fannin Street, Houston, TX 77034, United States. Email: anahita.dua@uth.tmc.edu.

References  

1. Barshes NR, Menard MT, Nguyen LL, Bafford R, 1.Vascular Surgery Board of the American Board of Surgery. Booklet of Information: Vascular Surgery. 2011. https://www.absurgery.org/xfer/BookletofInfo-VS.pdf. 
2. Reed AB. Advanced training in vascular surgery: how does it need to change? Semin Vasc Surg. 2006;19(4):191-193.
3. Dunkin BJ, Donovan M, Bass B. Methodist Institute for Technology, Innovation and Education. J Surg Educ. 2011;68(1):79-82. 
4. Dalsing MC, Makaroun MS, Harris LM, Mills JL, Eidt J, Eckert GJ. Association of Program Directors in Vascular Surgery (APDVS) survey of program selection, knowledge acquisition, and education provided as viewed by vascular trainees from two different training paradigms. J Vasc Surg. 2012;55(2):588-597.
5. Sidawy AN, Sumpio B, Clowes AW, Rhodes RS. Basic science curriculum in vascular surgery residency. J Vasc Surg. 2001;33(4):854-860.
6. Dua A, Sudan R, Desai SS. Improvement in American Board of Surgery In-Training Examination performance with a multidisciplinary surgeon-directed integrated learning platform. J Surg Educ. 2014;pii:S1931-7204(14)00056-7.
7. Pittenger AL, LimBybliw AL. Peer-led team learning in an online course on controversial medication issues and the US healthcare system. Am J Pharm Educ. 2013;77(7):150. 
8. Surgisphere Corporation. Vascular Surgery In-Training Examination (VSITE). https://clinicalreview.com/ClinicalReview/review-courses/vascular-surgery-review/vsite.html
9. Schlorhaufer C, Behrends M, Diekhaus G, Keberle M, Weidemann J. Implementation of a web-based, interactive polytrauma tutorial in computed tomography for radiology residents: How we do it. Eur J Radiol. 2012;81(12):3942-3946.
10. de Virgilio C, Stabile BE. Weekly reading assignments and examinations result in sustained improvement in American Board of Surgery In-Training Examination (ABSITE) scores. Am Surg. 2005;71(10):830-832.
11. Kim RH, Gilbert T, Ristig K, Chu QD. Surgical resident learning styles: faculty and resident accuracy at identification of preferences and impact on ABSITE scores. J Surg Res. 2013;184(1):31-36.  
12. Schwarz D, Štourač P, Komenda M, et al. Interactive algorithms for teaching and learning acute medicine in the network of medical faculties MEFANET. J Med Internet Res. 2013;15(7):e135. 
13. Harthun NL, Schirmer BD, Sanfey H. Remediation of low ABSITE scores. Curr Surg. 2005;62(5):539-542.
14. Corneille MG, Willis R, Stewart RM, Dent DL. Performance on brief practice examination identifies residents at risk for poor ABSITE and ABS qualifying examination performance. J Surg Educ. 2011;68(3):246-249.