Intensive simulation versus control in the assessment of time to skill competency and confidence of medical students to assess and manage cardiovascular and respiratory conditions—a pseudo-randomised trial

Setting

This study was conducted at the St Vincent’s Clinical Education and Simulation Centre, St Vincent’s Hospital, Melbourne. Ethical approval was obtained through The University of Melbourne and St Vincent’s Hospital, Melbourne.

Inclusion criteria

This includes The University of Melbourne, St Vincent’s Clinical School, final year medical students who were undertaking clinical rotations in cardiovascular and respiratory medicine at St Vincent’s Hospital. Prior to allocation, all students had received standard University of Melbourne education on cardiovascular and respiratory medicine as part of their undergraduate pre-clinical training.

Recruitment, consent and pre-intervention data collection

Eligible students were invited to attend the CPEP course and the Observed Structured Clinical Examination (OSCE) testing days. Attendance was voluntary and written consent was obtained for all participants. Participants were requested to provide information regarding their knowledge, skills and attitudes in cardiovascular and respiratory medicine; self-rated proficiency in clinical skills; and basic demographic information (gender, age, cultural heritage, whether English is their first language, number of years speaking English).

Participant allocation

Participants were pseudo-randomised to the project by St Vincent’s Hospital, The University of Melbourne Clinical School (Fig. 1) [18]. All participants were allocated a random identifier number to label all collectable data sheets during the study. The University of Melbourne Clinical School decided determination of control and intervention group allocation, and the course dates were set to coincide with the commencement of clinical rotations. Six CPEP courses were conducted over a 2-year period. Participants and rating clinicians were both blinded to group allocation.

Procedure

The control group participants attended the normal programme of the 6–8-week cardiovascular and respiratory medicine clinical rotation.

The intervention group participated in CPEP, a 4-day immersive simulation course, in the first week of their cardiac and respiratory medicine clinical rotation. They then completed the remaining 5–7 weeks of the cardiovascular and respiratory rotation.

CPEP course

The CPEP was a 4-day course designed to provide students with intensive education at the start of their cardiovascular and respiratory clinical placements (Additional file 1). It used a combination of interactive teaching methods to guide the students in essential areas of key knowledge, skills and attitudes. It encouraged students to reflect upon their current level of performance in patient assessment and management and to self-identify areas of deficit. The course was designed to augment and not replace the current cardiovascular and respiratory curriculum and integrate patient safety, communication skills, professionalism and a teaching model of Kolb throughout.

Control

The normal programme of the 6–8-week cardiovascular and respiratory medicine clinical rotations consisted of a combination of formal and informal teaching. The formal component included lectures and attendance at clinical meetings. The informal component included history taking and examination of patients, case presentations to registrar/consultant and observation of clinical staff in their day to day work. This observed clinical work would involve communicating with and examining patients, ordering investigations and performing procedures. There was no simulation-based education as part of the normal clinical rotations.

Outcomes—time to develop competencies

The primary outcome was the participant’s score on a subsequent OSCE exam—this was to analyse whether the intervention group could perform competently after CPEP in comparison to the control group who were being assessed at the end of their full rotations.

The OSCE score was taken at one time-point only for each participant; baseline measures were not taken.

The OSCE was chosen as an objective-testing tool as it was the current standardised examination for undergraduate cardiovascular and respiratory medicine used by The University of Melbourne. OSCEs are currently conducted around the world and provide an opportunity to be rated according to a global rating scale against a range of competencies. OSCEs are set for a range of knowledge areas and can be conducted in the same way in a variety of environments [16, 19, 20].

All participants attended an examination day, participating in the same two cardiovascular and respiratory OSCE stations. These OSCE stations consisted of examining and presenting an assessment of actual patients, as would occur during University of Melbourne examinations. The OSCEs were scored based on a marking criteria scoring key used by The University of Melbourne, out of possible total of 30 marks. OSCE data was collected at one point for each intervention and control group. The CPEP group attended OSCE data collection on the day after the course had finished; the control group attended in the week following their 6-week cardiovascular and respiratory rotation. All OSCE examiners were experienced, having previously examined OSCE stations for The University of Melbourne Medical School using the same scoring key. OSCE examiners were blinded to group allocation. The same examiners were not available for all the OSCE examinations.

Outcomes—confidence rating

Secondary outcome measures were obtained from participant surveys—baseline, post-course, and end of rotation. The surveys were designed to record self-rated confidence in knowledge, skills and management tasks for 17 clinical skills relevant to cardiovascular and respiratory medicine (Additional file 2). A five-point multi-category rating scale of confidence was used with descriptors (never confident: I am challenged or threatened by this topic or skill; rarely confident: I am well outside my comfort zone; sometimes confident: I am sometimes confident with this topic or skill but could improve; usually confident: I am comfortable with this topic or skill; always confident: I am expert at this topic or skill). Basic demographic information was also obtained to assist in the assessment of confounding factors for their clinical skill development.

The baseline survey was given to all participants on day 1 of their clinical rotations (day 1 of CPEP, prior to commencement of the teaching programme).

The post-course survey was given to the intervention group immediately following completion of the course.

The end of rotation survey was given to all participants at the end of their rotation. An analysis of these follow-up surveys, using the same confidence-rating scales, was conducted to examine the self-assessment scoring of the control and intervention groups participants.

Sample size estimation

Based on the existing data for OSCE performance, it was anticipated that the control group would have a mean OSCE score of 70.0 ± 23.1 (SD). In order to achieve a mean difference of 12 points between the simulation-exposed group and controls, a sample size of 59 per group was required with alpha set at 0.05 (two-tailed) and power set at 80 %. Assuming a dropout rate of approximately 10 %, a sample size of 65 per group was planned.

Statistical analyses

Demographic data were collected on a standard pro forma directly from the participants. OSCE scores were collected following each examination session. Baseline, post-course and follow-up survey data were collected directly from the participants. Data were analysed using the Statistical Package for the Social Sciences (SPSS; version 15.0, IBM). Descriptives (number, percentage) were calculated for demographics and OSCE data (median, interquartile range (IQR)). OSCE total scores for the intervention and control groups were not normally distributed and were therefore compared using the Mann-Whitney U test. OSCE subcomponents for intervention and control groups were analysed using Fisher’s exact test for categorical data.

For inferential analyses, confidence data were collapsed from five response categories (‘never confident’, ‘rarely confident’, ‘sometimes confident’, ‘usually confident’ and ‘always confident’) to two: ‘never/rarely/sometimes confident’ and ‘usually/always confident’. Confidence data for intervention and control groups were compared for baseline and for post-test using Fisher’s exact test. Two-tailed tests of significance were used in all instances and alpha was set at 0.05.

Demographics

The students were aged between 20 and 36 (mean 23.55, standard deviation 3.078) and included direct school leavers (47.1 %), graduates (23.1 %) and international students (31.4 %). There were 53 males (43.8 %) and 68 females (56.2 %). Respondents identified 22 different cultural backgrounds, the majority describing themselves as either Chinese (28.9 %) or Australian (26.4 %). English as a second language was spoken by 47 (38.8 %) students, and of these, some had only been speaking English for 2 years.

Competency outcomes—OSCE data

There were no significant differences in OSCE scores between the intervention group assessed at the end of CPEP and the control group assessed at the end of rotation.

The median respiratory OSCE score for the CPEP group (N = 64) was 25.0 (IQR 23.0–27.0) compared to 26.0 (IQR 22.5–28.5) for the control group (N = 58) (P = 0.08). The median cardiovascular OSCE score for the CPEP group (N = 58) was 26.0 (IQR 23.7–27.0) compared to 25.7 (IQR 23.7–27.0) for the control group (N = 58) (P = 0.176).

One of the actual patients (cardiac) was too unwell to participate in the OSCE.

Confidence rating—survey data

Data relating to students’ self-reported confidence levels are presented in Figs. 2, 3 and 4.

Students in both groups (CPEP and control) self-reported similar levels of confidence in managing cardiovascular and respiratory conditions at prior to commencement of CPEP and clinical placements (Fig. 2).

At the end of the 4-day CPEP intervention, CPEP students reported confidence levels that were similar to those reported by the control group at the end of their 6-week placement (Fig. 3). At the end of their 6-week placement, 84.2 % of CPEP students reported that they were ‘usually’ or ‘always’ confident in managing cardiorespiratory conditions compared with 48.2 % of those in the control group (Fig. 3).

Self-reported confidence in the assessment and management of acute myocardial infarction are shown in Fig. 4.

The full results from the end of rotation surveys are available as online supplemental material in Additional file 3.

Competency outcomes—OSCE data

The primary objective of this study was to assess whether an intensive 4-day simulation course would result in competency outcomes that were comparable to those achieved by a standard 6-week clinical rotation in cardiovascular and respiratory medicine. This was not with the intention of replacing the clinical rotation but to enhance the student’s experience of the rotation by providing them with a strong clinical platform. This comparison used the current assessable standard, the OSCE. The data collected from the OSCEs demonstrated that there was no statistical difference between the intervention group after the first week and the control group OSCE scores at the end of 6 weeks.

This data suggests that at the completion of 4 days of intensive simulation-based education, the intervention group were able to attain the same level of knowledge and skills that the control group had achieved over a 6–8-week period.

Confidence rating—survey data

The secondary objective was to assess student’s confidence in managing cardiovascular and respiratory problems. Self-reported confidence survey results provided a comparison of the ‘post-course’ (intervention) and the ‘end of rotation’ (control) groups. This demonstrated that students exposed to CPEP went into the remaining 5 weeks of their rotation with confidence levels that were similar to levels reported by the control group when they were at the end of their rotation. End of rotation survey confidence scores for the CPEP group showed nearly twice as many rating themselves as ‘usually, always confident’ compared to the control group. This displayed a further increase in confidence from the immediate post-course results.

Front-loading education for the rotation with the intervention group provided the students with an opportunity to build upon their knowledge, skills, and confidence for the remaining 5 weeks. In contrast, there is a more gradual development in the control group and they did not necessarily have the base to build their practical experiences upon. This may have wider curriculum applications with the option to concentrate key knowledge and skill components in the early phase of clinical rotations to optimise efficacy. This may be applicable with or without the use of simulation-based learning and is not limited to medical undergraduate teaching.

The CPEP course had originally been designed to clinically prepare the students for clinical practice, rather than how to pass an OSCE. In fact, the students had only two modules out of a total 16 that were directly related to clinical examination. These modules were presented as interactive sessions (‘how to learn how to do a cardiac/respiratory clinical examination’) and were designed to place the actual clinical relevance of an examination into a real-life context. The same skills and knowledge in examination were provided to the control group at various points in their rotation. Most clinicians and educators would expect students to emerge from a cardiovascular/respiratory term with some core fundamental knowledge and skills in place.

At the conclusion of the CPEP course, all of the intervention students went into their rotation rating themselves as either ‘sometimes or usually confident’. This effect continued to the end of rotation survey with all intervention responders rating themselves as ‘sometimes, usually or always confident’ at this point. Although it is possible that CPEP group of students developed falsely elevated confidence levels through their involvement in the course, the OSCE results suggested that the CPEP students were at an acceptable ‘end of rotation’ level at that point. If we compare this to the control group, it is concerning that there are a number of students who finish a key rotation with no confidence in their ability to practise as a doctor in this area. In fact, there were consistently higher responses that scored ‘rarely or never confident’ in the control group throughout the end of rotation subgroup analysis when compared to intervention group. Anecdotal comments from the participants suggest that some of the intervention group used the CPEP course to become aware early in their rotation about what they did and did not know, closing the gap between perceived and actual knowledge.

There are aspects of the clinical rotation that are difficult to replicate in simulation and are equally hard to measure. These include the real-time observation of clinical practice throughout days or weeks and the social and communication challenges that exposure to real patients provides. Allowing students to actively manage undifferentiated or acutely unwell patients is problematic due to the expectation that more senior staff would take over and the variability of exposure to these types of patients. Take the example of an acute myocardial infarction (AMI)—a heart attack, the leading cause of death in Australia [21]. The expectation would be that a student would learn how to assess and manage this common and life-threatening condition during their undergraduate clinical placement on a cardiology ward. A proportion of the control group finished their rotations being ‘rarely or never confident’ in their ability to either assess (11 % compared to 0 % intervention group) or manage (17 % compared to 0 % intervention group) an AMI (Fig. 4). CPEP did not specifically target the assessment and management of AMI; there were two simulation scenarios and an ECG facilitated discussion during the course.

Despite the prevalence of AMI, it has become increasingly common for medical students to progress through a cardiology term and not be exposed to this condition. Due to the different phases occurring along a patient’s journey from initial diagnosis and treatment (general practice, ambulance or emergency department) to definitive management (thrombolysis or primary coronary angioplasty) and ongoing care on the ward a student could easily go through their whole cardiology term without seeing an undifferentiated patient presenting with an AMI. On a cardiology ward, they are more likely to see an AMI patient after someone else more senior has diagnosed and treated them. The students therefore miss the learning process of taking a history in real time from a patient with chest pain, seeing where all the interventions (aspirin, oxygen, heparin, angiography, etc.) fit in, and communicating with a sick patient and other staff simultaneously. Simulation in a course such as CPEP has the ability to provide medical students with exposure to common conditions presenting as undifferentiated cardinal symptoms (chest pain and breathlessness) that they can attempt to assess and manage. This type of learning prepares the students to become clinically astute junior doctors who have been taught, via senior clinician modeling and feedback, how to approach sick patients with an unknown diagnosis, rather than a defined clinical condition—practicing being a doctor not a medical student. A clinically orientated simulation course provides the student with the opportunity to ask questions of and receive feedback from clinical experts. Time is quarantined for both the educator and the student without the need to compete with other students for patient or clinician time. Learning goals were not limited to specific clinical skills and the students received feedback on teamwork, communication and other non-technical/ professional skills. The key component of having expert feedback cannot be underestimated as previous problem-based curriculum changes may result in improved communication skills but reduced understanding of disease [2, 3].

Simulation fits the criteria of a modern, integrated educational approach designed to meet the challenge of finding ‘other educational strategies that promote student-centred rather than teacher-centred learning, promote active student enquiry, stimulate analytical and knowledge organisation skills, and foster lifelong learning skills’. [5]

Limitations

The authors identified several limitations to this study that may have impacted upon the results.

The OSCE data could suggest that the behaviour marking structure for the assessment is compromised as it assesses a student’s ability to undertake pattern recognition with little if any weighting in assessment being given to the student’s ability to understand why they are performing a task. The OSCE did not measure whether the intervention and control groups had similar levels of understanding and context of why the physical examinations were undertaken. Due to logistical reasons, not all of the participants were assessed by the same examiners which may have lead to some score variation.

A pseudo-randomisation strategy was used due to a number of logistical constraints. Access to the medical students was controlled and organised by The University of Melbourne’s Clinical School at St Vincent’s Hospital. As a result, the research team at St Vincent’s Clinical Education and Simulation Centre did not make the rotation group allocations.

We were advised that student allocation to clinical groups follows a process to maintain group diversity. Clinical groups are 7 to 8 persons in size, gender balanced and with the group population representing three criteria—direct school leavers, graduate students and international students. Groups are also gender balanced.

There are two issues that may have affected the control groups in this study. Not all of the clinical rotations were exactly 6 weeks, and therefore, control group OSCE testing did not always occur exactly at the 6-week mark. Where logistically possible, control groups were allocated OSCE testing times with the intervention group, on the Friday immediately following the 4-day CPEP. In order to achieve this, some control groups attended combined testing at weeks 5 or 7 and 8.

Two out of the six control groups involved in the study attended the end of rotation OSCE stations as an unmixed group (no intervention group participants). To minimise the impact of this potential bias for the OSCE examiners, all examiners were external to the CPEP and blinded as to what group is being examined. The examiners only dealt with a participant by number at the OSCE station they had been allocated to run. No teaching staff from the CPEP were in any way involved in the running of the OSCE examination stations.

In addition, due to limitations in access to resources and access to students the research group felt that the OSCE results may have provided greater assistance if there had been testing pre-rotation/pre-intervention, at the completion of the intervention/first week of rotation and at the completion of the rotational block. These three points for both the intervention and control group may have provided a further understanding as to whether there were any differences in rate of skill acquisition and retention as a result of the program. Although the data shows that the CPEP group after 1 week obtained similar OSCE scores to the control group who had undergone a full rotation, it is not possible to say that the time to skill competency was reduced, as there was no comparison control group OSCE taken at 1 week into the rotation.

Strengths of this study include the validity of groups for comparison, the collection of quantitative as well as qualitative data and the application of the standard assessment tool for this target group—the OSCE.

Research questions

Main messages