A realism questionnaire was designed, applied and analysed for the technical procedure of IUCD insertion. Insertion of IUCD (intrauterine contraceptive device) was chosen as it is a quick, simple procedure that would reasonably be practiced by doctors of varying experience (to capture a cross-section of clinicians), and multiple simulators are available for the procedure. This study was approved by the institution’s Human Research Ethics Committee to meet the requirements of low- and negligible-risk research.
Questionnaire design
Questionnaire design was informed by a review of the literature and semi-structured interview of expert IUCD inserters. In this interview, six gynaecology specialists were provided with an example questionnaire and were joined by a study investigator through a detailed discussion of the key steps of IUCD insertion, as well as the features of a simulator considered to be relevant for the procedure. The final questionnaire items were chosen by consensus and pilot tested. Items in the questionnaire were grouped in subscales similar to a previous publication [10]. The subscales included four aspects of simulator anatomical structure (in terms of ‘appearance’, ‘feel’, ‘response to instruments’ and ‘accuracy’ of composition) and five aspects of simulator function (including ‘action’ of the tissue, replication of ‘procedural steps’, ‘vision’, ‘setup’ and ‘perform procedure overall’) (see Additional file 1, a copy of the questionnaire supplied to participants). Additional items in the survey assessed the user’s overall impression of the simulator. These items were the overall realism, value for training and value for assessment. Within each subscale was a global question, designed to be a single non-procedure-specific question to address the same aspects of simulator realism as the procedure-specific questions in the corresponding subscale. Correlation between global and procedure-specific items was intended to assess the content validity of global components of the questionnaire. Additionally, an importance scale was included for realism items in order to capture user perceptions of the importance of each simulator feature in contributing to their assessment of simulator performance.
Scoring
The questionnaire used 7-point Likert items chosen due to related research [11, 13,14,15], similar reliability to 5-point scales, to reduce interpolations (in the non-electronic format of the questionnaire) [17] and to capture more sensitive degrees of assessment. [17]
Simulation models
Three simulators were investigated for realism assessment for IUCD insertion (see Fig. 1). Models were chosen that appeared to differ in realism without being obviously superior for training and represented the spectrum of IUCD insertion models available at the institution. The first model, the ‘Flat Uterus Model’, was a clear plastic circular representation of the cervix and uterine cavity allowing visualisation of the IUCD insertion but without further anatomy. The second model, the ‘Desktop Uterus Model’, also contained a clear plastic window into the uterine cavity representation as well as structures representing the vagina and a speculum. The third model chosen, the ‘Pelvic Model’ (ZOE Gynecologic Simulator; Gaumard Scientific®), was an opaque pelvic anatomy simulator (capable of being a simulation model for multiple procedures including IUCD insertion) that included a vulva, vagina, cervix and uterus but not allowing visualisation of the uterine insertion of the device.
Participants
Participants were doctors working at a large tertiary hospital in Brisbane. Doctors were from a range of experience levels that would reasonably be expected to attempt an IUCD insertion. Doctors were recruited via hospital education sessions. Participants’ role, age and experience are outlined in Table 1.
Procedure
Doctors were informed of the purpose of the study and given verbal and written instructions for completion. The rationale for the importance scale was explained, and participants were informed how to complete this section in addition to the realism ratings. The doctors performed an IUCD insertion on each of the three simulators and completed the realism questionnaire after each model. Participants performed the IUCD insertions beginning with different models, so there was no set order for the realism assessment. At the end of the three models, participants completed the questions related to the importance of each of the features of the simulator. Questionnaires from participants who were unable to complete all models in the allocated time were kept in the final analysis if all items for a selected model were completed.
Statistical analysis
Realism assessment tool evaluation
Mean scores for all procedure-specific items in each subscale (not including the global question) were determined, to create a ‘mean subscale score’ for each of the nine subscales. The global question scores for each subscale were compared to the mean subscale scores by Pearson’s correlation. Both the global and mean subscale scores were compared to the overall performance score by Pearson’s correlation.
Cronbach’s alpha was used to calculate the interrater reliability in the questionnaire for each model.
Assessment of simulator realism
Mean scores for the global items of each subscale, overall realism, value for training and value for assessment items were compared by ANOVA as an omnibus test for significance and if detected, followed with between-model pairwise comparison of means with Sidak correction. ANOVA was used to evaluate the relationship between experience and overall realism scores.
The mean scores for the importance rating in each subscale were used to assess differences in the participant-rated importance of aspects of simulator design.