The effect of implementing cognitive load theory-based design principles in virtual reality simulation training of surgical skills: a randomized controlled trial

In this randomized controlled trial of the effect of implementing two CL-theory-based instructional design principles in the initial VR training of novices in the mastoidectomy procedure, we found that worked examples followed by a problem completion exercise did not reduce the reaction time in the secondary task nor the performance in the primary task in the post-training procedures. In contrast, the intervention increased the reaction time and reduced the subsequent performance of the procedure. In the following, we will discuss the change in reaction time as a change in CL as established in the literature [5, 6].

Few studies have investigated CL during surgical skills training. Adding mental arithmetic problems, and thereby additional CL, to surgical tasks in a VR laparoscopic simulator was found to increase primary task time to completion [17]. CL measured by mental effort questionnaires has been found to be correlated with time and number of movements in a salpingectomy/salpingotomy VR simulator [18]. This is the first study on the effect of implementing CL-theory-based design principles in initial instructions in a VR surgical simulator with post-training outcome measurements directly on the CL as well as on the primary task performance. A strength of our study is the randomized, controlled study design. The difference in age and semester between the two groups, resulting from the randomization, could not be shown to impact the CL or the final-product performance of the procedure most likely because the participants were all medical students with no exposure to temporal bone surgery, which is not a part of the curriculum. We have no reason to suspect differences in the participants’ cognitive capacity because of the similar background and the randomization, in addition to such measurement being difficult to meaningfully conduct. We found that the study size was sufficient to significantly demonstrate an effect of the intervention even though the effect was the opposite of what we hypothesized and the sample size therefore sufficient.

A limitation to our study is that we did not evaluate the CL and performance during the initial training session and therefore have no knowledge on whether the suggested principles reduced CL during the instruction. Nevertheless, an effect of reducing the CL during initial training should be durable in following training to be meaningful. In addition, any effect of repeated practice using the two different instructional approaches and the effect on the long-term retention of the procedure remains uninvestigated and should be addressed in future studies.

The differences in performance and relative reaction time between the intervention and control groups were most marked for the first post-training session, and by the second session, differences in reaction time and mastoidectomy performance between the intervention and control groups started to even out. The effect of repetition on final product and time to completion of the procedure is substantial especially in the initial part of the learning curve [19–21] and is likely to contribute to the performance increase from the first to the second procedure. Any positive longer term effect on learning of the intervention with the investigated CL-lowering design principles would have to exceed the effect of repetition alone, which both could be difficult to achieve and would require a dedicated study.

In our study, we found the CL to be increased for both the interventional group and control group, and we have previously found that CL is very high in traditional cadaveric dissection training of mastoidectomy as well [4]. This indicates that the mastoidectomy procedure places heavy cognitive demands on the surgical novice. Consequently, there is every reason to consider the CL in the learning situation. Cognitive overload detrimental to learning and skills acquisition can easily ensue in complex learning situations especially in the context of novice training. Some components of CL can be difficult to adjust: germane load is essential for learning and consists of the construction of cognitive schemas and should be optimized or increased rather than reduced and intrinsic load is dependent on the task and the level of element interactivity: by design, simulation often reduces element interactivity compared with more high-fidelity training modalities and this could be beneficial for low-expertise learners in the initial training [22]. However, extraneous load is non-essential for learning and often results from a poorly designed instruction which is the obvious area to improve first to manage CL in the learning situation.

In this study, we implemented the worked example and completion CL-theory-based design principles [3] into the simulation software. However, this also resulted in a part-task segmentation of the procedure during initial training. Part-task training of surgical technical skills has been much debated: a recent review of training design in relation to procedural skills in laparoscopic surgery concludes that empirical evidence of the efficacy of part-task training is scarce [23]. Conflicting results are found for simulation-based training of other procedures too: part-task training is reported to benefit skills acquisition and retention in central venous catheter education [24] whereas part-task training is not superior to a whole-task approach in flexible fibreoptic intubation training [25].

One reason for the inconsistent effect of part-task training of surgical technical skills could relate to the overall complexity of the procedures. Part-task approaches “work well if there are few interactions between the elements, but they do not work well if the elements are interrelated because the whole is then more than the sum of its parts” [26]. This could be a major explanation for the increased CL of the intervention group in our study. Even though our instructional intervention in the initial training utilized the part-task method of segmentation with backward chaining (prior task as successively added) [26] and not fractionation (deconstruction into individual elements trained separately), the integration of inter-related part tasks could contribute to rather than deal with complexity, adding to the germane load.

Worked examples are suggested to work by focusing the learner’s attention on only the necessary and relevant steps for the solution and completion tasks by presenting a partial solution to be completed, not needing to take all other potential steps into consideration [22]. Both principles could potentially reduce the extraneous load and reduce the total CL of the learning situation. Nevertheless, we find evidence of a negative effect on the following compound procedure, possibly because of increased cognitive resources being used on combining and integrating the inter-related part tasks. A balance should be found “between the advantages of whole-task practice and the disadvantages of cognitive overload caused by whole tasks that are too complex for learners” [26]. In terms of CL theory, part-task segmentation could increase the cognitive load used for the mental formation of schemas and connections and thereby increase the germane load to a point resulting in cognitive over load. This is especially a concern for novices with limited prior knowledge to anchor the part-task learning experiences to. In our study, we cannot separate the different components of cognitive load and establish that the intervention increased the germane load. However, it should in general be considered that there is some controversy on the issue of separating the different components of CL [7], and more validity evidence is needed for the different measures of CL including both secondary task performance and self-reported measures [9].

Most advanced VR surgical simulators are commercial and therefore difficult to modify to the necessary extent. However, in our study, we used a non-commercial simulator developed by the authors (PTM and MSS), making changes to the instructions and instructional design in relation to the studied CL lowering principles feasible. Other of the proposed design principles could be feasible in relation to other VR simulators, and the different CL lowering strategies should be considered in the specific context. Close collaboration between clinician experts, simulator developers, and medical educationalists could guide future VR simulator design because evidence-based training strategies are necessary to create training programs that allow novice trainees to learn complex but essential surgical procedures as efficiently as possible.

Simulation developers need to carefully reflect on the consequences of the instructional design choices, and our results indicate that VR simulation-based training of very complex psychomotor skills and procedures such as mastoidectomy should consider other CL-theory-based principles than the worked example and completion principles. These could for example be the integration of instructions and information directly into the visual operating field and by scaffolding instructions and feedback accommodating the novices’ needs. This should be explored in the future and could provide a better strategy for the novice learner than the CL-lowering strategies implemented in this study.