Serious games for health: three steps forwards
© The Author(s) 2017
Received: 6 September 2016
Accepted: 11 January 2017
Published: 4 February 2017
Serious games are educational tools which are more and more used in patient and health professional education. In this article, we discuss three main points that developers and educators need to address during the development of a serious game for health. We first explain how to develop motivating serious games by finding a point where the intrinsic and extrinsic motivations of end users can converge. Then, we propose to identify the features of serious games which enhance their learning effectiveness on the basis of a framework derived from cognitive science and called “the four pillars of learning.” Finally, we discuss issues and solutions related to the evaluation of serious games.
KeywordsVideo game Health education Patient education Medical education
New technologies have invaded our world, and while they can impair learning , they should also be considered as important tools to enhance learning in the context of the major shift occurring in education. For thousands of years, the limiting resource in education was the learning content: during the Middle Age in Western Europe, knowledge was contained in rare and precious books which were copied in monasteries, one at a time. Learning content began to be disseminated after the invention of the printing press by Gutenberg around 1450, but this access vastly increased by the World Wide Web and its search engines such as Google (1998). Today, when anyone can find any information on the Internet, learning content is no longer rare and precious, whereas the educational methods, i.e. the process of how knowledge is acquired, draw increasing attention. Information corresponds to facts provided about something or someone. By contrast, knowledge acquisition is the process of extracting, structuring, and organizing information. It requires time and corresponds to the theoretical or practical understanding of a subject, through education, experience, or reflection.
Serious games are potentially interesting tools to acquire knowledge, both for their motivational effect and the pedagogical principles they include such as a user-centred approach, interactivity, repetition, and continuous feedback . They are attracting growing attention in the health area, with the development of large conferences dealing with games such as the “Games for Health” conferences in the USA (https://gamesforhealth.org/) and in Europe (http://www.gamesforhealtheurope.org/). The development of serious games is complex because of their twofold objective, being both motivating and educational. In this article, we discuss three main points that developers and educators need to address during the development of a serious game. We first explain how to develop motivating serious games by finding a point where the intrinsic and extrinsic motivations of the end users can converge. Then, we propose to identify the features of serious games which enhance their learning effectiveness on the basis of a framework derived from cognitive science and called “the four pillars of learning.” Finally, we discuss issues and solutions related to the evaluation of serious games.
The convergence of motivations
A major factor in education is the time allowed to the learning process, a longer time being associated with better learning outcomes [3, 4]. Because video game players spend many hours per week on their computer, video games were considered as an interesting educational method . By incorporating some learning content into video games, it was thought that players using these “serious” games would still enjoy the game while learning for hours . This proposal led to important disappointments about both the enjoyment that serious games may provide and their learning effectiveness. Arden: The World of Shakespeare is a classical illustration of such a failure. This project was intended to be a massively multiplayer online learning game for teaching undergraduate learners the works of William Shakespeare. It was designed to be a plug-in for the off-the-shelf game Neverwinter Nights. When the game was launched, the virtual world turned out to be a great looking but players were dissatisfied with the gameplay and abandoned the game. This led the MacArthur Foundation which initially founded the game to pull out of this ambitious project .
Thus, developing a motivating serious game requires the developer to be interested in the motivation of the end users. Following the self-determination theory of Deci and Ryan, three types of learners can be defined: the amotivated, the extrinsically motivated, and the intrinsically motivated learners . The amotivated learners do not value the activity of learning and do not believe that it will yield a desirable outcome. When asked to use a serious game, the only aspect they enjoy is the “sensory delight” related to the graphical environment and sound effects used in the game, known in the gaming business as “Eye Candy” . Once this transient enjoyment fades, they quickly abandon the serious game. The extrinsically motivated learners do not value the activity of learning but do believe that it can yield a desirable outcome, for example obtaining a high examination score at the end of the training. Their difficulty is to find the energy to learn in order to achieve a future, desirable outcome. Finally, intrinsically motivated learners enjoy the activity of learning itself and/or are interested in the subject matter: from their perspective, the serious game represents one opportunity among many others (conferences, textbooks, etc.) to learn.
Therefore, in our opinion, for both amotivated and intrinsically motivated learners, serious games have little or no motivational value compared to conventional instruction methods, because these tools represent the same kind of constraint for the former and the same kind of opportunity to learn for the latter. This may explain why a meta-analysis of the motivating effect of serious games found that serious games were not more motivating than conventional instruction methods .
This combination was present in a very successful serious game developed in the health area, the Re-Mission game. In this game developed for young cancer patients, players controlled a humanoid nanorobot which had the mission to destroy different types of cancer at a cellular level, while learning the importance of compliance to chemotherapy. Most of these young cancer patients were likely to be extrinsically motivated learners, with a profound hope to reach a very desirable outcome, being rid of their cancer, although not enjoying learning about the importance of being compliant with their chemotherapy. The strength of the game was to make this desirable outcome virtually present, by allowing players to destroy several cancers, while delivering important messages about compliance to chemotherapy. The serious game could preserve both its enjoyment, as attested by the more than 200,000 copies delivered, and its learning effectiveness, demonstrated in a randomized controlled trial involving 375 patients which revealed that playing this game was associated with better knowledge, self-efficacy, and adherence to oral chemotherapies .
By contrast, there are several examples of serious games that failed to be motivating because their developers thought that putting algebra problems in a 3D virtual world or placing the periodic table of the elements in a shooting arcade would motivate students to play while learning . In these situations, which did not involve any desirable outcome, students considered serious games as poorly motivating as their traditional lectures.
Combining the motivation for the activity of learning itself (intrinsic motivation) with the motivation for a future desirable outcome (extrinsic motivation) is thus essential to develop motivating serious games (Fig. 1). It may explain the enthusiasm of medical students for simulation games in which they play the role of medical doctors . The association of their final objective—practicing as a doctor—made virtually present and possible, with a learning activity—making decisions to save the life of a virtual patient—is promising to motivate these students to learn.
Developers of games in the health area should consider this “convergence of motivations” if they want their product to be motivating from the perspective of their learners. However, even a serious game played for hours can fail to reach its educational purpose if developers neglect to consider the key instructional features which will allow the learning process and which are developed below.
Learning benefits expected from serious games: insights from neurosciences
The four pillars of learning, a new framework to enhance the learning effectiveness of serious games
Pillars of learning
Practical implications for serious games development
Practical implications for serious game implementation
Future directions of research
- Alerting network
Reaching medium level of arousal
Links between serious game design, level of arousal, learning effectiveness
- Orienting network
Selecting relevant information
Promote strategies that help to select relevant information (modelling/examples, modality/use of the audio channel for verbal explanations to guide visual search, feedback, integration of relevant information in virtual tools)
Educators should discuss with learners at the end of the serious gaming session in order to ensure that they identified the relevant information
New strategies integrated in games for guiding the player towards relevant information
- Executive network
Staying focused on the serious game
Because learners are not always immersed in the game although they do learn, the environment shall stay clear of distractions during the serious gaming session
Differences between immersion, engagement, and repercussions on learning effectiveness
Being engaged during learning
Promote interactivity rather than convey the learning content via text or audio explanation
Description of game features that enhance interactivity
Evaluating the gap between the objective and the actual performance
Promote the use of feedback which deals with the task completed, not with the self-esteem
Educators can debrief the performance of players at the end of the serious gaming session
Exploration of forms of feedback which are the most effective in serious gaming
Achieving long-term memory
Promote the repetition of interactions with important learning content inside the game
Promote spaced-education with multiple training sessions and various educational methods
Description of forgetting curves after different types of serious gaming and determination on how often refreshers should be proposed
Neuroscientists consider that attention consists of three attentional networks: the alerting network, the orienting network, and the executive network [13, 14]. These three networks are essential in the learning process.
Firstly, learners should stay alert during the learning process. The effects of alertness on memory seem to follow an inverted U-shaped relationship, with medium level of arousal associated with the highest gains in knowledge . In serious games, the level of alertness is the result of three components: the graphic and sound environment of the game, the challenge proposed, and the motivation of the player. Technological advancements on video games’ effects, including the progress in video game graphics and the increased structural complexity of sound effects, were shown to be associated with higher levels of arousal in action video games [16, 17]. The level of challenge proposed by the game was also shown to be a key feature of the engagement of the player, with positive effects of learning . This challenge should be in line with the skills of the learner because a high challenge with low skills may lead to frustration whereas a low challenge with high skills may lead to boredom . Finally, the motivation of the player is also essential to reach sustained vigilance while playing, because motivation and arousal are closely linked .
The second network of attention (orienting network) can be described as a filter which selects some information from the environment at the expense of other information. There is evidence that guided instruction, which provides learners with direct instructional guidance on the concepts and procedures required by a particular discipline, leads to superior learning outcomes than unguided or minimally guided environment, in which learners must discover or construct essential information for themselves . This highlights the importance for the instructor (or the educational tool) to help the learner to select the relevant information. A meta-analytic review of the role of instructional support in serious games confirmed that the use of instructional support was associated with better learning outcomes . Among different categories of instructional support, facilitating learners in selecting relevant information was the most effective solution. Three types of support related to the selection of relevant information were studied: modelling (showing which information is important or how to solve a problem), modality (the use of the audio channel for verbal explanations to guide visual search), and feedback (which allows players to know if the information is relevant or not). All these types of instructional support were associated with improved learning outcomes. A fourth way to guide the player to select the relevant information is to integrate this information in the virtual tools that the player needs to use to reach the objective of the game. For example, in the Re-Mission game, players needed to search for chemotherapy to load their weapons in order to destroy the cancer cells. Players were orientated continuously towards chemotherapy during the game and thus learnt implicitly that adherence to chemotherapy was the key to cure cancer . Developers should consider such strategies to help their end users to learn efficiently.
Executive control is the third attentional network. It allows one to concentrate on the task without being distracted. The ability to maintain sustained attention on a task without being distracted by the environment or his own thoughts is essential for learning. In serious games, it was originally thought that the immersion experienced during the game would prevent users from being distracted. Indeed, immersion is defined as “the sensation of being surrounded by a completely other reality […] that takes over all of our attention, our whole perceptual apparatus” . Jennet et al. explained that “immersion involves a lack of awareness of time, a loss of awareness of the real world, involvement and a sense of being in the task environment” . However, if immersion allows learners to stay focused on the game, emerging evidence suggests that high levels of immersion may allow them to master the game, but not to achieve the learning outcomes . An explanation may be that high levels of immersion are associated with episodic memory, corresponding to the memory of the context (times, places, associated emotions, etc.) whereas low levels of immersion allow an appropriate distance for the creation of declarative memory, which is the memory of facts and abstract concepts . Thus, when the learning objectives are more complex than simple retention and require a deep understanding of concepts, lower levels of immersion should be preferred. This implies that the environment in which a serious game is played should be clear of other sources of distraction, because the users who are learning abstract concepts during serious gaming sessions are not necessarily immersed and need to use their inhibitory control to stay focused on the task.
Active learning engages students in the process of learning through activities, as opposed to passively listening to an expert . Active learning has been shown to be more effective than traditional didactic lecture, for both knowledge acquisition and behavioural change [27–29]. The strength of serious games is that they encourage active learning because they are interactive by essence. A serious game cannot be completed without the intervention of its user, as opposed to a lecture which will continue whether or not the learner is listening to it. A meta-analysis of the learning effectiveness of simulation games found that the games which actively engaged trainees in learning were more effective than those which passively conveyed the instructional material via text or audio explanation . The game Staying Alive (http://www.stayingalive.fr/index_us.html) which was designed to teach the management of a cardiac arrest to the general public is a good illustration because it involves different levels of interactivity. In a first part, the explanations on the recognition of a cardiac arrest are limited to text boxes which require the player to click to the next one. By contrast, when the player has to provide virtual chest compressions, he is asked to choose his hand position between different proposals and to click repeatedly with the correct time interval to deliver compressions with the correct rate. In a study still ongoing in our simulation centre (ClinicalTrials.gov registry number 02758119), it appears that second year medical students who were pre-trained with the serious game and assessed on a physical manikin were more likely to provide chest compressions with a correct hand position and rate, whereas no progress was observed regarding the recognition of the cardiac arrest. Thus, future serious games should not be limited to a self-paced environment in which the player simply click to move to the next text box containing the relevant information but rather be fully interactive by integrating this information in actions that the player need to perform. In order to maximize the learning effectiveness of the serious game, these actions need to be similar to those performed in the “real world.”
According to the temporal difference learning theory, the brain can be seen as a machine that generates predictions and verifies them to improve its next prediction . Learning occurs by minimizing the difference between the expected and actual outcome, through the feedback received . Increasing evidence suggests that computations described by temporal difference learning theory are actually performed in the human brain [32, 33]. Feedback is therefore essential for learning, and it was believed that “the positive effects of feedback interventions on performance has become one of the most accepted principles in psychology” . However, a comprehensive meta-analysis of laboratory experiments suggested that while feedback improved performance by 0.4 of a standard deviation on average, it reduced performance in over one third of the experiments . In searching for the moderators, the authors found that feedback effectiveness decreased when the feedback dealt more with the learner than with the task. For example, feedback interventions in the form of a praise or a blame were less effective than feedback which focused on the task. This same meta-analysis found that computerized feedback interventions, which are likely to focus attention on the task, were associated with higher effectiveness compared to verbal feedback interventions.
In serious games, feedback can be integrated in different forms: progress bars, scoring, achievements, experience points, virtual currencies, etc. . The player can perceive the difference between the performance expected in the serious game and his actual performance through the feedback received. More research is needed to define the forms of feedback which are the most effective in serious gaming. In cognitive training games, there is emerging evidence that the different types of feedback are not equally effective. It was shown that the inclusion of real-time scoring during play negatively impacted training improvements of the participants . The authors suggested that real-time scoring distracted the learners from the core cognitive training task, which may also happen in serious games. Therefore, while feedback is essential for learning, the way it should be integrated in serious games deserves further investigation.
The memory consolidation hypothesis Müller and Pilzecker described 100 years ago continues to guide memory research . Processes that allow memory to consolidate have become increasingly well understood . The first stages of the acquisition of a skill, such as reading or driving, require the learner to stay very concentrated, because these processes are conscious. Repeated trainings allow the brain to shift to faster and unconscious networks . Neuroimaging revealed that during acquisition of a motor skill, there is a shift from prefrontal regions of the cortex to the premotor, posterior parietal, and cerebellar cortex structures . In the field of serious games, learners learn more when multiple training sessions are involved  to allow these shifts in neural network utilization to occur.
This repetition is even more effective if the learner alternates training sessions and rest periods. Reinforcement using “spaced training,” where training sessions alternate with rest periods, leads to longer lasting memories than massed training [42, 43] and better transfer to real situations . On the basis of these findings, “spaced education” in the medical field has been shown in randomized trials to improve knowledge acquisition, boost learning retention for up to 2 years, and durably improve clinical behaviour [45–47]. Spaced education can be integrated in serious games: an online spaced-education game to teach and assess medical students was shown to be effective in improving and assessing students’ knowledge .
Finally, this repetition is more effective if it involves different educational methods. A meta-analysis on the learning effectiveness of simulation games, a subgroup of serious games, found that they should be used as a supplement to lecture, discussion, tutorials, or other instructional methods in order to maximize their learning potential .
In summary, the development of serious games may benefit from scientists’ growing understanding of the human brain and of the mechanisms involved in learning. Then, the learning effectiveness of serious games should be evaluated rigorously through a scientific evaluation that we propose next.
Evaluating serious games: towards evidence-based education
In medicine, the best evidence of the effectiveness of an intervention is the result of randomized controlled trials (RCTs) and of systematic reviews of these studies. It should be the same in the educational field [49, 50]. However, serious games on the market are so diverse in their content, objectives, and users and that they preclude high-quality meta-analyses . Several authors have tried to propose a classification system for serious games to form more homogeneous groups, but a universal classification system has failed to be adopted [52–54], possibly because research in serious games involves authors with different backgrounds. Graafland and colleagues recently provided the first consensus-based framework for the assessment of specific medical serious games to address this problem . Features linked to game description, rationale, functionality, validity, and data protection must be reported . This framework allows educators and game designers to compare and validate new serious games in a consistent way.
Furthermore, the study designs of the articles evaluating serious games are very diverse. Less than 20% are RCTs . Choosing an appropriate control group and a relevant outcome are the main difficulties encountered in RCTs by researchers.
Ideally, RCTs evaluating serious games should include two control groups, one group receiving no education and the other receiving another form of education . The use of a control group who receives no education is essential to demonstrate the effectiveness of a serious game and still remains ethical because it happened that an intervention group playing a serious game failed to reach higher levels of knowledge and/or skills than a control group receiving no education . The use of a second control group receiving another form of education is also important. Previous research in simulation-based education showed that simulation technology was more effective than no education, but the difference was less convincing relative to other forms of education [59, 60]. In academic settings, serious games should be compared to another form of education to promote their use. By using these two control groups, it also becomes possible to evaluate whether a serious game can be a pragmatic solution between no education and a form of education that is more effective but not readily accessible to learners, such as long and expensive trainings in simulation centres.
The choice of outcome is also complicated in the serious game field. Four levels of outcome can be used to evaluate medical education, following the model of Kirkpatrick [61, 62]: satisfaction (level 1), acquisition of knowledge and skills (2b), retention of knowledge and skills over a period of time (2c), behavioural change (3), and patient outcomes (4).
All these outcomes are complementary and relevant for the evaluation of serious games but do not have the same value. Each step-up among these levels is associated with both (1) more convincing evidence on the value of the educational intervention and (2) increased difficulty to conduct the study and demonstrate a difference between the experimental and control groups. Although most studies assess users’ satisfaction, research on serious games should at least address the acquisition of knowledge and skills (level 2), because there is no evidence that satisfaction alone leads to effective learning. Knowledge can be assessed simply by questionnaires, while physical simulation using manikins represents a new, safe, and appropriate material to evaluate the acquisition of skills after playing a serious game.
Besides RCTs which are essential but limited to a binary answer (yes/no) about the effectiveness of a serious game, other study designs can offer valuable information on how to develop effective serious games. A new approach is represented by neuroscientific studies evaluating the impact of playing a serious game on specific neural circuits. For example, the Re-Mission videogame for cancer patients was shown to activate the reward-related mesolimbic neural circuits which are involved in the motivation to play . The authors demonstrated that this activation was primarily the consequence of the gameplay of the Re-Mission game which involved a lot of interactivity rather than the consequence of the effects of vivid and dynamic sensory stimulation. Such studies can improve our understanding of the mechanisms by which serious games can be motivating and enhance learning.
Serious games are attracting growing attention in the health area. Developing a successful serious game is complex, and we proposed cues to avoid two classical pitfalls. First, in order to obtain a serious game played by its end users, we recommended to find a point where the intrinsic and extrinsic motivation of the players can converge, i.e. where the players can enjoy a future desirable outcome made virtually present in the game. Second, developers should consider the four pillars of learning described to avoid the development of a game which does not fulfil its educational objectives. Researchers in the field should continue to explore how these four pillars of learning can be used to enhance serious game engagement and effectiveness. Finally, evaluation of serious games using a standardized framework will help to legitimize the enthusiasm observed in the health area for these tools.
Randomized controlled trial
David Drummond received a grant from Assistance Publique – Hôpitaux de Paris (AP-HP) during the first year of his PhD. AP-HP had no influence on the ideas developed in this review. No funding was secured for this study.
DD drafted the manuscript. AH and AT made substantial contributions. All authors approved the final manuscript.
The authors work in the department of Simulation in Healthcare “iLumens” which developed the serious game Staying Alive (stayingalive.fr) described in this review.
Consent for publication
Ethics approval and consent to participate
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- McCoy B. Digital distractions in the classroom: student classroom use of digital devices for non-class related purposes. J Media Edu. 2013;4(4):5e14.
- Murphy M. Why games work and the science of learning. Report [Internet]. 2012. Available http://www.goodgamesbydesign.com/Files/WhyGamesWork_TheScienceOfLearning_CMurphy_2011.pdf. [Cited 12 Jan 2017]
- Cook DA, Hamstra SJ, Brydges R, Zendejas B, Szostek JH, Wang AT, et al. Comparative effectiveness of instructional design features in simulation-based education: systematic review and meta-analysis. Med Teach. 2013;35:e867–98.View ArticlePubMedGoogle Scholar
- Greenwood CR, Horton BT, Utley CA. Academic engagement: current perspectives on research and practice. Sch Psychol Rev. 2002;31:328.Google Scholar
- Eichenbaum A, Bavelier D, Green CS. Video games: play that can do serious good. Am J Play. 2014;7:50.Google Scholar
- Prensky M. The motivation of gameplay: the real twenty-first century learning revolution. Horiz. 2002;10:5–11.View ArticleGoogle Scholar
- Klopfer E, Osterweil S, Salen K. Moving learning games forward. 2009.Google Scholar
- Ryan RM, Deci EL. Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp Educ Psychol. 2000;25:54–67.View ArticlePubMedGoogle Scholar
- Wouters P, Van Nimwegen C, Van Oostendorp H, Van Der Spek ED. A meta-analysis of the cognitive and motivational effects of serious games. J Educ Psychol. 2013;105:249.View ArticleGoogle Scholar
- Kato PM, Cole SW, Bradlyn AS, Pollock BH. A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics. 2008;122:e305–17.View ArticlePubMedGoogle Scholar
- Kron FW, Gjerde CL, Sen A, Fetters MD. Medical student attitudes toward video games and related new media technologies in medical education. BMC Med Educ. 2010;10:50.View ArticlePubMedPubMed CentralGoogle Scholar
- Did neuroscience find the secrets of learning? [Internet]. Available from: http://www.paristechreview.com/2013/11/07/neuroscience-secret-learning/. [cited 17 Jul 2015].
- Petersen SE, Posner MI. The attention system of the human brain: 20 years after. Annu Rev Neurosci. 2012;35:73–89.View ArticlePubMedPubMed CentralGoogle Scholar
- Posner MI, Rothbart MK, Tang Y-Y. Enhancing attention through training. Curr Opin Behav Sci. 2015;4:1–5.View ArticleGoogle Scholar
- Baldi E, Bucherelli C. The inverted “u-shaped” dose-effect relationships in learning and memory: modulation of arousal and consolidation. Nonlinearity Biol Toxicol Med. 2005;3:9–21.View ArticlePubMedPubMed CentralGoogle Scholar
- Ivory JD, Kalyanaraman S. The effects of technological advancement and violent content in video games on players’ feelings of presence, involvement, physiological arousal, and aggression. J Commun. 2007;57:532–55.View ArticleGoogle Scholar
- Grimshaw M. Sound and immersion in the first-person shooter. Int J Intell Games Simul. 2008;5:119–24.Google Scholar
- Hamari J, Shernoff DJ, Rowe E, Coller B, Asbell-Clarke J, Edwards T. Challenging games help students learn: an empirical study on engagement, flow and immersion in game-based learning. Comput Hum Behav. 2016;54:170–9.View ArticleGoogle Scholar
- Csikszentmihalyi M. Flow: the psychology of optimal experience. New York: Harper Perennial; 1991.Google Scholar
- Cowley B, Ravaja N. Learning in balance: using oscillatory EEG biomarkers of attention, motivation and vigilance to interpret game-based learning. Cogent Educ. 2014;1:962236.View ArticleGoogle Scholar
- Kirschner PA, Sweller J, Clark RE. Why minimal guidance during instruction does not work: an analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educ Psychol. 2006;41:75–86.View ArticleGoogle Scholar
- Wouters P, Van Oostendorp H. A meta-analytic review of the role of instructional support in game-based learning. Comput Educ. 2013;60:412–25.View ArticleGoogle Scholar
- Murray JH. Hamlet on the holodeck: the future of narrative in cyberspace. Simon and Schuster. New York: The Free Press; 1997.
- Jennett C, Cox AL, Cairns P, Dhoparee S, Epps A, Tijs T, et al. Measuring and defining the experience of immersion in games. Int J Hum Comput Stud. 2008;66:641–61.View ArticleGoogle Scholar
- Cheng M-T, Lin Y-W, She H-C, Kuo P-C. Is immersion of any value? Whether, and to what extent, game immersion experience during serious gaming affects science learning. Br J Educ Technol. 2016. doi:10.1111/bjet.12386.
- Cheng M-T, She H-C, Annetta LA. Game immersion experience: its hierarchical structure and impact on game-based science learning. J Comput Assist Learn. 2015;31:232–53.View ArticleGoogle Scholar
- Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci. 2014;111:8410–5.View ArticlePubMedPubMed CentralGoogle Scholar
- Davis D, O’Brien M, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282:867–74.View ArticlePubMedGoogle Scholar
- Prince M. Does active learning work? A review of the research. J Eng Educ Wash. 2004;93:223–32.View ArticleGoogle Scholar
- Sitzmann T. A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Pers Psychol. 2011;64:489–528.View ArticleGoogle Scholar
- Sutton RS. Learning to predict by the methods of temporal differences. Mach Learn. 1988;3:9–44.Google Scholar
- O’Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ. Temporal difference models and reward-related learning in the human brain. Neuron. 2003;38:329–37.View ArticlePubMedGoogle Scholar
- Schultz W, Dayan P, Montague PR. A neural substrate of prediction and reward. Science. 1997;275:1593–9.View ArticlePubMedGoogle Scholar
- Pritchard RD, Jones SD, Roth PL, Stuebing KK, Ekeberg SE. Effects of group feedback, goal setting, and incentives on organizational productivity. J Appl Psychol. 1988;73:337–58.View ArticleGoogle Scholar
- Kluger AN, DeNisi A. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull. 1996;119:254–84.View ArticleGoogle Scholar
- Lameras P. Essential features of serious games design in higher education. Learning. 2015;4:5.Google Scholar
- Katz B, Jaeggi S, Buschkuehl M, Stegman A, Shah P. Differential effect of motivational features on training improvements in school-based cognitive training. Front Hum Neurosci. 2014;8:242.View ArticlePubMedPubMed CentralGoogle Scholar
- McGaugh JL. The perseveration-consolidation hypothesis: Mueller and Pilzecker, 1900. Brain Res Bull. 1999;50:445–6.View ArticlePubMedGoogle Scholar
- McGaugh JL. Memory—a century of consolidation. Science. 2000;287:248–51.View ArticlePubMedGoogle Scholar
- Froyen DJW, Bonte ML, van Atteveldt N, Blomert L. The long road to automation: neurocognitive development of letter–speech sound processing. J Cogn Neurosci. 2008;21:567–80.View ArticleGoogle Scholar
- Shadmehr R, Holcomb HH. Neural correlates of motor memory consolidation. Science. 1997;277:821–5.View ArticlePubMedGoogle Scholar
- Vlach HA, Sandhofer CM, Kornell N. The spacing effect in children’s memory and category induction. Cognition. 2008;109:163–7.View ArticlePubMedGoogle Scholar
- Goverover Y, Hillary FG, Chiaravalloti N, Arango-Lasprilla JC, DeLuca J. A functional application of the spacing effect to improve learning and memory in persons with multiple sclerosis. J Clin Exp Neuropsychol. 2009;31:513–22.View ArticlePubMedGoogle Scholar
- Wang Z, Zhou R, Shah P. Spaced cognitive training promotes training transfer. Front Hum Neurosci. 2014;8:217.PubMedPubMed CentralGoogle Scholar
- Kerfoot BP. Learning benefits of on-line spaced education persist for 2 years. J Urol. 2009;181:2671–3.View ArticlePubMedGoogle Scholar
- Kerfoot BP, Armstrong EG, O’Sullivan PN. Interactive spaced-education to teach the physical examination: a randomized controlled trial. J Gen Intern Med. 2008;23:973–8.View ArticlePubMedPubMed CentralGoogle Scholar
- Kerfoot BP, Kearney MC, Connelly D, Ritchey ML. Interactive spaced education to assess and improve knowledge of clinical practice guidelines: a randomized controlled trial. Ann Surg. 2009;249:744–9.View ArticlePubMedGoogle Scholar
- Kerfoot BP, Baker H, Pangaro L, Agarwal K, Taffet G, Mechaber AJ, et al. An online spaced-education game to teach and assess medical students: a multi-institutional prospective trial. Acad Med J Assoc Am Med Coll. 2012;87:1443–9.View ArticleGoogle Scholar
- Greenhalgh T. How to read a paper: the basics of evidence-based medicine. Chichester: John Wiley & Sons. 2014.
- Harden RM, Grant J, Buckley G, Hart IR. BEME guide no. 1: best evidence medical education. Med Teach. 1999;21:553–62.View ArticlePubMedGoogle Scholar
- Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills training. Br J Surg. 2012;99:1322–30.View ArticlePubMedGoogle Scholar
- de Freitas S, Oliver M. How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Comput Educ. 2006;46:249–64.View ArticleGoogle Scholar
- Sawyer B. From cells to cell processors: the integration of health and video games. IEEE Comput Graph Appl. 2008;28:83–5.View ArticlePubMedGoogle Scholar
- Wattanasoontorn V, Boada I, García R, Sbert M. Serious games for health. Entertain Comput. 2013;4:231–47.View ArticleGoogle Scholar
- Graafland M, Dankbaar M, Mert A, Lagro J, De Wit-Zuurendonk L, Schuit S, et al. How to systematically assess serious games applied to health care. JMIR Serious Games [Internet]. 2014;2. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4307812/. [cited 23 Jul 2015].
- Primack BA, Carroll MV, McNamara M, Klem ML, King B, Rich M, et al. Role of video games in improving health-related outcomes: a systematic review. Am J Prev Med. 2012;42:630–8.View ArticlePubMedPubMed CentralGoogle Scholar
- Girard C, Ecalle J, Magnan A. Serious games as new educational tools: how effective are they? A meta-analysis of recent studies. J Comput Assist Learn. 2013;29:207–19.View ArticleGoogle Scholar
- Huss K, Winkelstein M, Nanda J, Naumann PL, Sloand ED, Huss RW. Computer game for inner-city children does not improve asthma outcomes. J Pediatr Health Care. 2003;17:72–8.View ArticlePubMedGoogle Scholar
- Mundell WC, Kennedy CC, Szostek JH, Cook DA. Simulation technology for resuscitation training: a systematic review and meta-analysis. Resuscitation. 2013;84:1174–83.View ArticlePubMedGoogle Scholar
- Ilgen JS, Sherbino J, Cook DA. Technology-enhanced simulation in emergency medicine: a systematic review and meta-analysis. Acad Emerg Med Off J Soc Acad Emerg Med. 2013;20:117–27.View ArticleGoogle Scholar
- Kirkpatrick DL. Evaluating training programs: the four levels. San Francisco; Emeryville: Berrett-Koehler; 1994.Google Scholar
- Mosley C, Dewhurst C, Molloy S, Shaw BN. What is the impact of structured resuscitation training on healthcare practitioners, their clients and the wider service? A BEME systematic review: BEME guide no. 20. Med Teach. 2012;34:e349–85.View ArticlePubMedGoogle Scholar
- Cole SW, Yoo DJ, Knutson B. Interactivity and reward-related neural activation during a serious videogame. PLoS ONE. 2012;7:e33909.View ArticlePubMedPubMed CentralGoogle Scholar