Part of Evidence of Educative Game Design.
How large is the effect, really?
Education research measures the size of an effect with a single number, and almost nobody explains what the number means before using it. That is worth two minutes to fix, because it changes how every other figure in this series should be read.
What the number means
An effect size of about 0.2 counts as small. About 0.5 counts as moderate. 0.8 or above counts as large. These bands come from convention, not from a law of nature, but they are the convention this entire field uses, so they are worth knowing.
In plain terms: a large effect (0.8) means the average learner taught the new way did better than roughly four out of five learners taught the usual way. A moderate effect (0.5) means better than roughly two out of three. A small effect (0.2) means better than just over half, which still matters once it is applied across a whole cohort, but it is not the kind of difference you would notice by watching one classroom.
Where the numbers actually land
Simulation-based training sits at the top of what this field measures. Chernikova and colleagues combined 145 studies covering more than ten thousand learners and found an overall effect of 0.85, rising to 1.06 for technical performance specifically. Both are large by any convention.
Educational games in general sit much lower. Wouters and colleagues compared serious games against conventional instruction and found an effect of 0.29 on learning and 0.36 on retention. Clark and colleagues, examining digital games across school and university learners, found 0.33. All three are real and positive. None of them is large.
The distance between 0.33 and 0.85 is roughly a factor of three, and it is the question this whole series is organised around. It is not a gap between technologies. Simulations and educational games are both digital and both interactive. The gap is in the design, which is the subject of the next article.
A number belongs to the study it came from
Clark's 0.33 was measured on school and university learners. That detail is easy to drop and easy to misuse; a figure from that population does not automatically describe what happens with working adults in professional training, even though the underlying mechanism, that design quality drives the outcome, does carry across.
This matters enough that it is worth showing what happens when the detail is dropped by accident. Dunleavy and colleagues reviewed mobile learning for health professionals: 29 trials, 3,175 learners in total, a genuinely large review. Read quickly, it is tempting to attach that scale to every number the review reports.
Look closer and the review's two headline figures rest on very different ground. The knowledge effect, 0.43, comes from 11 of those trials. The skills effect, 1.12, the more impressive number, comes from only 5 trials covering 529 participants. The bigger number rests on the smaller study. Neither figure is wrong. Reporting the 1.12 beside the review's total N without saying so would be.
The rule worth keeping
Before trusting a figure, ask three things: what population was studied, how many studies fed the specific number being quoted, not just the review's total, and whether that number is the headline of the review or a subgroup buried inside it. All three questions have honest answers for every figure used across this series, and stating them costs a sentence. It is a cheap habit that catches an expensive mistake.
References
- Chernikova, O., Heitzmann, N., Stadler, M., Holzberger, D., Seidel, T., & Fischer, F. (2020). Simulation-based learning in higher education: A meta-analysis. Review of Educational Research, 90(4), 499–541.
- Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational Research, 86(1), 79–122.
- Dunleavy, G., Nikolaou, C. K., Nifakos, S., Atun, R., Law, G. C. Y., & Tudor Car, L. (2019). Mobile digital education for health professions. Journal of Medical Internet Research, 21(2), e12937.
- Wouters, P., van Nimwegen, C., van Oostendorp, H., & van der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249–265.