How large is a learning game’s effect, really?

An effect size is a percentage

An effect size translates into a percentage. An effect of 0.8 means the average learner taught the new way did better than about 79% of learners taught the usual way. An effect of 0.5 means better than about 69%. An effect of 0.2 means better than about 58%, a gain that shows up across a whole cohort.

Researchers label those three points large, moderate and small; the percentages are what the labels mean.

Where the gap comes from

The average effect of simulation-based training is roughly three times that of educational games, and the reviews behind the two figures differ in more than one way.

Chernikova and colleagues combined 145 studies covering more than ten thousand learners and found an overall effect of 0.85 for simulation, about 80% of learners coming out ahead, rising to 1.06 for technical performance. Wouters and colleagues, comparing serious games against conventional instruction, found 0.29 on learning and 0.36 on retention. Clark and colleagues, examining digital games across school and university learners, found 0.33. All three are positive, and all three put about 61 to 64% of learners ahead, in the small-to-moderate range.

The comparison is not like for like. Chernikova’s review covers higher education, mostly medical training, and includes non-digital simulations such as role plays and standardised patients; about half of its effects are measured against learners’ own pre-test scores rather than a separate group. Design is one plausible source of the gap, not the only one, and the next article sets out the design decisions the evidence does support.

Every number has a study

Clark’s 0.33 was measured on school and university learners, and that detail is easy to drop. A figure from that population describes that population, and adult studies describe adults. Whether a finding carries from one to the other is a separate question, with its own evidence.

What dropping it costs shows in a single review. Dunleavy and colleagues examined mobile learning for health professionals across 29 trials and 3,175 learners, a scale that invites every figure in the review to borrow it.

The two headline figures rest on different ground. The knowledge effect, 0.43, comes from 11 of those trials. The skills effect, 1.12, the more impressive of the two, comes from 5 trials covering 529 participants. The larger number rests on the smaller study. Both figures are right, and the care is in reporting each beside the trials that produced it.

Three questions for any figure

Which population was studied. How many studies stand behind that specific figure, rather than behind the review it sits in. Whether the figure is the review’s headline or a subgroup within it. Every figure in this series is reported with all three.