Measuring learning outcomes of serious gaming during the game

"After being elected OU Educational Product of the Year 2023, our own lecturers know about this tool. Now we want to realise more awareness among other educational institutions as well." Speaking is Hans Hummel, project leader of the Gamebrics project and theme leader in the field of serious gaming and simulations within the Open University. He knows: games are always about finding the right balance between having fun and achieving learning objectives. Two things are needed for an optimal balance: a good design approach to such a game and suitable tools to develop an effective game.

Hummel: "As the Open University, we had previously developed a design approach and development tool as part of a SURF project in the period 2006-2008. With this, more than 80 serious games were created, most of them scenario-based and focused on vocational preparation." 
Some of these are at an entry-level vocational level and train a single, simpler skill. "But we also use whole advanced games in our education. They allow you to train more complex skills within multiple scenarios at the same time," says Hummel. "The question is then always which skills a student develops the most during certain parts. We did not yet have a good assessment method to determine that properly during the game. That's how the idea of Gamebrics came about."

Scenario-based game for training skills

To illustrate this, he uses as an example Tycoonstructor, a serious game with the objective of training entrepreneurship skills. The player's game objective is to become the manager of a large international construction company. He has to deal with subsidiaries in different parts of Europe that face different challenges requiring different entrepreneurial skills. The manager-to-be thus goes through a career path that eventually turns him or her into a construction tycoon. Hummel: "The tasks in the game become more complex and you need more entrepreneurial skills at the same time. In the beginning, for instance, you come a long way with a good initiative for a new building project, but later you also need to show leadership qualities, master collaboration with several parties, and know how to motivate. You always ask yourself when designing a serious game: in which game activities does someone learn which skills and to what extent?"

Monitoring learning outcomes during the game

The Open University had previously used rubrics to indicate which skills are acquired per section. "But you could not yet zoom in on those specific components as a player," Hummel explains. "We wanted to give students feedback on their learning while playing the game, not only afterwards. We also wanted to monitor those learning results in a dashboard, so that both the student and the lecturer could see their progress during the game."

The OU also already used analytical rubrics to be able to measure learning outcomes after play, e.g. during an oral presentation on what had been learned through play. The Gamebrics project was therefore a logical follow-up to previous initiatives.

The team, led by Hummel, used the most recently developed serious games as Proofs of Concept for the new tooling. "We took a concrete and pragmatic approach to the project, very agile. We wanted to get a good and reusable idea of what it takes to integrate rubrics at the level of game components in a serious game," he says. You can read about the steps the Open University followed in this article.

Higher learning efficiency

The Open University investigated if the learning return is higher when students receive feedback on their progress towards the learning objectives. This was measured while playing two serious game. One game is used in the first year of learning and is relatively simple in terms of assignments. The other game is a bit more complex in terms of content. Hummel: "It turns out that for both games the learning gains indeed increase if you regularly show students how they are performing via a dashboard. They also feel more competent when they see that they actually learn something from the game. We also see that students who can see the dashboard go through the game activities more efficiently, they show higher 'performance' and make fewer mistakes, for instance. Perhaps some more specific feedback is still needed for games of higher complexity, as it is now sometimes perceived as too unspecific." The tool allows lecturers to get into that in the future.

Designing your own serious game

Hummel and his team are already focusing their energy on other projects, such as making digital didactics that you need when designing a good serious game more easily accessible. Enthusiastic, he says: "We are in the process of developing a design wizard that lecturers can use when they design their own serious game. Now it is the case that multidisciplinary teams are needed to develop a game. This is incredibly labour-intensive and a major reason why serious gaming is relatively little used. Universities of applied sciences are often more likely to resort to gamification, where you use game elements in your existing lessons to make them more attractive. Gamification may add to the fun, but the learning efficiency is often absent. Our design wizard should soon help lecturers develop a high-quality serious game that is both fun to play and contributes to the learning objectives." When developing that wizard, we will of course also use the Gamebrics approach.

How to use it?

A condition for eligibility for the Open and Online Education Incentive Scheme is that other universities of applied sciences can also use the innovation. Therefore, Gamebrics can be accessed via extensive documentation and tooling. The project website www.gamebrics.nl describes how to do this. There you will also find more information and a demo.