MERAG

Multimodal emotion recognition for adaptive gaming based on a smartband end-to-end solution

The project aims to measure players' emotions from physiological signals measured by the Ovomind smart band and contextual information obtained from screen videos. This will be achieved by developing multimodal deep learning algorithms which will be integrated in a software development kit (SDK) dedicated to game developers.

In today’s world, wearable technology has become a rapidly growing field. However, the quality of signals collected by these biosensors remains a major challenge. Current wearable technologies put a strong emphasis on health and wellness although they could be used in other domains such as entertainment. This project focuses on the measurement of users' experience and emotions in the context of video games.

Although physiological emotion recognition devices achieve reasonable accuracy in the laboratory, it is less the case in an ecological setting. This can be explained by two factors: the amount of noise in traditional biosensor measurements and missing contextual information from the game events. Consequently, the project objective is to improve the quality of signals collected by the Ovomind biosensors and to determine video games’ contextual information automatically.

Multimodal deep learning methods will be developed to (i) identify and correct noisy physiological signals, and (ii) identify context from game videos. This approach will be validated by measuring the impact of both methods on emotion recognition performance. A sufficient quantity of data will be obtained by combining existing databases with data collected during the project from gamers playing their games at home.

This scientific partnership will create value by contributing to the advancement of wearable technology and by improving the reliability of biosensing Ovmonind smartbands. The project will produce new insights and breakthroughs that will drive innovation by providing an emotional software development kit for game developers, so they can create a new generation of video games which enhance players’ experience. The project will also have the potential to generate significant commercial benefits, such as increased revenue and market share, by enabling the development of new

This project is a collaboration between the SIMS lab of the inTECH institute at HEPIA, the computer science departement of the University of Geneva and the company Ovomind SA.