UNIVERSITY PARK, Pa — In 2020, more than four-in-10 people in the United States had obesity, an increase from three-in-10 people in the year 2000, according to the Centers for Disease Control and Prevention. To better understand eating behaviors and food choices that may contribute to obesity, researchers investigate the conditions and environments people encounter while eating in the modern world — like smartphones, advertisements, and the company of others. Now, researchers in the Penn State College of Health and Human Development found that mixed reality, or the inclusion of real-world video footage into a virtual environment or vice versa, can simplify and improve the way this research is conducted.

With the support of the Center for Immersive Experiences, Travis Masterson, Broadhurst Career Development Professor for the Study of Health Promotion and Disease Prevention at Penn State and assistant professor of nutritional sciences, and his graduate student, John Long, evaluated the use of mixed reality in eating behavior and food choice studies.

They published their findings in Frontiers in Nutrition.  

Laboratories, restaurants, and virtual restaurants

Currently, eating behavior is most often studied in laboratories. Laboratories, however, are often sterile environments, unlike a restaurant full of people, sounds, sights, and smells.

Since not every laboratory can be turned into a realistic replica of an Italian restaurant or sushi bar, Masterson and Long said that mixed-reality can help researchers better emulate the real-world environments influencing eating behaviors, resulting in more accurate data. With the help of virtual-reality (VR) headsets and the ever-improving software behind them, scientists can simulate a specific environment instead of physically building one.

“Putting participants in these types of environments saves time and money,” Masterson said.  “Being able to conduct research in a way that promotes sustainability and cost effectiveness is key.”

Masterson explained that this technology could also allow researchers to engage with communities who are often omitted in research.

“The mixed-reality headsets are mobile, so we aren’t confined to a laboratory or participants who have the resources to travel to one," he said. "We could go out to a farmers' market in an under-represented town to conduct these studies, for example."

The promise of mixed reality

In VR, every aspect of the world that the user sees is computer-generated, even parts of their own body that enter their field of vision. Long explained that newer headsets can track a person’s hands and then render a computer-generated hand in the virtual world. This is excellent for sustaining immersion in the virtual world, but it is not so helpful when real-life objects are a main component of the research, Long said.

Small video cameras that are a component of the VR headsets, called pass-through cameras, allow for the inclusion of real objects in the virtual environment, creating a window to the real world. This creates the mixed-reality experience.

“The pass-through cameras scan for the user’s hand and place a digital disk around it, which allows them to see their real hand within the virtual display,” Long said. “They can also see the real food when their hand is near it. This way, the participants can easily interact with the food while still feeling immersed in the virtual environment.”

The researchers explained that attempting to eat real food while seeing a computer-rendered hand, fork and meal is awkward for diners, even when everything is in the proper location. Participants in the experiment indicated that the pass-through cameras made the process of eating while wearing a headset much smoother because they saw their real hand and the real food they were consuming within the virtual setting. Long explained that these cameras help to obtain results of eating behavior that mimic observations in the real world.

“If any aspect of the process was unnatural for the participants, the results of the study could be skewed,” Long said. “The pass-through cameras make the experience as realistic as possible while securing results that more accurately reflect a real-world experience than if they were taken in a sterile laboratory without the virtual environment.”

Testing the mixed-reality restaurant

For the study, the researchers recruited eight experts in eating behavior and sensory science to test a mixed-reality restaurant environment they had developed. The researchers refined the mixed-reality environment based on the participants’ feedback, and then the participants evaluated the second version of the mixed-reality restaurant. During both sessions, participants ate a basic meal in Masterson's laboratory while wearing a VR headset.

When creating the initial virtual environment, the researchers started with a minimalistic depiction of a restaurant with some ambient sounds and, most importantly, real food for the participants to interact with. After receiving feedback from the participants, the team improved the virtual restaurant to avoid any distractions that would limit the ability to become immersed in the experience.

The team improved the scene’s lighting and added virtual diners, more background noise, and restaurant-specific decorations. This way, the virtual restaurant felt less sterile and more like a real, bustling location. At the same time, the improved scene included many of the factors that typically affect a person’s eating behavior in the real world.

“People can’t work with an isolated or strange environment that doesn’t feel authentic,” Masterson said. “Seeing social interaction and even the environment outside of the restaurant makes the scene and the participants' interactions within it realistic.”

The future of eating-behavior research? 

Through this study and the participant surveys taken during the process, Masterson and Long confirmed that mixed-reality technology can elevate researchers’ ability to assess eating behavior without doing costly research in an actual restaurant. Understanding the aspects of a setting that can influence a person’s eating decisions is fundamental to creating a change in the population’s health, the researchers said.

“What our lab has done is confirm the validity of a variety of tools that researchers can utilize based on the question they have,” Masterson said. “Studying eating behavior is important in today’s world, and mixed reality can make that a much easier and more sustainable process.” 

Bart Masters and Pejman Sajjadi of the Center for Immersive Experiences at Penn State, and Christopher Simons, associate professor of food science and technology at Ohio State University, also contributed to this research.