UNIVERSITY PARK, Pa. — A cross-disciplinary team of Penn State faculty members from the College of Arts and Architecture and the College of Education has created a ChatGPT-powered, virtual reality (VR) app that provides professionals in a wide range of fields an interactive training tool aimed at improving communication and pedagogical skills with a focus on empathy.
The VR app, Classroom Coach, offers professional training in which users interact with avatars in a simulated professional environments, like classrooms or emergency rooms. While the app is currently in early development, the goal is to provide users a real-time conversation with an avatar in helping professions such as teaching, nursing, social work and counselling.
The project received a financial boost earlier this year when the team was awarded a $48,000 grant from Penn State’s Institute for Computational and Data Sciences and a $2,000 grant from the School of Music. Led by Jacob Holster, a music education lecturer in the School of Music, the team includes Aaron Knochel, associate professor of art education in the School of Visual Arts, and Priya Sharma, associate professor of education in the College of Education.
To access the app, a user slides on a VR headset and is greeted by avatars whose responses and voice are driven by ChatGPT, a type of artificial intelligence (AI) known as a large language model. It trains on immense amounts of data to learn human speech patterns and content well enough to converse with a person in real time.
“We are hoping that professionals in any number of scenarios will be able to respond in more empathetic ways,” Holster said.
Early work on the app focused on teacher-student interaction in a classroom setting, and testing is being completed in Holster’s College of Arts and Architecture course “AA297G: Creating and Learning with AI.”
Training educators to navigate challenging scenarios, such as conflict between students and classroom disruptions, is invaluable, Holster said. But developing methods for more benign moments, such as simply asking for the classes’ attention to begin the day, are equally as valuable, he added.
“The app should help preservice teachers plan for lessons, execute lessons and reflect on the interpersonal effects of their instruction,” Holster said.
Sharma, whose research and teaching focus on developing and using emerging technologies for teaching and learning in formal and informal contexts, said that combining VR technology with AI is one of several reasons why the app is compelling.
“Supporting teachers and others in helping professions to engage with their particular populations with care and equity is a key appeal to the project,” Sharma said.
Although the app is in the early stages of development, the team said their hope is that it will be hosted on a widely accessible VR platform. The grant funding is key, Holster said, to helping the team not only gain access to a platform but to also deliver on the aspirational goals.
The funding the team earned earlier this year will be used to purchase hardware, such as VR headsets, and to pay for consulting hours with Penn State research organizations such as the Center for Immersive Experiences and Research Innovations with Scientists and Engineers, according to Holster.
“We are incredibly grateful for this support, which will help us over the next year to make the language model work in the way we want it to, create the VR space we want to see and connect those two components,” Holster said.
Knochel said that the team is involved in conceptualization, research, design and field testing. Although the project is in its early phase, he said Holster’s in-class testing results are promising.
“Preparing future teachers is a complex process of arranging opportunities for novice teachers to work within supportive environments to perform pedagogy,” Knochel said. “Success looks like a functioning prototype whereby we can create VR and AI simulations for use within preservice teacher education that reinforce strong pedagogical training supportive of teachers and their future students.”