$5 Million in NSF Funding Boosts Development of Mixed Reality (XR) Platform for Workforce Training

What does the future of job training look like for manufacturing? With current innovations in augmented and virtual reality (AR/VR), it should look nothing like today’s inefficient and expensive processes based on paper pamphlets and in-person training from experts.

“The existing standard is the worst educational experience that I’ve ever seen,” says Kylie Peppler, a professor of informatics in the Donald Bren School of Information and Computer Sciences (ICS). Peppler is part of a multi-organizational team, led by Professor Karthik Ramani of Purdue University, that aims to greatly improve manufacturing workforce education with Skill-XR, a new training platform that leverages AR, VR and mixed reality (XR) applications.

Work on the platform started last year with $1 million in funding from the National Science Foundation (NSF) through its Convergence Accelerator program. “This program is about bringing together multiple fields to solve pressing problems that are facing the U.S. today,” says Peppler. “This is meant to transform industry to get new tools into their hands.” In 2019, the team’s Skill-LeARn platform was one of 43 projects to receive Phase I funding. Now, the team’s work is one of nine projects transitioning to Phase II. Over the next two years, they will receive an additional $5 million in funding to advance the current iteration of the project, “Skill-XR: An Affordable and Scalable X-Reality (XR) Platform for Skills Training and Analytics in Manufacturing Workforce Education.”

The Skill-XR Platform
The immediate goal is to provide more efficient training in manufacturing, leveraging advances is AR/VR technology so users can view graphics overlaid on machinery and receive tactile feedback about how to operate the controls. This training would be more effective, faster and safer than current practices.

The multidisciplinary team working on Skill-XR has expertise in manufacturing, computer vision and AI, spatial interfaces and interactions, learning sciences and learning analytics, cognitive psychology, and visual gamified interfaces.

The researchers at Purdue have partnered with several Indiana manufacturing-focused companies as well as with educational institutions to test prototypes both on the factory floor and with specific job training programs. Peppler, who is also a professor of education and serves as director of the Creativity Labs at UCI, is leading the learning and education portion of the project. Her team, which includes ICS postdoctoral researcher Joey Huang and informatics Ph.D. student Ariel Jiyeon Han, is focused on defining the project scope, analyzing the data and reviewing design decisions, and ensuring the team builds a marketable product.

“With the Skill-XR technology, it’s like we’re creating Lego building blocks,” explains Huang, who has a background in educational psychology and computing. “The customer has to actually put the puzzles together by themselves.” After conducting more than 90 interviews with education and workforce partners, the team is now determining what pieces must be included in the platform, what steps must be taken and how it all fits together to build the necessary knowledge for each skill set.

“What we’ve got the Purdue team thinking about right now is mapping the learning objectives for welding,” says Peppler. “We’re creating these experiences virtually, accelerating the learning process via just-in-time feedback that emulates one-to-one apprenticeship, providing employers with evidence of your abilities, and creating a way to scale reskilling and upskilling opportunities across the workforce.”

Furthermore, this isn’t only for on-the-job training. They hope such environments might help students explore different career options. “We’re thinking of the capabilities of Skill-XR as being like YouTube, where people can go to experience AR/VR training opportunities across a wide range of hands-on subjects, as well as author AR/VR content themselves to share with others,” says Peppler. “Imagine going to your local library and training for positions using Skill-XR, such as learning how to solder or weld.”

Building New Environments
The design of the Skill-XR platform is being informed by a variety of prototypes, including Meta-AR, an authoring platform for collaborative AR in STEM classrooms. Meta-AR guides students through the process of building a small city and lets them ask their peers for assistance throughout the process. Undergraduate students at UCI as well as local middle school students will test it this winter and spring.

“We’re looking at this curriculum and considering how it could also be applicable to our industry partners,” says Huang. Part of what they’re considering is not only the use of the virtual environments but also their creation, which was one of the main concerns that industry representatives and educators raised during the interviews.

“A lot of the technologies we’re looking at are scanning a 3D environment and then modeling it so you can pre-populate the build,” says Peppler. “We’re trying to get the end user to be somebody with a very limited skill set, so they can build a very robust educational experience just as they might put together a PowerPoint.”

The long-term goal is a low-cost reusable, plug-and-play approach that offers flexibility for a variety of educational and industrial use cases.

— Shani Murray