Informatics Capstone Team Develops ‘Amplify,’ Connects Kids With Speech Language Services

A team of UC Irvine (UCI) students has helped change the landscape of telepractice with Amplify – an audible adventure game system designed to deliver interactive speech therapy to children with cerebral palsy and other mental and emotional disabilities.

Informatics 191 Senior Capstone Course

The undergraduate team included Brian Cantwell, informatics; Michael Collins ’21, informatics; Sharifa Jesmin, informatics; Jennifer Kwon ’21, software engineering; and Eduardo Magdaleno ’21, software engineering. The students worked with sponsors Brian Cohn, Ph.D. and Christopher Laine, Ph.D. of Adventure BioFeedback, a software development company, to design and develop the pediatric speech therapy app for Children’s Hospital of Orange County (CHOC). Laine, an expert in oral-motor development, and Cohn, who specializes in health technology, created the Amplify prototype in 2018 and are research partners of CHOC.

Amplify is a game application for children in speech-language therapy designed to be fun, engaging, and accessible anywhere and anytime. The software platform creates a high level of interaction between a clinician and patient during a real-time session. It is, to date, the only voice-assistant device on the market for speech-language therapy.

Therapists set goals for their patients and, during a live virtual session, they listen and watch as the child practices sounds or words from home. If the therapist is satisfied with the progress, they allow the child to move on to the next level in the game.

The app was refined within the Informatics 191 Senior Capstone course taught by Hadar Ziv, Associate Professor of Teaching at the Donald Bren School of Information and Computer Sciences (ICS) at UCI.

Ziv has been teaching the Informatics 191 Senior Capstone course since its inception in 2009 with just 19 students. The three-quarter course coincided with the senior year of the very first UCI Informatics cohort that started in Fall 2005.

The course is now spread over two quarters and nurtures the talents of 150-200 students each year.

“Amplify was an outstanding project, and project team, and I would rank them in the top 5% of all time,” says Ziv, citing the students’ balance of hard and soft skills. The team maintained a high level of both technical abilities and technical productivity and consistency, along with exceptional communication and proficiency in project management.

“I have no doubt they will do well given the excellent skills they demonstrated – and hopefully improved upon – in their capstone project,” says Ziv.

Bridging The Gap for Vulnerable Populations

Telepractice, the use of telecommunications technology to remotely deliver speech language services, has been steadily on the rise for a number of years. Around 8% of American children, ages 3-7, are diagnosed with a communication disorder each year, and speech-language services are projected to grow by 25% over the next decade. However, with the onset of the COVID-19 pandemic, many health practitioners suddenly found themselves propelled into the spaces sooner than anticipated.

The speech and language pathologists at CHOC were not immune to this crisis, prompting researchers and clinicians at the CHOC Children’s Research Institute to expand telemedicine services and accelerate the development of digital tools when face-to-face visits became impossible.

“Children are a vulnerable population,” explain the mentors from CHOC Children’s Research Institute. Not only are children’s bodies different from adult bodies in numerous physiological ways, but also the mental stress felt from a pandemic can be harder for children to process.

Additionally, children with cerebral palsy (CP) suffer from a variety of movement and cognitive symptoms. CP broadly describes a group of non-progressive neurological injuries that occur when abnormal development or damage occurs to the fetus. The congenital disorder occurs in three out of every 1,000 live births and affects communication between the central nervous system (brain) and peripheral nervous system (muscles).

CP makes oral-motor control difficult for the patient and, thus, challenges their ability to effectively communicate with others. However, during early childhood, a critical window of opportunity opens for speech therapy to intervene. With the cancelation of in-person appointments due to COVID-19, a bridge was needed to reach this often marginalized – and time sensitive – patient population.

Developments like Amplify, which uses Google’s AIY Voice Kit voice-assisted technology, help CHOC reach more families and provide continuity of care, no matter the distance. “We recognize that health disparities pose a serious health threat to children,” says Afari. “Closing gaps in health and health outcomes is important to us.”

Developing Amplify

Just how many hours went into refining the Amplify app?

“Collectively? Hundreds,” says Michael Collins, who graduated in June and is now employed as a software engineer for Caroo in Los Angeles. Collins will begin his master’s degree in computer science at the Georgia Institute of Technology this fall.

The Amplify prototype’s goal was simple: gamify speech therapy in order to encourage and retain a child’s engagement at home. Achieving this, however, would not be as straightforward.

Over the course of two quarters, the students completed a monumental amount of work, which included developing exercises for established voice therapies, providing feedback on the at-home voice-assistant system, and helping to refine the overall Amplify system by analyzing voice data collection on the pitch, volume, and clarity of pronunciation.

“This project helped me to get experience with project and product management,” says Sharifa Jesmin, who is currently working as a software engineer intern at Northrop Grumman. The project has inspired Jesmin to look into a career as a product or project manager after graduation this fall.

Additionally, the team flexed their creative muscle writing stories for the interactive audible journey in the format of a choose-your-own-adventure story.

They progressed from initial designs, wireframes and mockups to application system diagrams, development, and, finally, user training and testing with CHOC speech language pathologists.

Luckily, the student team was well supported by their sponsors at Adventure Biofeedback. “Our sponsors [Laine and Cohn] were extremely helpful in defining and negotiating scope, allowing us to argue our case for different tools and implementations, and in providing additional resources to better understand the domain,” says Collins.

“In our weekly meetings with sponsors,” explains Jennifer Kwon, who also recently graduated and is currently working as a contract consultant and programmer, “we discussed ways of implementing use cases and overcoming any current challenges.”

The Results?

“These capstone students did a magnificent job. Perfection!” says Nadine Afari, a research programs manager at CHOC Children’s Research Institute.

Some of the final application features developed by the student team included a log in/log out screen, a navigation bar, activity templates, a level progress bar, and, finally, confetti. Yes, confetti.

“For gaming reinforcement, feedback and rewards,” explains Collins.

CHOC Research Institute plans to continue moving Amplify forward and will fund a small research pilot around the students’ work.

Future iterations of the app will be mobile friendly and available in Google Play and the Apple App Store. The simple interface is designed for ease of use by all ages and, best of all, Amplify’s library of interactive adventures continues to expand.

“Although we are no longer working on this project,” says Jesmin, “I want Adventure Biofeedback [i.e. Amplify] to evolve into a successful application that would be used by all the speech therapists in the United States, and beyond.”