UMass Amherst Professor Develops Interactive Computer Vision Apps to Study and Teach Dance Science
“When dance students see the apps I’ve created using MATLAB, they get interested in learning how to create apps themselves. The course has become a gateway into the many uses of MATLAB across different contexts in biology, computer science, and engineering.”
Challenge
Create an interactive digital tool that enables students to measure joint angle and dance movements
Solution
Use App Designer to create an easy-to-use interface for video processing algorithms and share them as a web-based app with students, teachers, and researchers
Results
- Initial app delivered in four weeks
- Broad user base reached
- New research ideas created
Scientific Foundations of Dance is a biological sciences general education course at the University of Massachusetts Amherst, in which undergraduate students analyze the biomechanics of dance motion. In the course’s lab component, students complete assignments in which they take measurements—for example, joint angles of a dancer’s arms and legs—to gain a deeper understanding of how the body moves.
To provide his students a virtual learning and lab environment, Dr. Aston K. McCullough, assistant professor of dance science at UMass Amherst, used MATLAB® to create a digital platform for teaching movement science. After developing the initial version in just four weeks, he used MATLAB Compiler™ and later used MATLAB Web App Server™ to share the platform’s kinematics analysis tools with students, other educators, and researchers.
“I wanted to create a new tool that students could use to turn their laptops into laboratories,” says Dr. McCullough. “App Designer and MATLAB Web App Server enabled me to create and easily share an app that dance and biology students alike can use to analyze movement in video, regardless of the computing platform they work on.”
Challenge
As the pandemic continued to force a suspension of most in-person classes at the start of the spring 2021 semester, Dr. McCullough had only weeks to build an interactive app that would serve as a virtual lab in which students could complete assignments remotely. He wanted to develop and share an interactive app that would enable students to analyze joint angles and motion based on recorded video data. In addition to being easy to use, the app needed to work across the spectrum of devices and computing platforms used by students and other educators.
Solution
Dr. McCullough started developing the app with App Designer, which he used to create the user interface. He used Image Processing Toolbox™ and Computer Vision Toolbox™ to develop algorithms that would enable students to measure joint angles and analyze motion from recorded video.
He then packaged the user interface and video processing algorithms together as standalone executables for Mac and Windows® using MATLAB Compiler. To accommodate students who needed access to the app from a Chromebook™ or similar device, Dr. McCullough built a version of the app in MATLAB Online™, which provides access to MATLAB from a standard web browser.
As Dr. McCullough continued to develop the app, he decided to deploy it as a web application using MATLAB Web App Server. This approach eliminated the need for students to install software or have a MATLAB Online license. Moreover, it enabled Dr. McCullough to continually add algorithms and enhancements to the app; students simply refresh the browser page to access the latest version of the software.
To complete lab assignments, students uploaded recorded images or videos of themselves performing different actions and used features in the app to analyze joint angles, distances, and other attributes of the recordings. They then pooled their data and performed analysis on it in MATLAB Online.
Dr. McCullough continues to enhance the app. He has recently added features to support using inertial measurement unit data and generating ground truth labels for deep learning and machine learning models.
As part of the university’s cooperative agreement with National Endowment for the Arts Research Labs, Dr. McCullough is currently developing signal processing and other app improvements that will be shared via MATLAB Web App Server with researchers in public health, neuroscience, and engineering.
The interactive MATLAB app enables UMass Amherst students to analyze joint angles and motion based on recorded video data.
Results
- Initial app delivered in four weeks. “I created the initial desktop versions of the app in MATLAB in just four weeks, finishing them just before classes started,” says Dr. McCullough. “During the semester, I created the web app version with MATLAB Web App Server over the course of two weeks.”
- Broader user base reached. “By using MATLAB Web App Server to share the apps I’ve developed, more people have access to valid and reliable tools for quantifying movement activity, in particular dance exposures,” says Dr. McCullough. “Not only students, but also neuroscientists, engineers, and public health researchers can use them to analyze kinematic data.”
- New research ideas created. “Conversations with the team at MathWorks generated more ideas for my research,” says Dr. McCullough. “What’s more, researchers at other institutions are reaching out about direct collaborations or to suggest ideas. By providing a virtual lab platform, the MATLAB Web App Server version of these apps has evolved into a wonderful opportunity for growth.”
