Senior Design Projects

ECS193 A/B Winter & Spring 2023

“Virtual” Inertial Measurement Units: Integrating OpenPose single-camera video wireframe positional detection into the Walk4Me gait evaluation tool

Email **********
Name Erik Henricson
Affiliation UCD Med Physical Medicine & Rehabilitation / CSGG

Project's details

Project title “Virtual” Inertial Measurement Units: Integrating OpenPose single-camera video wireframe positional detection into the Walk4Me gait evaluation tool
Background Background: There is a critical need to develop community-based methods for gait evaluation in the community setting for children and adults affected by muscle disease. Typical markerless camera/sensor-based gait evaluation tools are limited to lab or small field settings, and require complex and costly specialized equipment. We have developed a software suite and phone application that measures gait characteristics using the iPhone’s triaxial inertial measurement unit (IMU – accelerometers, gyroscopes, GPS). The Walk4Me system is capable of both live stream data collection during clinical evaluations and passive long-term data recording in the community. Signals are processed using a combination of machine learning and deep learning (ML/DL) techniques to identify steps and to estimate stride lengths and distance traveled with an accuracy within 5% of ground-truth observations.

Sensor signals are integrated with high definition video. At present, video is used by evaluators to identify and code participant gait events while they perform a variety of functional ambulatory tasks, including walking, running and positional transfers. Coded events serve as the basis for “supervised” machine learning categorization. Use of additional ML/DL techniques with IMU data allows the system to differentiate between typically-developing individuals and patients with muscular dystrophy-related gait disorders with up to 95% accuracy. However, performance of single sensor systems in the field is unlikely to equal or surpass that of multiple sensor or multi-camera systems.

OpenPose (Carnegie Mellon University) is an open-source real-time ML/DL-based wireframe human pose estimation tool that is capable of identifying anatomic landmarks (key points) from single-camera video.
Description Impact: There are multiple markerless multi-camera gait and positional analysis systems that are commonly used in the research setting, but none that facilitate large-scale community measurement efforts. Our proposed approach integrating single camera video-based positional analysis with data from a single IMU is unique. Successful integration would allow us to “anchor” our IMU data to an anatomic keypoint (lumbosacral junction) on video, thus providing directional/rotational data to the human wireframe representation. That integration should also facilitate use of other anatomic keypoints (eg. arm, leg, torso segments) as “virtual IMUs” to immitate multi-sensor/camera systems using just a single, consumer-level iPhone and camera toolkit. This will provide researchers with an inexpensive and scalable way to evaluate community ambulation in nearly any location, and will lead to development of research and clinical patient evaluation tools for use in clinical, field research and telehealth settings.
Deliverable Design Team Challenge: Using IMU and video data from >50 existing participants, the design team will:
1. Develop methods to anchor the OpenPose base of spine (Point 8) keypoint as the 3-D origin (0,0,0) coordinate around which hip and spine keypoints (Points 9, 12 and 1) travel, regardless of position in the video frame.
2. Link IMU data (accelerometer, gyroscope) to Point 8 to align axis of “3-D travel” of the IMU signal with the “spine” (Segment 1,8) and “hip” (Segment 9,12) segments.
3. Develop methods to simulate “virtual” IMU output representing 3-d travel of hip Points 9 and/or 12 based on IMU signals linked to Point 8. Once tools are developed, we will conduct additional studies to validate and refine these virtual IMU estimates, comparing them to actual IMU signals.

NOTE: This project may generate IP - students will share fair acknowledgement in IP, which will be managed on behalf of the research team by UC Davis Innovation Access per University policy.
Skill set desirable Ability to configure and run OpenPose (Carnegie Mellon U.); familiarity with accelerometer/gyroscope (IMU) signals and basic physics; basic familiarity with principles of human mobility / anatomy; curiosity and creativity!
Phone number **********
Client time availability 30-60 min weekly or more
IP requirement Client wishes to keep IP of the project
Attachment Click here
Selected No
Team members N/A
TA N/A