Researchers at Cornell University have developed a new system that can track a person's entire body movement using nothing more than an ordinary smartwatch, according to a report from the Cornell Chronicle. The technology, called EchoMotion, relies on ultrasonic sound waves emitted by the watch's built-in speaker and picked up by its microphone, creating a sonar-like map of the wearer's limb positions.

The system was created by the Smart Computer Interfaces for Future Interactions (SciFi) Lab, led by Cheng Zhang, associate professor of information science at Cornell's Ann S. Bowers College of Computing and Information Science. Saif Mahmud, a doctoral student in information science and lead author of the study, described the work as "a very important jump for body tracking."

Current wearable devices typically use accelerometers and gyroscopes that can only detect the movement of the device itself, not the rest of the body. Full-body tracking has traditionally required more complex setups involving multiple devices such as smartphones, earbuds, or VR headsets, and often cameras. EchoMotion's use of sound waves avoids the privacy concerns of camera-based systems — users don't need to set up a camera at the gym — and generates smaller amounts of data that are cheaper and easier to process.

The researchers first built a custom 3D-printed watch replica with a microphone and speaker pair on each side. The two speakers emit ultrasonic signals that surround the wearer in a sphere of inaudible sound waves, and the microphones pick up the echoes that bounce off the person's limbs as they move. They collected echo patterns from 10 volunteers who wore the watch on their nondominant hand while performing 18 exercises, then used that data to train a deep learning model to translate the echo patterns into body movements. Accuracy was measured by comparing the predicted positions of 12 joints, from head to ankles, against body poses captured by cameras.

Commercial smartwatches have only a single microphone and speaker pair, so the team tested EchoMotion on an unmodified Samsung Galaxy Watch 6 with 15 new volunteers. For people using the system for the first time, the average error in predicting each joint's location was under four inches. A one-time, 12-minute calibration reduced that error by more than one third, to about 2.4 inches. The same AI model, without any retraining, also worked on a Samsung Galaxy Watch 7 and a Google Pixel Watch 3.

To test real-world conditions, the researchers asked 10 participants to wear a smartwatch while doing the 18 exercises in a small room, a narrow hallway, and next to a noisy road. Furniture and ambient noise did not interfere with the system, but performance suffered slightly in the hallway due to echoes bouncing off other people. Mahmud told the Cornell Chronicle he expects that with more data — from thousands of users instead of dozens — the model will improve at interpreting movements even in crowded environments like a subway.

Co-author François Guimbretière, professor of information science at Cornell Bowers, noted that while large language models have dominated headlines, "EchoMotion shows how very small models running on wearables have also revolutionized sensing in human-computer interaction." The system is also energy-efficient: sensing drew about 12 milliwatts, roughly 40 times less than a wearable camera, and a 30-minute session on a Galaxy Watch 6 used only 5% to 6% of the battery.

The team is currently developing the technology for fitness tracking, where a personal fitness agent could count a user's repetitions and even correct their form — for example, during a Bulgarian split squat. They are also exploring combining EchoMotion with first-person vision on smartglasses to track objects a person handles. Mahmud envisions a future where "with wearables like smartwatches or smartglasses, a personal agent will write a description of your whole day — what you did, what you ate. This project enables the body tracking part of that vision."

The research was presented at the Association for Computing Machinery's international joint conference on Pervasive and Ubiquitous Computing and International Symposium on Wearable Computing (UbiComp/ISWC) in Shanghai on October 13. Co-authors include Jiwan Kim of KAIST, Tianhong Catherine Yu (a doctoral student in information science), Kenny Liang '24, and Vineet Parikh '20, M.S. '24. The work was supported by the National Science Foundation.