Researchers at Carnegie Mellon University’s Human-Computer Interaction Institute have developed a tool, EgoTouch, that uses AI to transform the user’s skin into an interface for controlling augmented reality (AR) and virtual reality (VR) environments. This innovation, described in a recent paper, employs a machine learning model to interpret touch interactions by analyzing visual cues such as shadows and skin deformations captured by standard cameras integrated into AR/VR headsets.
EgoTouch was inspired by earlier methods like OmniTouch, which required specialized, bulky equipment such as depth-sensing cameras. The new system eliminates this need by using conventional cameras in combination with machine learning algorithms. According to Vimal Mollyn, a Ph.D. student leading the project under the guidance of Professor Chris Harrison, the model identifies skin touches with over 96% accuracy, distinguishing between actions such as pressing, lifting, and dragging. Additionally, it can assess the force of a touch—whether light or firm—with 98% accuracy.
The team collected extensive data to train the model, including interactions from 15 users with varying skin tones and hair densities. Using a custom touch sensor placed on the underside of the index finger and palm, researchers recorded different touch types and forces under diverse lighting conditions. This dataset allowed the AI to associate visual features of touch with corresponding actions without manual annotation.
EgoTouch demonstrated consistent performance across different skin tones, hair densities, and areas of the hand and forearm, such as the palm and back of the arm. However, the system struggled on bony regions like knuckles due to insufficient skin deformation, which limits touch detection in those areas.
The tool opens possibilities for integrating common touchscreen gestures, such as scrolling, zooming, and swiping, into skin-based interfaces. It also supports functions like a “right-click” equivalent for enhanced control. The researchers are exploring additional advancements, including the use of night vision cameras for functionality in low-light conditions and adapting the technology for non-skin surfaces.
Mollyn emphasized that EgoTouch operates seamlessly with existing cameras in AR/VR headsets and requires no calibration, offering practical applications for developers and interface designers. The development marks a step forward in making on-skin interfaces viable and user-friendly in AR/VR systems.
Photo credit: Carnegie Mellon University
