PHI-NOM: Physics-Based Hand Interactions for Natural Object Manipulation in Virtual Reality

Training for space missions or surgical procedures is inherently expensive, logistically complex, and resource-intensive, making immersive virtual reality (VR) an appealing alternative for rehearsing mission-critical tasks. However, current VR systems often rely on handheld controllers or gesture recognition, failing to simulate the physics necessary for equipment handling and assembly tasks. This paper introduces PHI-NOM (Physics-Based Hand Interactions for Natural Object Manipulation). This controller-free VR framework provides biomechanically accurate, real-time hand interactions by modeling fingertip contact and grasp physics. Built using motion capture and Unreal Engine, PHI-NOM employs a new algorithm for physics-based object manipulation, enabling natural actions such as grasping, lifting, rotating, and passing objects without the need for controllers, predefined gestures, or intermediary devices. Unlike existing methods that rely on computationally intensive inverse kinematics or physics solvers, PHI-NOM achieves low computational overhead through a lightweight model that provides visually accurate real-world physics. This design enables real-time interaction, supports diverse object geometries, allows for two-handed manipulation, and facilitates both individual and collaborative training scenarios, including cooperative tool use and system assembly. By enhancing tactile realism, PHI-NOM promotes motor learning and improves skill transfer from virtual to real-world environments.