Vision-Based Multimodal Teleoperation Interface for Firefighting Robots Using Hand Gestures and Eye Blinks

Robotic teleoperation is widely used in hazardous environments such as firefighting and disaster response, but conventional handheld controllers can be difficult to operate in high-risk situations. Vision-based interfaces provide a natural alternative, yet gesture-only systems often struggle to support both continuous motion control and discrete command triggering.

We propose a vision-based multimodal teleoperation interface using hand gestures and eye blinks. Left-hand finger joint angles and palm direction generate throttle commands, right-hand orientation generates steering commands, and blink patterns trigger auxiliary commands. The resulting signals are transmitted via UDP to a Unity-based firefighting robot simulator.

The system consists of perception, command generation, and robot control modules. A monocular webcam captures the user's hand gestures and facial expressions, and MediaPipe extracts hand and face landmarks in real time. Finger joint angles, palm orientation, and eye aspect ratio (EAR) are used to generate throttle, steering, and blink-triggered auxiliary commands.

Hand gesture recognition used 21 landmark coordinates from MediaPipe Hands. The left hand controlled throttle through finger openness and hand tilt; the right hand controlled steering through palm edge orientation and the palm normal vector. This geometric feature-based control enabled intuitive and smooth vehicle operation.

Eye blink events were detected using EAR from MediaPipe facial landmarks. A double short blink toggled water spray; a long blink triggered an emergency stop that zeroed throttle and steering and disabled spray output. Two consecutive short blinks released the emergency lock and restored normal control.

Experimental validation showed stable vehicle motion and water spraying with negligible latency. Users could intuitively control the vehicle without physical controllers, demonstrating the feasibility of contactless multimodal robot interaction using only a monocular webcam.