Yao-Ting (Daniel) Huang

Reinforcement learning (RL) and sim-to-real transfer have advanced robotic manipulation of rigid objects. Yet, policies remain brittle when applied to articulated mechanisms due to contact-rich dynamics and under-modeled joint phenomena such as friction, stiction, backlash, and clearances. We address this challenge through dexterous in-hand manipulation of articulated tools using a robotic hand with reduced articulation and kinematic redundancy relative to the human hand. Our controller augments a simulation-trained base policy with a sensor-driven refinement learned from hardware demonstrations, conditioning on proprioception and target articulation states while fusing whole-hand tactile and force feedback with the policy’s internal action intent via cross-attention-based integration. This design enables online adaptation to instance-specific articulation properties, stabilizes contact interactions, regulates internal forces, and coordinates coupled-link motion under perturbations. We validate our approach across a diversity of real-world examples, including scissors, pliers, minimally invasive surgical tools, and staplers. We achieve robust transfer from simulation to hardware, improved disturbance resilience, and generalization to previously unseen articulated tools, thereby reducing reliance on precise physical modeling in contact-rich settings.

Humans rely on directional fingertip cues, particularly normal forces, to probe and regulate contact during manipulation, yet most wearable gloves render only vibration or single-axis force, leaving direction ambiguous. Without directional cues, users must infer contact force from vision alone, which leads to over-pressing, inconsistent control, and reduced precision in robotic teleoperation. We present the N2D Haptic Glove, a multi-finger wearable device that renders planar vector forces at each fingertip using a compact, transparency-focused mechanical design with on-hand actuation that preserves natural motion. Through benchtop validations and teleoperation user studies involving haptically teleoperating a humanoid robot, we demonstrate that multi-directional fingertip feedback significantly improves user ability to regulate contact, reduce overshoot, and improve consistency in interactions compared to visual-only or 1D haptic feedback. These findings establish the N2D Haptic Glove and directional finger-based haptics device as an important modality for contact-rich teleoperation, immersive simulation, and robot learning from demonstrations.