Overview of our VASK-DP framework. The system integrates tactile sensing with visual language models to enable gentle manipulation in long-horizon tasks. The framework consists of three main components: (1) atomic skill learning through reinforcement learning, (2) VLM-based task decomposition and skill selection, and (3) diffusion policy distillation for compliant manipulation.
Abstract
Autonomous execution of long-horizon, contact-rich manipulation tasks traditionally requires extensive real-world data and expert engineering, posing significant cost and scalability challenges. This paper proposes a novel framework integrating hierarchical semantic decomposition, reinforcement learning (RL), visual language models (VLMs), and knowledge distillation to overcome these limitations. Complex tasks are decomposed into atomic skills, with RL-trained policies for each primitive exclusively in simulation. Crucially, our RL formulation incorporates explicit force con- straints to prevent object damage during delicate interactions. VLMs perform high-level task decomposition and skill planning, generating diverse expert demonstrations. These are distilled into a unified policy via Visual-Tactile Diffusion Policy for end-to-end execution. We conduct comprehensive ablation studies exploring different VLM-based task planners to identify optimal demonstration generation pipelines, and systematically compare imitation learning algorithms for skill distillation. Extensive simulation experiments and physical deployment validate that our approach achieves policy learning for long-horizon manipulation without costly human demonstrations, while the VLM-guided atomic skill framework enables scalable generalization to diverse tasks.
Component pipeline
Atomic Skill Illustration.
Framework diagram of VASK.
Point cloud sequence during a long-horizon manipulation task.
Data generation
Object Stack Task
Cabinet and Place Task
Open Box and Place Task
Poru Ball Task
Poru Ball Pointcloud Task
BibTeX
@article{zhou2025gentlemanipulationpolicylearning,
title={Gentle Manipulation Policy Learning via Demonstrations from VLM Planned Atomic Skills},
author={Jiayu Zhou and Qiwei Wu and Jian Li and Zhe Chen and Xiaogang Xiong and Renjing Xu},
year={2025},
eprint={2511.05855},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2511.05855}
}