🤖 Tabero: Learning Gentle Manipulation with Closed-Loop Force Feedback from Vision, Touch, and Language

📄 Paper • 📦 Assets • 🖐️ Tactile Assets • 🤗 Model Weights

Tactile sensing is essential for robots to achieve human-like gentle manipulation capabilities. However, existing Vision-Language-Action (VLA) models struggle to exploit tactile feedback for gentle manipulation due to the scarcity of aligned vision-tactile-language data and the lack of effective closed-loop force feedback mechanisms.

Tabero addresses these challenges with:

• Tabero Benchmark — A data-efficient pipeline that repurposes open-source robot manipulation trajectories to generate diverse vision-tactile-language tasks in a high-fidelity tactile simulator (Isaac Lab + Taxim/FOTS), paired with a multidimensional evaluation protocol.
• Tabero-VTLA — A Vision-Tactile-Language-Action architecture featuring a decoupled force-position command interface executed by a fixed hybrid controller for real-time, force-aware manipulation.

🔬 Key Result: Our model maintains high task success while reducing average grip force by over 70% under gentle instructions, demonstrating its ability to modulate interaction forces based on multimodal experience.

Tabero-VTLA modulates grip force according to natural language instructions. Watch how the same task is executed differently under "gentle" vs "firm" language commands.

🟢 Gentle — "Gently pick up the cream cheese and place it in the basket." Significantly reduced grip force ✓

🟡 Failure Case Illustrating the gentleness-reliability trade-off

🔴 Firm — "Tightly pick up the milk and place it in the basket." Standard force level applied ✓

🟢 Gentle — "Softly pick up the milk and place it in the basket." Force reduced by ~70% while maintaining stable grasping ✓

• Isaac Sim 5.0+ with Isaac Lab 2.2+
• Python 3.10+
• CUDA 12.0+

# Clone the repository
git clone https://github.com/NathanWu7/TacManip.git
cd TacManip

# Install the Isaac Lab extension
python -m pip install -e source/tac_manip

# Install the OpenPI inference client
python -m pip install -e benchmarks/openpi/openpi-client

# LIBERO data (required for all workflows)
huggingface-cli download NathanWu7/Isaaclab_Libero \
  --repo-type dataset \
  --local-dir /path/to/Isaaclab_Libero

# Tactile calibration assets (for tactile environments)
huggingface-cli download china-sae-robotics/Tactile_Manipulation_Dataset \
  --repo-type dataset \
  --local-dir /path/to/Tactile_manipulation_dataset

LIBERO_DATA=/path/to/Isaaclab_Libero

ln -sfn "$LIBERO_DATA/assembled_hdf5" benchmarks/datasets/libero/assembled_hdf5
ln -sfn "$LIBERO_DATA/USD" benchmarks/datasets/libero/USD
ln -sfn "$LIBERO_DATA/replayed_demos" benchmarks/datasets/libero/replayed_demos
ln -sfn "$LIBERO_DATA/video_datasets" benchmarks/datasets/libero/video_datasets

# Tactile calibration assets
ln -sfn /path/to/Tactile_manipulation_dataset source/tac_manip/tac_manip/assets/data

# Download model weights
huggingface-cli download NathanWu7/pi0_lora_tacfield_tabero \
  --local-dir /path/to/pi0_lora_tacfield_tabero

# Start the model server (from the Tabero-VTLA repository)
# bash server.sh pi0_lora_tacfield_tabero 49999    #49999 denotes to the training step

# Then run evaluation:
python benchmarks/openpi/openpi_inference_client.py

# or batch evaluation:
python scripts/tools/run_task_evaluations.py

Model training code is maintained in the companion repository NathanWu7/Tabero-VTLA. This repository provides the Isaac Lab environments, data conversion tools, and inference client.

Paper results (Table 3) were obtained with Isaac Lab 2.2 + Isaac Sim 5.0, with the contact force sensor bound to the gelpad (the sensor contact surface).

In Isaac Lab 2.3 + Isaac Sim 5.1, binding the force sensor to the gelpad fails due to an unknown bug. A viable workaround is to bind the sensor to the minicase (the sensor housing) instead — this introduces some performance degradation but produces functional results (see the Minicase Rerun table below).

⚠️ If you encounter or solve this force sensor binding issue, please help by opening a GitHub Issue — contributions are greatly appreciated!

Detailed workflow documentation is organized under docs/:

If you find Tabero useful in your research, please cite:

@misc{wu2026taberolearninggentlemanipulation,
      title={Tabero: Learning Gentle Manipulation with Closed-Loop Force Feedback from Vision, Touch, and Language}, 
      author={Qiwei Wu and Rui Zhang and Xin Xiang and Tao Li and Weihua Zhang and Junjie Lai and Renjing Xu},
      year={2026},
      eprint={2605.27886},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2605.27886}, 
}

This project is released under the Apache License 2.0.