DVS: Dynamic Virtual-Real Simulation Platform for Mobile Robotic Tasks

Zijie Zheng, Zeshun Li, Yunpeng Wang, Qinghongbing Xie, Long Zeng.
Tsinghua University
Paper arXiv Code
DVS Platform Overview

Overview of DVS Platform

DVS offers a variety of large-scale indoor scene types and dynamic element plugins on the left, enabling users to construct dynamic environments. In the middle, the platform supports various data types that can be generated, such as RGB, depth, and semantic labels. On the right, the data created using this platform can be applied to train robots for tasks such as navigation, trajectory prediction, and grasping. Through a virtual-real fusion feedback mechanism, the platform allows bidirectional mapping of the states of real and virtual agents, enriching the research scenarios.

Abstract

With the development of Embodied AI, robotic research has increasingly focused on complex tasks. Existing simulation platforms, however, are often limited to idealized environments, simple task scenarios and lack data interoperability. This restricts task decomposition and multi-task learning. Additionally, current Simulation Platforms face challenges in dynamic pedestrian modeling, scene editability, and synchronization between virtual and real assets. These limitations hinder real-world robot deployment and feedback. To address these challenges, we propose DVS (Dynamic Virtual-Real Simulation Platform), a platform for dynamic virtual-real synchronization in mobile robotic tasks. DVS integrates a random pedestrian behavior modeling plugin and large-scale, customizable indoor scenes for generating annotated training datasets. It features a optical motion capture system, synchronizing object poses and coordinates between virtual and real worlds to support dynamic task benchmarking. Experimental validation shows that DVS supports tasks such as pedestrian trajectory prediction, robot path planning, and robotic arm grasping, with potential for both simulation and real-world deployment. In this way, DVS represents more than just a versatile robotic platform; it paves the way for research in human intervention in robot execution tasks and real-time feedback algorithms in virtual-real fusion environments.

Video

Use Cases of DVS Platform

Explore different applications of our platform in robotic tasks

RGB Perception

The RGB perception from the first view of robot

Segment Perception

The Segment perception from the first view of robot

Social Navigation

Robot avoids pedestrians to navigate

VR Interation

Edit the virtual scen via VR headset

Grasping Data Collection Manipulation

Collect training data from DVS platform

Real Robot Deployed

Real2Sim2Real training and real time deployment and monitoring

These demonstrations showcase different applications of the DVS platform in various robotic tasks. The platform enables realistic simulation and training for robots in dynamic environments.

Experiment

Virtual-Real Fusion VLA Experiment

Virtual-Real Fusion VLA Experiment

Performance comparison of models trained on DVS data

This experiment demonstrates the effectiveness of our virtual-real fusion approach for visual language alignment. The models trained in our DVS platform show significant improvements in performance metrics when deployed in real-world scenarios.

Navigation Experiment

Navigation Experiment

Comparison of navigation performance in different scenarios

Our navigation experiments show how robots trained in the DVS platform can effectively navigate complex environments with dynamic obstacles. The results indicate superior performance compared to traditional simulation approaches.

Trajectory Prediction Experiment

Trajectory Prediction Experiment

Performance of trajectory prediction models in dynamic environments

The trajectory prediction experiments highlight the accuracy of models trained with our platform, showing how robots can anticipate and respond to human movements in shared spaces.

Our experiments validate the effectiveness of DVS in various robotic tasks, demonstrating improved performance in real-world deployments.

BibTeX