Background
In disaster response, underground exploration, smart manufacturing, laboratory automation, and service robot applications, a single robot is often insufficient for observation, mobility, manipulation, and cooperative tasks. A multi-robot team can distribute work according to platform capability, where mobile robots handle navigation and sensing, robot arms perform fine manipulation, and dual-arm systems execute synchronized or handover actions. However, practical deployment depends not only on individual robot control, but also on stable communication, clear data separation, and operator awareness of the whole team. Conventional teleoperation based on multiple 2D image windows increases spatial reasoning burden, especially when heterogeneous robots operate simultaneously. Therefore, this project establishes an operation framework based on intuitive VR control, digital twin feedback, and decentralized wireless communication to support coordinated work among a robot team.
Research Objectives
The objective of this project is to establish a scalable VR-based multi-robot team operation and communication system. The system allows an operator to understand, switch between, and control different robot platforms within one immersive interface. It supports task allocation, state feedback, and image data exchange among robots, while maintaining stable communication over Wi-Fi and preserving each platform’s independent execution and computation through a decentralized architecture. The project has three goals: first, to develop an intuitive VR control process for mobile control, viewpoint control, and robot arm end-effector control; second, to provide digital twin feedback so that the operator can observe relative positions, poses, and task progress of the robot team in 3D; and third, to establish a scalable communication and data separation method, allowing future robots to be added more easily.
Methods
The technical method focuses on functional integration, with the technical approach organized into three layers: control, feedback, and communication. The control layer converts the operator’s hand movements in VR into executable control commands for different robots, allowing mobile platforms to perform base and viewpoint control, robotic arms to perform pose control, and dual-arm robots to execute synchronized operations. The feedback layer integrates each robot’s visual and pose information into a digital twin, enabling the operator to understand the working status of the robot team within a consistent three-dimensional scene, rather than relying only on multiple scattered video windows. The communication layer adopts a decentralized design, in which each robot retains its own computation and control process while exchanging necessary data. The system also uses clear data naming and isolation mechanisms to prevent data from different platforms from being confused.
Innovation
The innovation of this project lies in integrating intuitive VR-based control, digital twins, and decentralized multi-robot communication into an operational framework designed for robot-team tasks. Unlike conventional approaches that treat different robots as independent devices to be controlled separately, this system emphasizes task relationships, data-flow communication, and complementary platform capabilities among robots, allowing the operator to monitor and understand the entire robot team from a unified spatial perspective. The system supports not only remote operation of a single robot, but also cross-platform switching, coordination between mobility and manipulation tasks, and simultaneous transmission of sensing data from multiple robots.
Expected Outcomes
This project is expected to complete a VR-based multi-robot team control and wireless communication demonstration system, enabling the operator to monitor the working status of multiple robots within the same immersive interface and to perform control switching and task operations in an intuitive manner. The expected outcomes include a decentralized wireless communication architecture, a VR control workflow that supports both mobile robots and robotic arms, and a digital twin interface capable of presenting the status of the robot team. In terms of benefits, this system can reduce the operator's spatial understanding burden in multi-robot tasks and improve the collaboration efficiency of the robot team. In addition, through load testing of compressed RGB-D image streams from multiple robots, this project can provide a basis for communication planning when more robots are added in the future, establishing a scalable and repeatedly deployable foundation for multi-robot team operation.