Background
Intelligent team-based robotic systems require multimodal interfaces that enable human operators to understand robot states and task environments. VR digital twins can enhance spatial understanding and immersive interaction; however, their presentation quality may be affected by sensing noise, compression artifacts, depth errors, semantic labeling errors, incomplete reconstruction, and latency. These imperfections may lead to incorrect judgments or over-trust in the system. Therefore, it is essential to develop quality assessment and trustworthy interaction models centered on human perception and task performance.
Research Objectives
First, to establish robot-perceived VR digital twin data and quality degradation models. Second, to quantify the effects of different quality conditions on human situational awareness, trust, cognitive workload, and task decision-making. Third, to develop multimodal models for predicting VR digital twin quality and trustworthiness. Fourth, to explore quality-aware interaction optimization and evaluation methods for generative AI-assisted explanations.
Methods
This project will use simulated or semi-synthetic scenes to generate robot first-person views, depth maps, semantic maps, and robot state information, and will construct a VR digital twin interface accordingly. Quality degradations such as compression, blur, low resolution, depth noise, point-cloud sparsification, semantic errors, and latency will be introduced. User studies will then be conducted to collect subjective quality scores, trust ratings, cognitive workload, task accuracy, and response time. Based on these data, multimodal deep learning models will be developed to predict digital twin quality, trustworthiness, and human decision reliability.
Innovation
The innovation of this project lies in integrating image/video quality assessment, VR digital twins, multimodal learning, and human-robot collaboration. The project evaluates the usability of robot-perceived scenes from the perspective of human perception and trust. Unlike conventional digital twin studies that focus primarily on system construction or control, this project emphasizes whether the digital twin interface can be correctly understood, trusted, and used for decision-making by human operators. It further proposes task-aware models for predicting quality and decision reliability.
Expected Outcomes
This project is expected to produce a VR digital twin quality assessment dataset, user subjective evaluation data, multimodal quality prediction models, and a prototype of a quality-aware interaction interface. The outcomes can support human-robot collaboration scenarios such as teleoperation, smart manufacturing, inspection, warehouse logistics, healthcare, and disaster response. The project is expected to improve operator situational awareness, decision accuracy, system trust, and deployment safety, while strengthening the center’s capabilities in human-robot collaboration and multimodal digital twin technologies.