Virtual Agent Positioning Driven by Scene Semantics in Mixed Reality

Yining Lang1    Wei Liang1    Lap-Fai Yu2

1Beijing Institute of Technology    2George Mason University   



When a user interacts with a virtual agent via a mixed reality device, such as a Hololens or a Magic Leap headset, it is important to consider the semantics of the real-world scene in positioning the virtual agent, so that it interacts with the user and the objects in the real world naturally. Mixed reality aims to blend the virtual world with the real world seamlessly. In line with this goal, in this paper, we propose a novel approach to use scene semantics to guide the positioning of a virtual agent. Such considerations can avoid unnatural interaction experiences, e.g., interacting with a virtual human floating in the air. To obtain the semantics of a scene,we first reconstruct the 3D model of the scene by using the RGB-D cameras mounted on the mixed reality device (e.g., a Hololens). Then, we employ the Mask R-CNN object detector to detect objects relevant to the interactions within the scene context. To evaluate the positions and orientations for placing a virtual agent in the scene, we define a cost function based on the scene semantics, which comprises a visibility term and a spatial term. We then apply a Markov chain Monte Carlo optimization technique to search for an optimized solution for placing the virtual agent. We carried out user study experiments to evaluate the results generated by our approach. The results show that our approach achieved a higher user evaluation score than that of the alternative approaches.


Virtual Agent Positioning Driven by Scene Semantics in Mixed Reality
Yining Lang, Wei Liang, Lap-Fai Yu
IEEE Virtual Reality 2019
Paper , Video


    title= "Virtual Agent Positioning Driven by Scene Semantics in Mixed Reality",
    author = "Lang, Yining and Liang, Wei and Yu, Lap-Fai",
    booktitle = "IEEE Virtual Reality",
    year = "2019"


  • 媒体计算与智能系统实验室

  • Media Computing and Intelligent Systems Lab

Beijing Institute of Technology Copyright Address: 5 South Zhongguancun

Street, Haidian District, Beijing Postcode: 100081