IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2021

Depth-Aware Mirror Segmentation

Haiyang Mei¹ Bo Dong² Wen Dong¹ Pieter Peers³ Xin Yang^1,*
Qiang Zhang¹ Xiaopeng Wei¹

¹ Dalian University of Technology ² SRI International ³ College of William & Mary

Contact us: xinyang@dlut.edu.cn mhy666@mail.dlut.edu.cn

Abstract

We present a novel mirror segmentation method that leverages depth estimates from ToF-based cameras as an additional cue to disambiguate challenging cases where the contrast or relation in RGB colors between the mirror re-flection and the surrounding scene is subtle. A key observation is that ToF depth estimates do not report the true depth of the mirror surface, but instead return the total length ofthe reflected light paths, thereby creating obvious depth dis-continuities at the mirror boundaries. To exploit depth information in mirror segmentation, we first construct a large-scale RGB-D mirror segmentation dataset, which we subse-quently employ to train a novel depth-aware mirror segmentation framework. Our mirror segmentation framework first locates the mirrors based on color and depth discontinuities and correlations. Next, our model further refines the mirror boundaries through contextual contrast taking into accountboth color and depth information. We extensively validate our depth-aware mirror segmentation method and demonstrate that our model outperforms state-of-the-art RGB and RGB-D based methods for mirror segmentation. Experimental results also show that depth is a powerful cue for mirror segmentation.

Downloads

Paper	: [ PDNet.pdf ]
Experimental results	: [ Google Drive ] [ Baidu Disk, fetch code: rrr3 ]
Pre-trained model	: [ Google Drive ] [ Baidu Disk, fetch code: rtf4 ]
Source code	: [ Code ]
Dataset	: [ Application Link ]

BibTex

@InProceedings{Mei_2021_CVPR_PDNet,
    author = {Mei, Haiyang and Dong, Bo and Dong, Wen and Peers, Pieter and Yang, Xin and Zhang, Qiang and Wei, Xiaopeng},
    title = {Depth-Aware Mirror Segmentation},
    booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2021}
}