IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023

Deep Polarization Reconstruction with PDAVIS Events

Haiyang Mei1,2     Zuowen Wang2     Xin Yang1,*     Xiaopeng Wei1     Tobi Delbruck2    

1 Dalian University of Technology, Dalian, China    
2 Institute of Neuroinformatics, University of Zurich and ETH Zurich, Zurich, Switzerland    

  Contact us:    tobi@ini.uzh.ch    xinyang@dlut.edu.cn    haiyang.mei@outlook.com



1. Abstract

The polarization event camera PDAVIS is a novel bio-inspired neuromorphic vision sensor that reports both conventional polarization frames and asynchronous, continuously per-pixel polarization brightness changes (polarization events) with fast temporal resolution and large dynamic range. A deep neural network method (Polarization FireNet) was previously developed to reconstruct the polarization angle and degree from polarization events for bridging the gap between the polarization event camera and mainstream computer vision. However, Polarization FireNet applies a network pre-trained for normal event-based frame reconstruction independently on each of four channels of polarization events from four linear polarization angles, which ignores the correlations between channels and inevitably introduces content inconsistency between the four reconstructed frames, resulting in unsatisfactory polarization reconstruction performance. In this work, we strive to train an effective, yet efficient, DNN model that directly outputs polarization from the input raw polarization events. To this end, we constructed the first large-scale event-to-polarization dataset, which we subsequently employed to train our events-to-polarization network E2P. E2P extracts rich polarization patterns from input polarization events and enhances features through cross-modality context integration. We demonstrate that E2P outperforms Polarization FireNet by a significant margin with no additional computing cost. Experimental results also show that E2P produces more accurate measurement of polarization than the PDAVIS frames in challenging fast and high dynamic range scenes. Code and data are publicly available at: https://github.com/SensorsINI/e2p.

2. Downloads

Paper : [ E2P.pdf ]
Experimental results : [ Google Drive ] [ Baidu Disk, fetch code: e2pr ]
Pre-trained model : [ Google Drive ] [ Baidu Disk, fetch code: e2pm ]
E2PD Dataset : [ Google Drive ] [ Baidu Disk, fetch code: e2pd ]
Code : [ Github ]

3. BibTex

@InProceedings{Haiyang:E2P:2023,
    author = {Mei, Haiyang and Wang, Zuowen and Yang, Xin and Wei, Xiaopeng and Delbruck, Tobi.},
    title = {Deep Polarization Reconstruction with PDAVIS Events},
    booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2023}
}

4. Website Visit Statistics