Deep Reverse Tone Mapping

Yuki Endo, Yoshihiro Kanamori, and Jun Mitani

University of Tsukuba



Abstract:

Inferring a high dynamic range (HDR) image from a single low dynamic range (LDR) input is an ill-posed problem where we must compensate lost data caused by under-/over-exposure and color quantization. To tackle this, we propose the first deep-learning-based approach for fully automatic inference using convolutional neural networks. Because a naive way of directly inferring a 32-bit HDR image from an 8-bit LDR image is intractable due to the difficulty of training, we take an indirect approach; the key idea of our method is to synthesize LDR images taken with different exposures (i.e., bracketed images) based on supervised learning, and then reconstruct an HDR image by merging them. By learning the relative changes of pixel values due to increased/decreased exposures using 3D deconvolutional networks, our method can reproduce not only natural tones without introducing visible noise but also the colors of saturated pixels. We demonstrate the effectiveness of our method by comparing our results not only with those of conventional methods but also with ground-truth HDR images.


Overview

We train 3D convolutional neural networks in order to infer bracketed LDR images from a single LDR image. The inferred LDR images are then merged to generate an HDR image.



Comparisons with Existing Methods

The left-most LDR images are expanded to HDR images using each rTMO and visualized using the consistent tone mapping operator [Kim and Kautz 2008]. To visualize the clipped (i.e., under/overexposed) pixels in the input LDR images, pixels of the images in the second column are painted in red/green if they are completely under/overexposed (i.e., black/white) or in magenta/cyan if one or two channels are under/overexposed, respectively.
[Nov. 8 2017] Note: We used the OpenCV implementations of HDR merging methods (i.e., [Mertens et al. 2007] and [Debevec and Malik 1997]) to generate the results in our paper, but we found that they sometimes yield low-quality HDR images. Using more sophisticated implementation (e.g., Photoshop) is recommended. We will update the results soon.
[Nov. 22 2017] Note: We have updated the results below.
Input images Clipped pixels [Akyüz et al. 2007] [Huo et al. 2014] [Kovaleski and Oliveira 2014] [Masia et al. 2015] Ours (MergeMertens) Ours (MergePhothoshop)
Input images Clipped pixels [Akyüz et al. 2007] [Huo et al. 2014] [Kovaleski and Oliveira 2014] [Masia et al. 2015] Ours (MergeMertens) Ours (MergePhothoshop)


Comparisons with Concurrent Work

The exposures of the input LDR images are reduced or increased, revealing loss of information in clipped image regions.
Input images [Eilertsen et al. 2017] Ours (MergePhotoshop)
Input images [Eilertsen et al. 2017] Ours (MergePhotoshop)
More results (coming soon)

Publications:

  1. Yuki Endo,Yoshihiro Kanamori, and Jun Mitani, "Deep Reverse Tone Mapping," ACM Transactions on Graphics (Proc. of SIGGRAPH ASIA 2017), 36, 6, Article 177 (November 2017), 10 pages. [Paper (90MB)] [Paper (low resolution, 1MB)] [Code (20MB)] [Trained models (1.2GB)]

BibTeX Citation

@Article{endoSA2017,
Title = {Deep Reverse Tone Mapping},
Author = {Yuki Endo and Yoshihiro Kanamori and Jun Mitani},
Journal = {ACM Transactions on Graphics (Proc. of SIGGRAPH ASIA 2017)},
volume = {36},
number = {6},
month = nov,
Year = {2017}
}