EyeIR: Single Eye Image Inverse Rendering In the Wild

Abstract

We propose a method to decompose a single eye region image in the wild into albedo, shading, specular, normal and illumination. This inverse rendering problem is particularly challenging due to inherent ambiguities and complex properties of the natural eye region. To address this problem, first we con- struct a synthetic eye region dataset with rich diversity. Then we propose a synthetic to real adaptation framework to leverage the supervision signals from synthetic data to guide the direction of self-supervised learning. We design region-aware self-supervised losses based on image formation and eye region intrinsic properties, which can refine each predicted component by mutual learning and reduce the artifacts caused by ambiguities of natural eye images. Particularly, we address the demanding problem of specularity removal in the eye region. We show high-quality inverse rendering results of our method and demonstrate its use for a number of applications.

Our method can perform high-quality inverse rendering on a single input eye region image, producing albedo, shading, specularity, normal, which can be used for downstream applications such as eye-makeup, specularity removal and relighting.

EyeIR Framework

Given a single eye image, we aim to recover its ASNLC (albedo, diffuse shading, normal, lighting and specularity) components. We design EyeIR-Net to achieve this goal, which is composed of ASNL-Net and C-Net to predict corresponding components.

In Stage I, the EyeIR-Net is trained on SEIR by supervision of ground truths.
In order to obtain high-quality results on REIR, we propose S→R adaptation: in b) Stage II, the EyeIR-Net is trained simultaneously on SEIR and REIR, leveraging the supervised signals from SEIR to guide the self-supervised learning for REIR.
We facilitate S→R adaptation by proposing a novel RBDecoder.
Eye-specific losses c) are designed for the training. Particularly, to handle the eye-specific ambiguities analyzed in Sec. 1, quality-based losses consider different region's properties.
The effect of our EyeIR framework is shown in d).

Handling of Corneal Specularity Ambiguity

Corneal specularity removal is of vital importance for downstream applications like relighting. The strong corneal reflections usually lead to overexposure in the image, thus the specularity leakage problem in the predicted albedo is hard to address since the color information has already been lost. We develop a data-based method to perform corneal specularity removal for predicted albedo by training

By Specular Augmentation a), we can obtain eye images with augmented corneal specularity, the area of which is known as ground truth.
Then we train a SpecRem-Net b), taking the predicted albedo in Sec. 4 as input to output the albedo without leaked corneal specularity.
The effect of SpecRem-Net is shown in c).