Detailed Human Shape Estimation from a Single Image by Hierarchical Mesh Deformation

Abstract

    This paper presents a novel framework to recover detailed human body shapes from a single image. It is a challenging task due to factors such as variations in human shapes, body poses, and viewpoints. Prior methods typically attempt to recover the human body shape using a parametric based template that lacks the surface details. As such the resulting body shape appears to be without clothing. In this paper, we propose a novel learning based framework that combines the robustness of parametric model with the flexibility of free-form 3D deformation. We use the deep neural networks to refine the 3D shape in a Hierarchical Mesh Deformation (HMD) framework, utilizing the constraints from body joints, silhouettes, and per-pixel shading information. We are able to restore detailed human body shapes beyond skinned models. Experiments demonstrate that our method has outperformed previous state-of-the-art approaches, achieving better accuracy in terms of both 2D IoU number and 3D metric distance.

[Paper]（CVPR 2019 Oral）    [Code]

Method

    We design a hierarchical update structure, incorporating body joints, silhouettes, and photometric stereo to improve shape accuracy without losing the robustness. There are four stages in our framework: First, an initial SMPL mesh is estimated from the source image. Starting from this, the next three stages serve as refinement phases which predict the deformation of the mesh so as to produce a detailed human shape. Please refer to our paper for details.

Result

From green bounded frame, there are:

source image → initial guess → joint deform → anchor deform → vertex deform (final result)

Citation

@inproceedings{zhu2019detailed,
title={Detailed human shape estimation from a single image by hierarchical mesh deformation},
author={Zhu, Hao and Zuo, Xinxin and Wang, Sen and Cao, Xun and Yang, Ruigang},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2019}
}