Overview GitHub

Fisheye Images Rectification from Deep Straight Lines

Zhu-Cun Xue, Nan Xue, Gui-Song Xia,
CAPTAIN, Wuhan University, Wuhan, China

Brief overview

This paper presents a novel line-aware rectification network (LaRecNet) to address the problem of fisheye imagesrectification based on the classical observation -- the straight lines in 3D space should be still straight in image planes. Specifically, the proposed LaRecNet contains three sequential modules to (1) learn the distorted straight lines from fisheye images; (2) estimate thedistortion parameters from the learned heatmaps and the image appearance; (3) rectify the input images via a proposed differentiablerectification layer. To better train and evaluate the proposed model, we create a synthetic line-rich fisheye (SLF) dataset that containsdistortion parameters and well-annotated distorted straight lines of fisheye images. The proposed method enables us to simultaneously calibrate the geometric distortion parameters and rectify fisheye images. Extensive experiments demonstrate that our model achieves the state-of-the-art performance in both aspects of geometric correctness and image quality on several evaluation metrics. In particular, the images rectified by LaRecNet achieve an average reprojection error of 0.33 pixels on the SLF dataset, and report the highest peak signal-to-noise ratio (PSNR) and structure similarity index (SSIM) compared with the groundtruth.

  • the network of LaRecNet, and the code is coming soon.

    		•

    Dataset

    Thanks to the recently released wireframe dataset which has the labelings of straight line and the large-scale 3D scenes SUNCG which provides diverse semantic 3D scenes, we create a new synthetic line-rich fisheye (SLF) dataset based on the 2D wireframes and 3D surface models of man-made environ-ments for fisheye lens calibration and the image rectification. The proposed SLF dataset has well-annotated 2D/3D line segments as well as the corresponding ground truth distortion parameters for training and testing. The two subsets of SLF dataset, the distorted wireframe collection (D-Wireframe) from the wireframe dataset and the fisheye SUNCG collection (Fish-SUNCG) from the 3D model repository.

  • D-Wireframe ()
    • For any normal image without distortion, we give a group of distortion parameters to convert it into fisheye effect.
  • Fish-SUNCG ()
    • For any virtual 3d scene in Blender, we render fish eye images by controlling imaging formation mode under different camera poses.

    Campared with state of the arts
  • Results on Geometry Rectification

    • Distortion line rectification results of various methods. From left to right are the input RGB fisheye images, the distorted lines detected infisheye images, the rectified results by different method (Bukhari [1], AlemnFlores [2], Rong [3]), our proposed method, and the ground truth.

    Fisheye

    Distorted Lines

    Bukhari [1]

    AlemnFlores [2]

    Rong [3]

    Ours

    GT
  • Results on SLF Dataset

    • Qualitative comparison results of fisheye image rectification on D-Wireframe and Fish-SUNCG. From left to right are the input fisheye images, rectification results ofthree state-of-the-art methods (Bukhari [1], AlemnFlores [2], Rong [3]), our results as well as the ground truth.

    fisheye

    Bukhari [1]

    AlemnFlores [2]

    Rong [3]

    ours

    gt
  • Results on fisheye video dataset

    • To further justify our proposed method, we use the Fisheye Video dataset proposed in [5] for evaluation. To obtain the ground truth of the Fisheye Video dataset, we use the calibration toolbox to estimate internal and external parameters from the video of the calibration pattern in this dataset. From left to right are the input fisheye images, rectification results of three state-of-the-art methods (Bukhari [1], AlemnFlores [2], Rong [3]), our results as well as the ground truth images.

    More results
  • Results on

    • The following image sequences is combined by a series of images with with continuously varying distortion parameters. First , we generate a collection of fisheye images with linear distortion parameters, then the rectified images are generated by our proposed LaRecNet network, and finally all the images are combined frame by frame respectively to create the following image sequences.

  • Results on

    • The following fisheye videos is rendered by the designed camera's movement in rendering engine -- blender[4], and the corresponding videos is rectified by our proposed LaRecNet network.

  • Results on real videos

    • Qualitative rectification comparison results on real fisheye video dataset[5], and the we use the calibration toolbox to estimate camera parameters from the video of the calibration pattern in this dataset as the groundtruth parameters.

    References

    [1] F. Bukhari and M. N. Dailey, “Automatic radial distortion estimation from a single image,” J MATH IMAGING VIS, vol. 45, no. 1, pp. 31–45, 2013.
    [2] M. Alem´an-Flores, L. Alvarez, L. Gomez, and D. Santana-Cedr´es, “Automatic lens distortion correction using one-parameter division models,” IPOL, vol. 4, pp. 327–343, 2014.
    [3] J. Rong, S. Huang, Z. Shang, and X. Ying, “Radial lens distortion correction using convolutional neural networks trained with synthesized images,” in ACCV, 2016.
    [4] Online Community, “Blender - a 3d modelling and rendering package,” Blender Foundation, Blender Institute Amsterdam, 2014.
    [5] A. Eichenseer and A. Kaup, “A data set providing synthetic and real-world fisheye video sequences,” in ICASSP, 2016.