Deep learning pipeline to generate partial 3D structures of unconstrained image sequence

Structure from Motion (SfM) is a technique to recover a 3D scene or an object from a set of images. The images are collected from different angles of the object or the scene then the SfM software systems find matching 2D points between the images. The software triangulates the 3D position of the matched points. SfM is used in many applications such as virtual and augmented realities to enable virtual tours as well as scientific applications to scan and study various specimens. Close-range photogrammetry is a low-cost, simple method to attain high-quality 3D object reconstruction. However, software systems need a static scene or a controlled setting (usually a turntable setup with a blank backdrop), which can be a constraining component for scanning an object or a scene. Our research introduces a preprocessing pipeline based on deep learning to mitigate the turntable constraints. The pipeline uses detection and tracking techniques to isolate the different objects from the scene before feeding the imagery to a SfM software system. We assess multiple SfM software systems with and without the pipeline. The results show the pipeline line improves the 3D reconstruction quality and even recover the 3D structure of an object that cannot be reconstructed otherwise.