Pixel-Perfect Structure-from-Motion

We introduce a framework that improves the accuracy of Structure-from-Motion (SfM) and visual localization by refining keypoints, camera poses, and 3D points using the direct alignment of deep features. It is presented in our paper:

• Pixel-Perfect Structure-from-Motion with Featuremetric Refinement
• Authors: Philipp Lindenberger*, Paul-Edouard Sarlin*, Viktor Larsson, and Marc Pollefeys
• Website: psarlin.com/pixsfm (videos, slides, poster)

Here we provide pixsfm, a Python package that can be readily used with COLMAP and our toolbox hloc. This makes it easy to refine an existing COLMAP model or reconstruct a new dataset with state-of-the-art image matching. Our framework also improves visual localization in challenging conditions.

The refinement is composed of 2 steps:

1. Keypoint adjustment: before SfM, jointly refine all 2D keypoints that are matched together.
2. Bundle adjustment: after SfM, refine 3D points and camera poses.

In each step, we optimize the consistency of dense deep features over multiple views by minimizing a featuremetric cost. These features are extracted beforehand from the images using a pre-trained CNN.

With pixsfm, you can:

• reconstruct and refine a scene using hloc, from scratch or with given camera poses
• localize and refine new query images using hloc
• run the keypoint or bundle adjustments on a COLMAP database or 3D model
• evaluate the refinement with new dense or sparse features on the ETH3D dataset

Our implementation scales to large scenes by carefully managing the memory and leveraging parallelism and SIMD vectorization when possible.