XFeat: Accelerated Features for Lightweight Image Matching

Guilherme Potje · Felipe Cadar · Andre Araujo · Renato Martins · Erickson R. Nascimento

[ArXiv] | [Project Page] | [CVPR'24 Paper]

• Training code is now available ->
• 🎉 New! XFeat + LighterGlue (smaller version of LightGlue) available! 🚀

Real-time XFeat demonstration (left) compared to SIFT (right) on a textureless scene. SIFT cannot handle fast camera movements, while XFeat provides robust matches under adverse conditions, while being faster than SIFT on CPU.

TL;DR: Really fast learned keypoint detector and descriptor. Supports sparse and semi-dense matching.

Just wanna quickly try on your images? Check this out:

Table of Contents

• Introduction
• Installation
• Usage
  • Inference
  • Training
  • Evaluation
• Real-time demo app
• XFeat+LightGlue
• Contribute
• Citation
• License
• Acknowledgements

Introduction

This repository contains the official implementation of the paper: XFeat: Accelerated Features for Lightweight Image Matching, to be presented at CVPR 2024.

Motivation. Why another keypoint detector and descriptor among dozens of existing ones? We noticed that the current trend in the literature focuses on accuracy but often neglects compute efficiency, especially when deploying these solutions in the real-world. For applications in mobile robotics and augmented reality, it is critical that models can run on hardware-constrained computers. To this end, XFeat was designed as an agnostic solution focusing on both accuracy and efficiency in an image matching pipeline.

Capabilities.

• Real-time sparse inference on CPU for VGA images (tested on laptop with an i5 CPU and vanilla pytorch);
• Simple architecture components which facilitates deployment on embedded devices (jetson, raspberry pi, custom AI chips, etc..);
• Supports both sparse and semi-dense matching of local features;
• Compact descriptors (64D);
• Performance comparable to known deep local features such as SuperPoint while being significantly faster and more lightweight. Also, XFeat exhibits much better robustness to viewpoint and illumination changes than classic local features as ORB and SIFT;
• Supports batched inference if you want ridiculously fast feature extraction. On VGA sparse setting, we achieved about 1,400 FPS using an RTX 4090.
• For single batch inference on GPU (VGA), one can easily achieve over 150 FPS while leaving lots of room on the GPU for other concurrent tasks.

Paper Abstract. We introduce a lightweight and accurate architecture for resource-efficient visual correspondence. Our method, dubbed XFeat (Accelerated Features), revisits fundamental design choices in convolutional neural networks for detecting, extracting, and matching local features. Our new model satisfies a critical need for fast and robust algorithms suitable to resource-limited devices. In particular, accurate image matching requires sufficiently large image resolutions -- for this reason, we keep the resolution as large as possible while limiting the number of channels in the network. Besides, our model is designed to offer the choice of matching at the sparse or semi-dense levels, each of which may be more suitable for different downstream applications, such as visual navigation and augmented reality. Our model is the first to offer semi-dense matching efficiently, leveraging a novel match refinement module that relies on coarse local descriptors. XFeat is versatile and hardware-independent, surpassing current deep learning-based local features in speed (up to 5x faster) with comparable or better accuracy, proven in pose estimation and visual localization. We showcase it running in real-time on an inexpensive laptop CPU without specialized hardware optimizations.

Overview of XFeat's achitecture. XFeat extracts a keypoint heatmap $\mathbf{K}$, a compact 64-D dense descriptor map $\mathbf{F}$, and a reliability heatmap $\mathbf{R}$. It achieves unparalleled speed via early downsampling and shallow convolutions, followed by deeper convolutions in later encoders for robustness. Contrary to typical methods, it separates keypoint detection into a distinct branch, using $1 \times 1$ convolutions on an $8 \times 8$ tensor-block-transformed image for fast processing, being one of the few current learned methods that decouples detection & description and can be processed independently.

Timing Analyses on CPU.

We show that both detection branch & match refinement module costs are small and bring significant advantages in accuracy (please check the ablation section in the paper).

Furthermore, XFeat performs effectively in both indoor and outdoor scenes, achieving an excellent compute-accuracy trade-off as demonstrated below. Note that in the paper, the teaser figure has a VGA resolution on the x-axis and 1,200 pixels on the y-axis. Below, we present an updated figure for improved clarity, maintaining the same x-y axis resolution.

Installation

XFeat has minimal dependencies, only relying on torch. Also, XFeat does not need a GPU for real-time sparse inference (vanilla pytorch w/o any special optimization), unless you run it on high-res images. If you want to run the real-time matching demo, you will also need OpenCV. We recommend using conda, but you can use any virtualenv of your choice. If you use conda, just create a new env with:

git clone https://github.com/verlab/accelerated_features.git
cd accelerated_features

#Create conda env
conda create -n xfeat python=3.8
conda activate xfeat

Then, install pytorch (>=1.10) and then the rest of depencencies in case you want to run the demos:

#CPU only, for GPU check in pytorch website the most suitable version to your gpu.
pip install torch==1.10.1+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html
# CPU only for MacOS
# pip install torch==1.10.1 -f https://download.pytorch.org/whl/cpu/torch_stable.html

#Install dependencies for the demo
pip install opencv-contrib-python tqdm

Usage

For your convenience, we provide ready to use notebooks for some examples.

Description	Notebook
Minimal example
Matching & registration example
Torch hub example
Training example (synthetic)
XFeat + LightGlue

Inference

To run XFeat on an image, three lines of code is enough:

from modules.xfeat import XFeat

xfeat = XFeat()

#Simple inference with batch sz = 1
output = xfeat.detectAndCompute(torch.randn(1,3,480,640), top_k = 4096)[0]

Or you can use this script in the root folder:

python3 minimal_example.py

If you already have pytorch, simply use torch hub if you like it:

import torch

xfeat = torch.hub.load('verlab/accelerated_features', 'XFeat', pretrained = True, top_k = 4096)

#Simple inference with batch sz = 1
output = xfeat.detectAndCompute(torch.randn(1,3,480,640), top_k = 4096)[0]

Training

To train XFeat as described in the paper, you will need MegaDepth & COCO_20k subset of COCO2017 dataset. You can obtain the full COCO2017 train data at https://cocodataset.org/. However, we make available a subset of COCO for convenience. We simply selected a subset of 20k images according to image resolution. Please check COCO terms of use before using the data.

To reproduce the training setup from the paper, please follow the steps:

1. Download COCO_20k containing a subset of COCO2017;
2. Download MegaDepth dataset. You can follow LoFTR instructions, we use the same standard as LoFTR. Then put the megadepth indices inside the MegaDepth root folder following the standard below:

{megadepth_root_path}/train_data/megadepth_indices #indices
{megadepth_root_path}/MegaDepth_v1 #images & depth maps & poses

3. Finally you can call training

python3 -m modules.training.train --training_type xfeat_default  --megadepth_root_path <path_to>/MegaDepth --synthetic_root_path <path_to>/coco_20k --ckpt_save_path /path/to/ckpts

Evaluation

---

MegaDepth-1500

Please note that due to the stochastic nature of RANSAC and major code refactoring, you may observe slightly different AuC results; however, they should be very close to those reported in the paper.

To evaluate on the MegaDepth dataset, you need to first get the dataset:

python3 -m modules.dataset.download --megadepth-1500 --download_dir </path/to/desired/folder>

Then, you call the mega1500 eval script, you can choose between xfeat, xfeat-star and alike. It should take about a minute to run the benchmark:

python3 -m modules.eval.megadepth1500 --dataset-dir </data/Mega1500> --matcher xfeat --ransac-thr 2.5

---

ScanNet-1500

To evaluate on the ScanNet eval dataset, you need to first get the dataset:

python3 -m modules.dataset.download --scannet-1500 --download_dir </path/to/desired/folder>

Then, you can call the scannet1500 eval script, it should take a couple of minutes:

python3 -m modules.eval.scannet1500 --scannet_path </data/ScanNet1500> --output </data/ScanNet1500/output> && python3 -m modules.eval.scannet1500 --scannet_path </data/ScanNet1500> --output </data/ScanNet1500/output> --show

---

Real-time Demo

To demonstrate the capabilities of XFeat, we provide a real-time matching demo with Homography registration. Currently, you can experiment with XFeat, ORB and SIFT. You will need a working webcam. To run the demo and show the possible input flags, please run:

python3 realtime_demo.py -h

Don't forget to press 's' to set a desired reference image. Notice that the demo only works correctly for planar scenes and rotation-only motion, because we're using a homography model.

If you want to run the demo with XFeat, please run:

python3 realtime_demo.py --method XFeat

Or test with SIFT or ORB:

python3 realtime_demo.py --method SIFT
python3 realtime_demo.py --method ORB

XFeat with LightGlue

We have trained a lighter version of LightGlue (LighterGlue). It has fewer parameters and is approximately three times faster than the original LightGlue. Special thanks to the developers of the GlueFactory library, which enabled us to train this version of LightGlue with XFeat. Below, we compare the original SP + LG using the GlueFactory evaluation script on MegaDepth-1500. Please follow the example to test on your own images:

Metrics (AUC @ 5 / 10 / 20)

Setup	Max Dimension	Keypoints	XFeat + LighterGlue	SuperPoint + LightGlue (Official)
Fast	640	1300	0.444 / 0.610 / 0.746	0.469 / 0.633 / 0.762
Accurate	1024	4096	0.564 / 0.710 / 0.819	0.591 / 0.738 / 0.841

Contributing

Contributions to XFeat are welcome! Currently, it would be nice to have an export script to efficient deployment engines such as TensorRT and ONNX. Also, it would be cool to train other lightweight learned matchers on top of XFeat local features.

Citation

If you find this code useful for your research, please cite the paper:

@INPROCEEDINGS{potje2024cvpr,
  author={Guilherme {Potje} and Felipe {Cadar} and Andre {Araujo} and Renato {Martins} and Erickson R. {Nascimento}},
  booktitle={2024 IEEE / CVF Computer Vision and Pattern Recognition (CVPR)}, 
  title={XFeat: Accelerated Features for Lightweight Image Matching}, 
  year={2024}}

License

Acknowledgements

• We thank the agencies CAPES, CNPq, and Google for funding different parts of this work.
• We thank the developers of Kornia for the kornia library!

VeRLab: Laboratory of Computer Vison and Robotics https://www.verlab.dcc.ufmg.br