VLBenchmarks is a MATLAB framework for testing image feature detectors and descriptors. The latest version can be downloaded here.
If you use this project in your work, please cite it as:
@misc{lenc12vlbenchmarks, Author = {K. Lenc and V. Gulshan and A. Vedaldi}, Title = {VLBenchmkars}, Year = {2011}, Howpublished = {\url{http://www.vlfeat.org/benchmarks/}xsxs} }
This project is sponsored by the PASCAL Harvest programme as part of this project. VLBenchmkar is a sibling of the VLFeat Library, which it uses as suppot, but is otherwise independent of it.
The authors would like to thank Andrew Zisserman, Jiri Matas, Krystian Mikolajczyk, Tinne Tuytelaars, and Cordelia Schmid for helpful discussion and supports.
VLBechmarks is a MATLAB framework to evaluate feature detector and descriptors automatically. Benchmarking your own features is as simple as writing a single wrapper class. Then VLBenchmarks takes care of downloading the required benchmarking data from the Internet and running the evaluation(s). The framework ships with wrappers for a number of publicly available features to enable comparing to them easily. VLBenchmarks has a number of functionalities, such as caching of intermediate results, that allow running benchmarks efficiently.
The current version of VLBechmarks implements:
The code is distributed under the permissive MIT license.
The archive linked above contains the complete implementation of the VLBenchmarks suite, including a copy of this documentation. The minimal required version of MATLAB is R2010a (7.10) or later (due to extensive use of the new class type introduced in that release). To install simple unpack the archive, start MATLAB, change thea directory just created, and run:
This script downloads and installs a copy of VLFeat as the only dependency, and it compiles a number of MEX files. To successfully compile the code, you will need to be able to compile MEX files in your MATLAB environment, for detail see MATLAB documentation.
To test the benchmark installation run:
Note that this script will download and install the VGG Affine Dataset.
Here is a script for computing image feature extractor repeatability [1]. This script gets pair of images from the 'Graffiti' dataset, run the feature extractor and compute there repeatability:
With a result:
Running this script once again you will get the results immediately as
they are being loaded from cache. Now let's play with the descriptor
parameters. For example we can test what would happen if we compute features
of up-sampled images (by setting vl_sift
parameter 'FirstOctave'
to -1):
Few moments later we get the results:
Now let's run your custom feature extractor instead. Suppose you have
a command line utility called "DETECTORX" that implements it. Start by copying
a template wrapper located in 'localFeatures/TemplateWrapper.m'
and edit it according to your feature extractor:
Let's compute matching score of your feature extractor [1]. Because created feature extractor is not able to compute descriptors we will use SIFT descriptors instead.
Retrieval benchmark puts the feature extractor into a simple image retrieval system and calculates its mean Average Precision. Interface of the retrieval benchmark is similar to the repeatability benchmark. This minimal example shows how to calculate the Mean Average Precision of an image retrieval system which uses VLFeat SIFT feature extractor.