Visual place recognition (VPR) enables autonomous robots to identify previously visited locations, which contributes to tasks like simultaneous localization and mapping (SLAM). VPR faces challenges such as accurate image neighbor retrieval and appearance change in scenery.
Event cameras, also known as dynamic vision sensors, are a new sensor modality for VPR and offer a promising solution to the challenges with their unique attributes: high temporal resolution (1MHz clock), ultra-low latency (in μs), and high dynamic range (>120dB). These attributes make event cameras less susceptible to motion blur and more robust in variable lighting conditions, making them suitable for addressing VPR challenges. However, the scarcity of event-based VPR datasets, partly due to the novelty and cost of event cameras, hampers their adoption.
To fill this data gap, our paper introduces the NYC-Event-VPR dataset to the robotics and computer vision communities, featuring the Prophesee IMX636 HD event sensor (1280x720 resolution), combined with RGB camera and GPS module. It encompasses over 13 hours of geotagged event data, spanning 260+ kilometers across New York City, covering diverse lighting and weather conditions, day/night scenarios, and multiple visits to various locations.
Furthermore, our paper employs the VPR-Bench framework to conduct generalization performance assessments, promoting innovation in event-based VPR and its integration into robotics applications.
| Duration (hr) | Data size (GB) | Modality | Distance (km) | Weather | Lighting conditions | Resolution (px) |
|---|---|---|---|---|---|---|
| 13.5 | 466.7 | event, RGB, GPS | 259.95 | rainy, cloudy, sunny | day, night | 1280 x 720 |
NYC-Event-VPR dataset statistics.
NYC-Event-VPR covers New York City, focusing on Chinatown area in Manhattan with overlapping traversal.
Sensor setup and mounting design: RGB camera is mounted on top of event camera, and the sensor suite is positioned facing forward behind vehicle`s front windshield.
| Type | Specification |
|---|---|
| Prophesee EV4 HD |
IMX636ES (HD) event vision sensor,
Resolution (px): 1280x720,
Latency (µs): 220,
Dynamic range (dB): >86,
Power consumption: 500mW-1.5W,
Pixel size (µm): 4.86x4.86,
Camera max bandwidth (Gbps): 1.6,
Interface: USB 3.0
|
| ELP USB Camera |
CMOS 1080p sensor,
Resolution (px): 1280x720,
Interface: USB 2.0,
5-50mm varifocal lens
|
| Sparkfun GPS-RTK-SMA |
Horizontal accuracy: 2.5m w/o RTK,
Max altitude: 50km,
Max velocity: 500m/s,
GPS, GLONASS, Galileo, BeiDou
|
Sensor specifications.
Example images in processed dataset (from left to right columns): naive conversion, E2VID reconstruction, RGB reference. Each row is the same visual scene. Each column is the same dataset.
| Datasets | NetVLAD | RegionVLAD | HOG | AMOSNet | HybridNet | CALC |
|---|---|---|---|---|---|---|
| NYC-Event-VPR-Naive-5m | 32.53 | 24.85 | 25.13 | 65.76 | 68.69 | 74.17 |
| NYC-Event-VPR-Naive-2m | 25.68 | 20.35 | 21.02 | 57.74 | 61.84 | 66.87 |
| NYC-Event-VPR-E2VID-5m | 74.06 | 77.56 | 86.03 | 86.22 | 85.51 | 84.69 |
| NYC-Event-VPR-E2VID-2m | 68.08 | 73.44 | 83.84 | 82.99 | 82.12 | 81.05 |
| NYC-Event-VPR-RGB-5m | 92.52 | 92.58 | 92.43 | 94.52 | 94.33 | 92.86 |
| NYC-Event-VPR-RGB-2m | 88.34 | 89.94 | 91.48 | 91.82 | 92.36 | 90.16 |
| Pittsburgh250K | 94.36 | 73.34 | 0.27 | 8.53 | 8.70 | 2.05 |
| Nordland | 8.49 | 13.33 | 2.89 | 30.13 | 17.52 | 12.91 |
Quantitative results of Area Under the Curve- Precision Recall (AUC-PR) in percentage.
Top 6 datasets are from NYC-Event-VPR. Each row represents a set of 6 benchmark tests on that dataset. Bottom 2 datasets are curated subsets of their respective datasets provided by VPR-Bench. All tests are done using pretrained weights provided by VPR-Bench.
| CCT384+NetVLAD | Naive-5m | E2VID-5m | RGB-5m | Naive-15m | E2VID-15m | RGB-15m | Naive-25m | E2VID-25m | RGB-25m |
|---|---|---|---|---|---|---|---|---|---|
| Recall@1 | 33.2 | 48.7 | 57.7 | 51 | 70.5 | 77.8 | 57.3 | 77.9 | 86 |
| Recall@5 | 45.1 | 54.4 | 61.3 | 64 | 78.2 | 81.4 | 74.1 | 85.4 | 90.1 |
| Recall@10 | 48.7 | 55.8 | 62.4 | 68.6 | 80.3 | 82.4 | 79.5 | 87.1 | 91 |
| Recall@20 | 51.2 | 57.1 | 63.2 | 73.2 | 81.7 | 83.1 | 83.9 | 88.7 | 91.7 |
| ResNet50+NetVLAD | Naive-5m | E2VID-5m | RGB-5m | Naive-15m | E2VID-15m | RGB-15m | Naive-25m | E2VID-25m | RGB-25m |
|---|---|---|---|---|---|---|---|---|---|
| Recall@1 | 39.1 | 51.9 | 59.1 | 54.9 | 73.1 | 79 | 62.3 | 80.5 | 86.9 |
| Recall@5 | 48.9 | 54.7 | 62.2 | 68.6 | 78.4 | 82.4 | 76.4 | 85.8 | 91 |
| Recall@10 | 51.5 | 55.6 | 62.5 | 72.6 | 79.8 | 83.1 | 80.7 | 87.5 | 91.5 |
| Recall@20 | 53.1 | 56.2 | 62.8 | 75.8 | 80.9 | 83.5 | 83.7 | 89 | 92 |
Quantitative results of recall@k in percentage.
First table is test results of deep learning model trained on NYC-Event-VPR. Backbone is CCT384 (Compact Convolutional Transformer). Aggregation is done via NetVLAD.
Second table is also test results of deep learning model trained on NYC-Event-VPR. Backbone is ResNet50 (Residual Network). Aggregation is done via NetVLAD.
All benchmarks are done by training the model on NYC-Event-VPR dataset using Deep Visual Geo-localization Benchmark framework.