[Dataset] Autonomous Driving Open Dataset: nuScenes Dataset(+ nuImages, nuPlan, Occupancy, nuReality)

💡 본 문서는 'Autonomous Driving Open Dataset: nuScenes Dataset'에 대해 정리해놓은 글입니다.
자율주행 차량의 센서 데이터셋 중 하나인 nuScenes Dataset에 대해 정리하였으니 참고하시기 바랍니다.

nuScenes Dataset

1) Sensor 구성

camera 6개 + lidar 1개 + radar 5개

\

nuScenes dataset은 2019년에 공개된 오픈데이터로 detection, tracking, prediction & localization task을 지원하는 multi modal dataset입니다. 단순히 image만 제공하는 것이 아닌 camera로 수집한 image, Lidar로 수집한 point cloud, radar로 수집한 point cloud 등이 제공됩니다.

dataset안에는 140만 개의 카메라 이미지, 39만 개의 라이다 정보, 140만 개의 레이더 정보, 140만 개의 object bounding box가 포함되어 있습니다. 전체적인 데이터의 구성은 다음과 같습니다.

2) A Introduction to nuScenes

In this part of the tutorial, let us go through a top-down introduction of our database. Our dataset comprises of elemental building blocks that are the following:

1. log - Log information from which the data was extracted.
2. scene - 20 second snippet of a car's journey.
3. sample - An annotated snapshot of a scene at a particular timestamp.
4. sample_data - Data collected from a particular sensor.
5. ego_pose - Ego vehicle poses at a particular timestamp.
6. sensor - A specific sensor type.
7. calibrated sensor - Definition of a particular sensor as calibrated on a particular vehicle.
8. instance - Enumeration of all object instance we observed.
9. category - Taxonomy of object categories (e.g. vehicle, human).
10. attribute - Property of an instance that can change while the category remains the same.
11. visibility - Fraction of pixels visible in all the images collected from 6 different cameras.
12. sample_annotation - An annotated instance of an object within our interest.
13. map - Map data that is stored as binary semantic masks from a top-down view.

The database schema is visualized below. For more information see the nuScenes schema page. 

3) nuScenes Schema

nuScenes에서 detection에서 사용하는 class는 총 10가지로 Car, Bus, Bicycle, Barrier, Construction_vehicle, Motorcycle, Pedestrian, Traffic_cone, Trailer, Truck 입니다.

annotation을 한 기준을 살펴보면

• 물체는 위치와 모양을 알 수 있도록 적어도 LiDAR나 Radar point 1개가 포함되어야 합니다.
• 물체를 나타내는 직육면체는 매우 tight해야합니다.
• 물체와 끝과 끝은 모두 포함되어야 합니다.
  → 물체가 이어져있다면 끊기지 않고 하나에 전체를 포함해야 합니다.
• 보행자가 운반하는 물건도 포함합니다.
• 간혹 움직이지 않는 물체가 움직이는 경우 에러를 방지하기 위해 별도의 bbox를 만듭니다.
• LiDAR나 Radar에 잘 찍히지 않는 물체는 카메라 이미지를 통해 크기를 판별합니다.
• 모든 카메라가 볼 수 있는 물체는 특별한 속성을 부여합니다.

자세한 data의 내용은 nuScenes 논문이나 홈페이지를 가면 확인할 수 있고 데이터를 확인해보고 싶다면 아래의 tutorial을 진행해보시길 바랍니다.

• [Official] nuScenes Dataset tutorial: https://www.nuscenes.org/tutorials/nuscenes_tutorial.html

nuScenes Download Scripts

#!/usr/bin/bash

mkdir done_unzipping

# The download links may change over time.
wget https://d36yt3mvayqw5m.cloudfront.net/public/v1.0/v1.0-trainval_meta.tgz
wget https://motional-nuscenes.s3.amazonaws.com/public/v1.0/v1.0-trainval01_blobs.tgz
wget https://motional-nuscenes.s3.amazonaws.com/public/v1.0/v1.0-trainval02_blobs.tgz
wget https://d36yt3mvayqw5m.cloudfront.net/public/v1.0/v1.0-trainval03_blobs.tgz
wget https://motional-nuscenes.s3.amazonaws.com/public/v1.0/v1.0-trainval04_blobs.tgz
wget https://d36yt3mvayqw5m.cloudfront.net/public/v1.0/v1.0-trainval05_blobs.tgz
wget https://d36yt3mvayqw5m.cloudfront.net/public/v1.0/v1.0-trainval06_blobs.tgz
wget https://d36yt3mvayqw5m.cloudfront.net/public/v1.0/v1.0-trainval07_blobs.tgz
wget https://motional-nuscenes.s3.amazonaws.com/public/v1.0/v1.0-trainval08_blobs.tgz
wget https://motional-nuscenes.s3.amazonaws.com/public/v1.0/v1.0-trainval09_blobs.tgz
wget https://motional-nuscenes.s3.amazonaws.com/public/v1.0/v1.0-trainval10_blobs.tgz
wget https://motional-nuscenes.s3.amazonaws.com/public/v1.0/v1.0-test_blobs.tgz
wget https://d36yt3mvayqw5m.cloudfront.net/public/v1.0/v1.0-trainval_meta.tgz

tar -xzvf v1.0-trainval_meta.tgz && mv v1.0-trainval_meta.tgz ./done_unzipping/
tar -xzvf v1.0-trainval01_blobs.tgz && mv v1.0-trainval01_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-trainval02_blobs.tgz && mv v1.0-trainval02_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-trainval03_blobs.tgz && mv v1.0-trainval03_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-trainval04_blobs.tgz && mv v1.0-trainval04_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-trainval05_blobs.tgz && mv v1.0-trainval05_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-trainval06_blobs.tgz && mv v1.0-trainval06_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-trainval07_blobs.tgz && mv v1.0-trainval07_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-trainval08_blobs.tgz && mv v1.0-trainval08_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-trainval09_blobs.tgz && mv v1.0-trainval09_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-trainval10_blobs.tgz && mv v1.0-trainval10_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-test_blobs.tgz && mv v1.0-test_blobs.tgz ./done_unzipping/
tar -xzvf v1.0-trainval_meta.tgz && mv v1.0-trainval_meta.tgz ./done_unzipping/

 

nuScenes Related Source

• [colab] nutonomy/nuscenes-devkit: https://colab.research.google.com/github/nutonomy/nuscenes-devkit/
• [Github] nutonomy/nuscenes-devkit: https://github.com/nutonomy/nuscenes-devkit
• [Github] chiyukunpeng/nuscenes_viz: https://github.com/chiyukunpeng/nuscenes_viz
• [Github] clynamen/nuscenes2bag: https://github.com/clynamen/nuscenes2bag

nuScenes Other Dataset

1. nuImages

nuImages is a large-scale autonomous driving dataset with image-level 2d annotations. It features:

• 93k video clips of 6s each (150h of driving)
• 93k annotated and 1.1M un-annotated images
• Two diverse cities: Boston and Singapore
• The same proven sensor suite as in nuScenes
• Images mined for diversity
• 800k annotated foreground objects with 2d bounding boxes and instance masks
• 100k 2d semantic segmentation masks for background classes
• Attributes such as rider, pose, activity, emergency lights and flying
• Free to use for non-commercial use
• For a commercial license contact nuScenes@motional.com

2. nuPlan

nuPlan is the world's first large-scale planning benchmark for autonomous driving. It features:

• The world's first ML planning benchmark
• 1200h of driving data from 4 cities (Boston, Pittsburgh, Las Vegas and Singapore)
• Sensor data released for 120h (5x LIDAR, 8x camera, IMU, GPS)
• Left versus right hand traffic
• Detailed map information
• 5B 3D bounding boxes auto labeled for 7 classes
• Open and closed loop planning simulation
• 30+ mined scenario types (e.g. lane change, unprotected turn, jaywalker)
• 20+ open and closed loop planning simulation and metrics to score planners (traffic rule violation, human driving similarity, vehicle dynamic, goal achievement)
• Traffic light statuses inferred from agent movement
• Baselines and framework to train reactive agents and ML based planners
• Upcoming challenges around planning and smart agents in 2022
• Free to use for non-commercial use
• For a commercial license contact nuPlan@motional.com

3. nuScenes Occupancy

• [Paperwithcode] Prediction Of Occupancy Grid Maps on Occ3D-nuScenes: https://paperswithcode.com/sota/prediction-of-occupancy-grid-maps-on-occ3d
• [eval.ai] 3D Occupancy Prediction Challenge: https://eval.ai/web/challenges/challenge-page/2045/overview
• [archive] NuScenes Occupancy Grids Dataset: https://archive.org/details/nuscenes-occupancy-grids-dataset
• [Git] Occupancy Dataset for nuScenes: https://github.com/FANG-MING/occupancy-for-nuscenes
• [Git] CVPR2023-3D-Occupancy-Prediction
  : https://github.com/CVPR2023-3D-Occupancy-Prediction/CVPR2023-3D-Occupancy-Prediction
• [Git] huang-yh/selfocc: https://github.com/huang-yh/selfocc
• [Paper] Predicting Future Occupancy Grids in Dynamic Environment with Spatio-Temporal Learning:  https://arxiv.org/pdf/2205.03212v1.pdf
• [Git] OccupancyGrid-Predictions: https://github.com/ksm26/OccupancyGrid-Predictions 

4. nuReality

 

참고

• [Blog] 15 Best Open-Source Autonomous Driving Datasets: https://medium.com/analytics-vidhya/15-best-open-source-autonomous-driving-datasets-34324676c8d7
• [Official] nuScenes Dataset Overview: https://www.nuscenes.org/nuscenes#overview
• [Official] nuScenes Dataset Downloads: https://www.nuscenes.org/nuscenes#download