Scenarios 36 - 39
Illustration of the DeepSense V2V testbed utilized for data collection. Please refer to the detailed description of the testbed presented here.
Scenarios 36 – 39 are collected in an outdoor wireless environment and are explicitly designed to study high-frequency V2V communication in the real world. The DeepSense testbed 6 is utilized in the data collection. It consists of two units: (i) Unit 1, mobile receiver (vehicle) equipped with four 60-GHz mmWave Phased arrays facing four different directions, i.e., front, back, left, right. Each phased array adopts a uniform linear array (ULA) with 16 elements and utilizes an over-sampled pre-defined codebook of 64 beam directions to receive the transmitted signal. It is further equipped with a 360-degree RGB camera, four mmWave single-chip FMCW radars (operating between 76-81GHz), one 3D LiDAR with 32 vertical and 1024 azimuth channels, and one GPS RTK kit. (ii) Unit 2, a mobile transmitter equipped with a 60-GHz quasi-omni antenna always oriented towards the receiver unit and a GPS receiver to capture the real-time position information. Please refer to the detailed description of the testbed presented here.
The DeepSense V2V dataset consists of four different scenarios, namely Scenarios 36-39, each capturing different driving conditions and characteristics. Scenarios 36 and 37 are collected in long drives between cities, targeting long travels, and are referred to as inter-city scenarios. Scenarios 38 and 39 are predominantly focused on emulating short urban commutes, capturing data primarily within city boundaries. Hence, we classify Scenarios 38 and 39 as urban scenarios. The distinction between these sets is supported by the variations in traveled distances and average speeds of the vehicles. Scenarios 36 and 37 involve traveling long distances at relatively high average speeds, while Scenarios 38 and 39 cover shorter distances at lower speeds due to speed limits imposed within cities.
This figure presents the satellite images with the locations of each scenario. It further includes several macro statistics of the data collection providing contextual and objective information derived mainly from the GPS sensors.
Note: the statistics hereby displayed refer to the totality of scenario data. For purposes of fair performance testing, a small portion of samples (~10%) was removed leading to the number of samples presented in the actual dataset being decreased from ~125k to ~112k. In the rest of this page, we refer to the development dataset.
Number of Data Collection Units: 2 (using DeepSense Testbed #6)
Total Number of Data Samples: 112200
Scenario-wise Number of Samples:
Scenario 36: 24800 || Scenario 37: 31000 || Scenario 38: 36000 || Scenario 39: 20400
Data Modalities: 360° RGB images, 4 x 64-dimensional received power vector, GPS locations, 3D LiDAR point-cloud, and 4 x FMCW radar signal
Sensors at Unit 1: (Mobile Receiver)
- Wireless Sensor [Phased Array]: Four 60-GHz mmWave Phased arrays facing four different directions, i.e., front, back, left, and right. Each phased array adopts a uniform linear array (ULA) with 16 elements and utilizes an over-sampled pre-defined codebook of 64 beam directions to receive the transmitted signal
- Visual Sensor [Camera]: The main visual perception element in the testbed is a 360-degree camera. It is used to export four 90º views and two 180º views around the car, effectively covering all angles and emulating the existence of multiple cameras around the vehicle. This camera has the capability to capture 360-degree videos at a resolution of 5.7 K and a frame rate of 30 FPS
- Position Sensor [GPS Receiver]: To capture precise real-time locations for the unit 1 vehicle, a GPS-RTK receiver is employed. This receiver utilizes the L1 and L2 bands, enabling enhanced accuracy. As per the manufacturer’s specifications and the device’s horizontal dilution measures, the horizontal accuracies consistently remain within one meter of the true position
- 3D LiDAR Sensor: The LiDAR sensor in the testbed offers range and angle data for objects within its field of view. With a maximum motor spin frequency of 20Hz, it generates a 32 thousand-point 3D point cloud. The LiDAR has a maximum range capability of 200 meters, allowing for comprehensive scanning and perception of the environment
- FMCW radar: Similar to the phased arrays, the testbed incorporates four Frequency Modulated Continuous Wave (FMCW) radar sensors, each positioned to cover a different direction, ensuring complete 360-degree coverage. The FMCW radar samples consist of a 3D complex radar frame comprising the number of receive (RX) antennas, the number of samples per chirp, and the number of chirps per frame [(# of RX antennas) x (# of samples per chirp) x (# of chirps per frame)]
Sensors at Unit 2: (Mobile Transmitter)
- Wireless Sensor [Phased Array]: A 60 GHz mmWave transmitter. The transmitter (Unit 2) constantly transmits using one antenna element of the phased array to realize omnidirectional transmission
- Position Sensor [GPS Receiver]: A GPS-RTK receiver for capturing accurate real-time locations for the mobile unit 2
|Instances||Scenario 36: 24800 || Scenario 37: 31000 || Scenario 38: 36000 || Scenario 39: 20400|
|Number of Units||2|
|Data Modalities||360° RGB images, 4 x 64-dimensional received power vector, GPS locations, 3D LiDAR point-cloud, and 4 x FMCW radar signal|
|Hardware Elements||360-degree RGB camera, 4 x mmWave phased array receiver, GPS receiver, 3D LiDAR, 4 x FMCW radar|
|Data Modalities||360-degree RGB images, 4 x 64-dimensional received power vector, GPS locations, LiDAR point cloud, radar data|
|Hardware Elements||mmWave omni-directional transmitter, GPS receiver|
|Data Modalities||GPS locations|
The DeepSense scenario creation pipeline involves two important stages: data processing and data visualization.
Data Processing: In the data processing stage, the collected sensor data is transformed into a common format that can be easily ingested and synchronized. This stage consists of two phases. Phase 1 focuses on converting data from different sensors into timestamped samples and organizing them in clear CSVs. It also involves interpolating GPS data to ensure proper synchronization. Phase 2 filters, organizes, and synchronizes the extracted data into a processed DeepSense scenario. This phase includes tasks such as data synchronization, filtering, sequencing, labeling, and data compression.
Data synchronization aligns samples from different sensors to a uniform sample rate based on timestamp proximity. Filtering involves rejecting samples based on specific criteria, including acquisition errors, non-coexistence of sensor sampling, and sequence filtering. Sequencing groups continuous samples together, ensuring a maximum 0.1-second gap between samples. Data labels provide additional contextual information, which can be derived from sensors or manually added. Data compression is performed to reduce file size, improve distribution efficiency, and allow selective downloading of specific modalities.
Data Visualization: The data visualization stage plays a crucial role in interpreting and understanding the dataset. It leverages the DeepSense Viewer library, providing a user interface (UI) for data visualization. Scenario videos are rendered, displaying all processed data synchronized over time. These videos facilitate navigation through the dataset, identification of relevant samples, and observation of propagation phenomena and sensor conditions.
Scenario Data Visualization
Please login to download the DeepSense datasets
How to Access Scenario Data?
Step 1. Download Scenario Data
1.1 – Click one of the Download buttons above to access the download page of each scenario.
1.2 – Select the modalities of interest and download (individually!) all zip files of the modality. E.g., if interested in Scenario 36 GPS and RGB images with 180º FoV, download the following files:
- scenario36_rgb180.zip.001, scenario36_rgb180.zip.002, (…)003, (…)004, and scenario36_rgb180.zip.005.
Step 2. Extract modality data
2.1 – Install the 7Zip utility (download here) to be able to extract ZIP files split in parts
2.2 – Select all zip files downloaded and select “Extract here”.
(Explanation: This will extract the scenario folder from inside the zip files. Each zip file has a scenario folder with the same name, but different modality data inside. As a results, the scenario folders from the zips of different modalities will overlap and merge automatically, leading to a single scenario folder with the right structure. )
Note: if there is a problem with the extraction, ensure the files downloaded have the right sizes listed in this table.
2.3 – Move the CSV file inside the scenario folder extract from the ZIPs.
Scenario X folder consists of three data sub-folders:
- unit1: Includes the data captured by unit 1
- unit2: Includes the data captured by unit 2
- resources: Includes scenario-specific annotations, data labels, and other additional information. For more details, refer to the resources section below.
Besides data, the scenario folder includes two useful resources to provide context and facilitate the use of the data:
- “scenario X.csv” file: contains all sample indices, timestamps, sensor-specific metadata, and paths to each modality. Each sampling interval leads to a new row in this CSV, and it can be used to load all scenario data
- “scenarioX.p” and “scenarioX.mat”: dictionaries for Python and Matlab that contain all the information present in the CSV file. The advantage of these structures compared with the CSV is that data such as the GPS positions or received power vectors are already pre-loaded, allowing quicker and more convenient access to the data.
Next, is a video explaining the steps of accessing the dataset
What are the Additional Resources?
The additional resources in this scenario consist of satellite images. One clear example of these images is presented in the Scenario videos in the “Data Visualization” section of this page. The satellite images are the satellite background that changes as the GPS positions change. These images provide context in the immediate surroundings of vehicles and allow for informative GPS plots. The paths to the satellite images resources are listed in the CSV and in the pre-loaded dictionaries with the column/keyword name of satellite_img.
Additional Information: Explanation of Data in CSV
We further provide additional information for each sample present in the scenario dataset. The information provided here gives respect to the data listed in scenarioX.csv. Here we explain what each column means.
- abs_index: Represents the sample absolute index. This is the sample index that is displayed in the video. This index remains unchanged from the sample collection all the way to the processed scenario. The advantage is a perfect backtrace of information across processing phases. Moreover, if problems with the data exist, mentioning this sample index allows easier interactions with the DeepSense team.
- Timestamp: This column represents the time of data capture in “hr-mins-secs.us” format with respect to the current timezone. It not only tells the time of the data collection but also tells the time difference between sample captures.
- seq_index: The index of a sequence. Samples have different sequence indices for two reasons. The main reason is to separate continuous sampling intervals. Samples with the same index have been collected at precisely the same sampling rate (10 Hz in this case). As such, when the sequence index changes, it is because some circumstances of the data collection lead to a sample loss or similar, thus disrupting the continuity. However, it is possible that adjacent samples with different sequence indices have a 100 ms interval between them, thus actually respecting the continuity. In this case, the second reason for changing sequences applies. The second reason for changing sequences is to create 20-second chunks of data that can be separated for training and testing purposes. No sequence is larger than 20 s (200 samples) because of this reason. If the last sample of a sequence and the first sample of the next have consecutive absolute indices (abs_index column), then the sequence changed because of the 20 s rule, not because of a break in continuity.
Sensors can be gps1, pwr2, lidar1, radar3, etc. Sensors are associated with units. unit1_gps1 refers to the first GPS of unit 1. unit2_gps1 refers to the first GPS of unit 2. In these scenarios, only radar and power have more than one sensor per unit. It is important to recognize that for the power and radar sensors, the order (1, 2, 3, 4) is not arbitrary – it represents the directions they face. pwr1 regards the received power from the phased array in the front of the car. pwr2, pwr3, and pwr4 are from right, back, and left (clockwise direction), similarly with radar 1 to 4. Therefore, unit1_pwr1 will have information about the phased array in the front, and unit1_radar4 will have the raw radar samples from the radar pointing to the left. Sensors may additionally have labels. Labels are metadata acquired from that sensor. In these scenarios, only two sensors have metadata, and the labels are as follows:
- gps1_lat / gps1_lon / gps1_altitude: Latitude, longitude, and altitude.
- gps1_pdop / gps1_hdop / gps1_vdop: Position, horizontal and vertical, dilutions of precision. Measures of GPS confidence.
- pwr1_best-beam: a value between 0 and 63 representing the index of the beam that received the maximum power in the current time instant.
- pwr1_max-pwr / pwr1_min-pwr: values of the baseband received powers for a receive power vector of a certain array. These are the maximum and minimum powers computed in the baseband.
Information regarding a certain unit. unit1 refers to the receiver, unit2 to the transmitter. These scenarios have mostly sensor data associated with each unit, except the unit1 which has one label computed to facilitate the use of the dataset:
- unit1_overall-beam: a value between 0 and 255 indicating the best beam across all phased arrays. Beam 0 corresponds to the first beam of the first array (unit1_pwr1) and beam 255 corresponds to the last beam of the last phased array (unit1_pwr4). Note further that the phased arrays sweep 64 beams in azimuth and the first beam is the leftmost beam. As such, one can imagine beam 0 pointing to 45º to the left of the direction the vehicle is moving, beam 63 pointing 45º to the right of this direction, and the remaining beams pseudo-uniformly covering the angular domain clockwise. Please refer to the image below.