README.md

# DROID Annotations

This repo contains additional annotation data for the DROID dataset which we completed after the initial dataset release.

Concretely, it contains the following information:
- the full language annotations for the DROID dataset (3x annotations for 95+% of the 75k successful demonstrations)
- higher-accuracy camera calibrations (for 36k episodes)


## Language Annotations
The released RLDS dataset only contains a subset of the language labels. 
Here, we provide a single json file with a mapping from episode ID to three natural language annotations, 
for 75k success episodes in DROID (95% of all DROID success episodes).


## Camera Calibrations
The original DROID release included noisy camera calibrations. In post-hoc processing, we generated more accurate extrinsic camera calibration parameters
for a subset of the DROID episodes. Concretely, we provide the following three calibration files:
- `cam2base_extrinsics.json`: Contains ~36k entries with either the left or right camera calibrated with respect to base.
- `cam2cam_extrinsics.json`: Contains ~90k entries with cam2cam relative poses and camera parameters for all of DROID.
- `cam2base_extrinsic_superset.json`: Contains ~24k unique entries, total ~48k poses for both left and right camera calibrated with respect to the base.


## Accessing Annotation Data

All annotations are stored in `json` files which you can download from this repository.
To access the respective annotation for a particular episode, you can compute the episode's ID. 
It corresponds to the file name `metadata_<episode_id>.json` in every episode's folder in the raw DROID data. 
To extract episode IDs and corresponding file paths, you can use the code below. 
You can then e.g. match the filepath to the one stored in the RLDS dataset if you want to match language annotations / camera extrinsics to RLDS episodes.

```
import tensorflow as tf

episode_paths = tf.io.gfile.glob("gs://gresearch/robotics/droid_raw/1.0.1/*/success/*/*/metadata_*.json")
for p in episode_paths:
    episode_id = p[:-5].split("/")[-1].split("_")[-1]
```