update

README.md

@@ -1,5 +1,4 @@
 # DROID Annotations
-
 This repo contains additional annotation data for the DROID dataset which we completed after the initial dataset release.
 
 Concretely, it contains the following information:
@@ -19,7 +18,60 @@ for a subset of the DROID episodes. Concretely, we provide the following three c
 - `cam2base_extrinsics.json`: Contains ~36k entries with either the left or right camera calibrated with respect to base.
 - `cam2cam_extrinsics.json`: Contains ~90k entries with cam2cam relative poses and camera parameters for all of DROID.
 - `cam2base_extrinsic_superset.json`: Contains ~24k unique entries, total ~48k poses for both left and right camera calibrated with respect to the base.
+These files map episodes' unique ID (see Accessing Annotation Data below) to another dictionary containing metadata (e.g., detection quality metrics, see Appendix G of paper), as well as a map from camera ID to the extrinsics values. Said extrinsics is represented as a 6-element list of floats, indicating the translation and rotation. It can be easily converted into a homogeneous pose matrix:
+```
+from scipy.spatial.transform import Rotation as R
+
+# Assume extrinsics is that 6-element list
+pos = extrinsics[0:3]
+rot_mat = R.from_euler("xyz", extracted_extrinsics[3:6]).as_matrix()
+
+# Make homogenous transformation matrix
+cam_to_target_extrinsics_matrix = np.eye(4)
+cam_to_target_extrinsics_matrix[:3, :3] = rot_mat
+cam_to_target_extrinsics_matrix[:3, 3] = pos
+```
+This represents a transformation matrix from the camera's frame to the target frame. Inverting it gets the transformation from target frame to camera frame (which is usually desirable, e.g., if one wants to project a point in the robot frame into the camera frame).
+
+As the raw DROID video files were recorded on Zed cameras and saved in SVO format, they contain camera intrinsics which can be used in conjunction with the above. For convenience, we have extracted and saved all these annotations to `intrinsics.json` (~72k entries). This `json` has the following format:
+```
+<episode ID>:
+    <external camera 1's serial>: [fx, cx, fy, cy for camera 1]
+    <external camera 2's serial>: [fx, cx, fy, cy for camera 2]
+    <wrist camera 1's serial>: [fx, cx, fy, cy for wrist camera]
+```
+One can thus convert the list for a particular camera to a projection matrix via the following:
+```
+import numpy as np
+
+# Assume intrinsics is that 4-element list
+fx, cx, fy, cy = intrinsics
+intrinsics_matrix = np.array([
+        [fx, 0, cx],
+        [0, fy, cy],
+        [0, 0, 1]
+])
+```
+Note that the intrinsics tend to not change much between episodes, but using the specific values corresponding to a particular episode tends to give the best results.
+
+## Example Calibration Use Case
+Using the calibration information, one can project points in the robot's frame into pixel coordinates for the cameras. We will demonstrate how to map the robot gripper position to pixel coordinates for the external cameras with extrinsics in `cam2base_extrinsics.json`, see <TODO> for the full code.
+```
+gripper_position_base = <Homogeneous gripper position in the base frame, as gotten from TFDS episode. Shape 4 x 1>
+cam_to_base_extrinsics_matrix = <extrinsics matrix for some camera>
+intrinsics_matrix = <intrinsics matrix for that same camera>
+
+# Invert to get transform from base to camera frame
+base_to_cam_extrinsics_matrix = np.linalg.inv(cam_to_base_extrinsics_matrix)
+
+# Transform gripper position to camera frame, then remove homogeneous component
+robot_gripper_position_cam = base_to_cam_extrinsics_matrix @ gripper_position_base
+robot_gripper_position_cam = robot_gripper_position_cam[:3] # Now 3 x 1
 
+# Project into pixel coordinates
+pixel_positions = intrinsics_matrix @ robot_gripper_position_cam
+pixel_positions = pixel_positions[:2] / pixel_positions[2] # Shape 2 x 1 # Done!
+```
 
 ## Accessing Annotation Data
 
@@ -35,4 +87,13 @@ import tensorflow as tf
 episode_paths = tf.io.gfile.glob("gs://gresearch/robotics/droid_raw/1.0.1/*/success/*/*/metadata_*.json")
 for p in episode_paths:
     episode_id = p[:-5].split("/")[-1].split("_")[-1]
-```
+```
+
+As using the above annotations requires these episode IDs (but the TFDS dataset only contains paths), we have included `episode_id_to_path.json` for convenience. The below code snippet loads this `json`, then gets the mapping from episode paths to IDs.
+```
+import json
+episode_id_to_path_path = "<path/to/episode_id_to_path.json>"
+with open(episode_id_to_path_path, "r") as f:
+    episode_id_to_path = json.load(f)
+episode_path_to_id = {v: k for k, v in episode_id_to_path.items()}
+```

camera_serials.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c8d346c51dcef71248e280e44dcd7985a94433f6911460b31dcc098cab30acc4
+size 12743876

episode_id_to_path.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e88ee7da94a40602cde4aacf22f2b48068f4f582c8ab38cf1888e06162a8085
+size 7237770

intrinsics.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78c76755b075ae53e74a28c543bb1b185c50aa976458e95fbc9ba880a8cd2d51
+size 125812944