Add Gemma-GR00T model weights

README.md

@@ -3,29 +3,44 @@ language:
 - en
 license: mit
 library_name: transformers
-pipeline_tag: reinforcement-learning
+# Using text-to-video as the pipeline tag since the model generates action sequences from vision and language inputs
+pipeline_tag: text-to-video
 datasets:
 - lerobot/robot_sim.PickNPlace
 - lerobot/so100_strawberry_grape
 base_model: NVEagle/eagle_er-qwen3_1_7B-Siglip2_400M_stage1_5_128gpu_er_v7_1mlp_nops
 tags:
 - robotics
+- vision-language-action
 - reinforcement-learning
 - imitation-learning
-- gemma
-- gr00t
 - nvidia
-- lerobot
-- vision-language-action
-- robot-manipulation
-- gemma-le
+- gr00t
+- gemma
 - diffusion-policy
-- le-robot
+- lerobot
 - robot-learning
 - embodied-ai
+- humanoid-robots
+- robot-manipulation
+- computer-vision
+- natural-language-processing
+- deep-learning
+- transformer
+- vision-transformer
+- flow-matching
+- foundation-model
+- multi-modal
+- human-robot-interaction
+- autonomous-robots
+- robot-control
+- robot-perception
+- robot-vision
 ---
 
-# Gemma-GR00T: Multimodal Robotic Manipulation with Language Models
+# Gemma-GR00T: A Vision-Language-Action Model for Robotic Control
+
+This is a fine-tuned version of the NVIDIA GR00T N1.5 model, adapted for robotic control tasks using the LeRobot framework. The model combines vision, language, and action generation capabilities to enable robots to perform complex manipulation tasks based on natural language instructions.
 
 ## Model Description
 
@@ -33,46 +48,124 @@ Gemma-GR00T is a state-of-the-art multimodal vision-language-action policy that
 
 ## Model Details
 
-- **Developed by:** [Gyanateet Dutta](https://huggingface.co/Ryukijano)
-- **Model type:** Vision-Language-Action Policy
-- **Language(s) (NLP):** English
-- **License:** MIT
-- **Finetuned from model:** [NVEagle/eagle_er-qwen3_1_7B-Siglip2_400M_stage1_5_128gpu_er_v7_1mlp_nops](https://huggingface.co/NVEagle/eagle_er-qwen3_1_7B-Siglip2_400M_stage1_5_128gpu_er_v7_1mlp_nops)
+- **Model type:** Vision-Language-Action (VLA) model
+- **Base Model:** [NVEagle/eagle_er-qwen3_1_7B-Siglip2_400M_stage1_5_128gpu_er_v7_1mlp_nops](https://huggingface.co/NVEagle/eagle_er-qwen3_1_7B-Siglip2_400M_stage1_5_128gpu_er_v7_1mlp_nops)
+- **Task:** text-to-video (robot action generation from vision and language)
 - **Training Data:** Trained on LeRobot datasets using the `fourier_gr1_arms_only` configuration
 - **Framework:** PyTorch with Hugging Face Transformers
-- **Related Models:** [LeRobot Models](https://huggingface.co/lerobot)
+- **Related Models:** [NVIDIA GR00T-N1.5-3B](https://huggingface.co/nvidia/GR00T-N1.5-3B), [LeRobot Models](https://huggingface.co/lerobot)
 - **Related Datasets:** [LeRobot Datasets](https://huggingface.co/lerobot/datasets)
 
 ### Model Architecture
 
 The model is built on a sophisticated multimodal architecture that combines state-of-the-art vision and language models for robotic control:
 
-1. **Backbone**: `Eagle2_5_VLForConditionalGeneration` - A powerful vision-language model that processes both visual and textual inputs.
+1. **Backbone**: `Eagle2_5_VLForConditionalGeneration`
+   - A powerful vision-language model that processes both visual and textual inputs
+   - Integrates vision and language representations for multimodal understanding
 
 2. **Text Encoder**: `Qwen3-1.7B`
+   - Base Model: [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B)
    - Type: Causal Language Model
    - Parameters: 1.7B
    - Layers: 28
    - Attention: 16 heads for Q, 8 heads for KV (GQA)
    - Context Length: 32,768 tokens
-   - Features: Strong reasoning and instruction-following capabilities
+   - Features: 
+     - Strong reasoning and instruction-following capabilities
+     - Optimized for long-context understanding
+     - Supports complex language understanding and generation
 
 3. **Vision Encoder**: `SigLIP` (Sigmoid Loss for Language-Image Pre-training)
+   - Base Model: [google/siglip-base-patch16-224](https://huggingface.co/google/siglip-base-patch16-224)
    - Type: Vision Transformer (ViT)
-   - Patch Size: 14x14
-   - Hidden Size: 1,152
-   - Layers: 27
-   - Attention Heads: 16
-   - Features: Strong visual representation learning with localization capabilities
+   - Patch Size: 16x16
+   - Image Size: 224x224
+   - Hidden Size: 768
+   - Layers: 12
+   - Attention Heads: 12
+   - Features:
+     - Strong visual representation learning
+     - Excellent zero-shot classification capabilities
+     - Robust to various visual domains
 
 4. **Action Head**: Diffusion-based Policy
    - Type: Flow-matching action head
-   - Architecture: 16-layer transformer
-   - Hidden Size: 1,024
-   - Attention Heads: 32
-   - Features: Generates smooth, continuous actions for robotic control
-
-The model processes visual inputs through the SigLIP vision encoder and textual instructions through the Qwen3-1.7B language model, then fuses these representations in the Eagle2.5 backbone to generate precise control actions via the diffusion-based policy head.
+   - Architecture: 4-layer transformer (ScaledDP)
+   - Hidden Size: 512
+   - Feed-Forward Size: 2,048
+   - Attention Heads: 8
+   - Features: 
+     - Generates smooth, continuous actions for robotic control
+     - Uses diffusion process for action generation
+
+## Training & Evaluation
+
+### Training Performance
+
+- **Total Training Steps**: 30,000
+- **Final Epoch**: 114.5
+- **Initial Loss**: 1.27
+- **Final Loss**: 0.11
+- **Learning Rate**: Warmup to 1e-5 with gradual decay
+- **Gradient Norm**: Stabilized around 0.3-1.0 (initial: 11.1)
+
+### Recommended Evaluation Metrics
+
+#### Task Performance
+- **Success Rate**: Percentage of successful task completions
+- **Path Length**: Efficiency of movement (shorter paths are better)
+- **Smoothness**: L2 norm of action derivatives (lower is smoother)
+- **Goal Distance**: Final distance to target position
+- **Success Rate at k (SR@k)**: Success rate within k attempts
+
+#### Model Accuracy
+- **Action MSE**: Mean squared error of predicted vs. ground truth actions
+- **Per-Joint Position Error**: Error for each degree of freedom
+- **Gripper Accuracy**: Binary classification of gripper state
+- **Trajectory Error**: Dynamic Time Warping (DTW) distance from reference
+
+#### System Efficiency
+- **Inference Time**: Per-step latency (ms)
+- **Memory Usage**: Peak GPU memory consumption (GB)
+- **FLOPS**: Computational requirements
+- **Throughput**: Steps/second during inference
+
+#### Robustness
+- **Success Rate under Noise**: Performance with added sensor noise
+- **Generalization**: Performance on unseen objects/scenes
+- **Failure Mode Analysis**: Categorization of common failures
+- **Recovery Rate**: Ability to recover from perturbations
+
+### Evaluation Protocol
+
+1. **Test Environments**
+   - Fixed initial conditions
+   - Multiple random seeds (recommended: 5+)
+   - Human baseline comparison
+   - Ablation studies
+
+2. **Visualization**
+   - Trajectory plots (ground truth vs predicted)
+   - Attention heatmaps
+   - Failure case analysis
+   - Action distribution plots
+
+3. **Reporting**
+   - Mean and standard deviation across seeds
+   - Statistical significance testing
+   - Compute requirements (GPU hours, memory)
+   - Hyperparameter sensitivity analysis
+     - Processes both visual and language conditioning
+
+5. **Training Configuration**:
+   - Optimizer: AdamW (lr=1e-4, weight_decay=1e-6)
+   - Diffusion Steps: 100
+   - Chunk Size: 16
+   - Action Steps: 8
+   - Observation Steps: 1
+
+The model processes visual inputs through the SigLIP vision encoder and textual instructions through the Qwen3-1.7B language model, then fuses these representations in the Eagle2.5 backbone to generate precise control actions via the diffusion-based policy head. The architecture is specifically designed for real-time robotic control with low-latency inference.
 
 ## Uses