orrav/sae-clip-b32-x64-layer11-resid-post-cls-lr5e-3

CLIP-B-32 Sparse Autoencoder x64 vanilla - L1:1e-05

Training Details

• Base Model: CLIP-ViT-B-32 (LAION DataComp.XL-s13B-b90K)
• Layer: 11
• Component: hook_resid_post

Model Architecture

• Input Dimension: 768
• SAE Dimension: 49,152
• Expansion Factor: x64 (vanilla architecture)
• Activation Function: ReLU
• Initialization: encoder_transpose_decoder
• CLS_only: true

Performance Metrics

• L1 Coefficient: 1e-05
• L0 Sparsity: 691.3
• Explained Variance: 87.84%

Training Configuration

• Learning Rate: 0.005
• LR Scheduler: Cosine Annealing with Warmup (200 steps)
• Epochs: 10
• Gradient Clipping: 1.0