xocialize/VJEPA2-ViTL-SSv2-Embed-CoreAI

V-JEPA 2 ViT-L (SSv2) — Core AI, two-output (logits + router embedding)

A Core AI export of facebook/vjepa2-vitl-fpc16-256-ssv2 (375M, MIT) that returns both the SSv2 action logits and the pooled [1,1024] router embedding.

pixel_values_videos [1,16,3,256,256] fp16
    ├─ logits     [1,174]    SSv2 action classes
    └─ embedding  [1,1024]   pooled attentive-pooler output

Why two outputs

The stock export returns logits only. That is the wrong contract for anything using V-JEPA 2 as a content/router feature extractor rather than an action classifier: the SSv2 checkpoint ships no label names, its vocabulary is hand-object interactions, and it is semantically wrong for most non-action domains. The [1024] pooled embedding is the reusable signal — feed it to a small calibrated head for your own task.

Built for a video-optimization planner that fits a logistic probe over this embedding to pre-classify content before an encode search.

Measured

Apple M5 Max / 128 GB, fp16, GPU:

Bundle	708 MB (675 MiB)
Load	0.16 s
Warm forward (16-frame clip)	44 ms
Engine logits vs torch oracle	cos 0.999964, top-1 match
Engine embedding vs torch reference	cos 0.999940, relL2 1.1e-02 (fp16 noise)
Recomposition classifier(pooler(backbone(x))) vs .logits	cos 1.0000000, max|Δ| 0 — bit-exact

⚠️ Latency is silicon-specific. Published M4 Max figures for the single-output bundle are 150–180 ms; that is different hardware, not a difference between these bundles.

Embedding separation (6 signage clips, a semantic check no numeric gate performs): pairwise cosine spans 0.146 → 0.987, and the highest-similarity pair is the same content at two different encodes (_1080p vs _master, cos 0.987). Content-identical inputs collapse together; unrelated content separates.

Preprocessing — on the host

The model does not normalize internally.

1. 16 frames, uniformly sampled across the clip
2. aspect-fill to 256×256, centre-crop (not a squash)
3. RGB, 0..1
4. ImageNet mean [0.485, 0.456, 0.406] / std [0.229, 0.224, 0.225]
5. [1,16,3,256,256], fp16

Use

import asyncio, numpy as np, coreai.runtime as rt

async def main():
    opts = rt.SpecializationOptions.from_preferred_compute_unit_kind(rt.ComputeUnitKind.gpu())
    fn = (await rt.AIModel.load("vjepa2_ssv2_embed_fp16.aimodel", opts)).load_function("main")
    x = ...  # [1,16,3,256,256] fp16, preprocessed as above
    out = await fn(inputs={"pixel_values_videos": rt.NDArray(np.ascontiguousarray(x))})
    logits    = out["logits"].numpy()      # [1,174]
    embedding = out["embedding"].numpy()   # [1,1024]  ← the signal

asyncio.run(main())

Reproduce

export_fp16_embed.py is included. It needs coreai_models, torch, and transformers ≥4.57, plus oracles from the reference run (oracle_*.npy are included here, so the gate can be re-run without recomputing them).

Two things in that script are load-bearing and easy to lose in a re-derivation:

• skip_predictor=True. VJEPA2ForVideoClassification.forward passes it; VJEPA2Model defaults it to False, which runs the entire JEPA predictor stack. last_hidden_state is sequence_output either way, so omitting it changes no number — every parity gate still passes, and the bundle silently carries a transformer stack nothing reads. Parity gates certify values, never graph extent.
• The 3D-RoPE squeeze(-1) overlay. The reference squeezes a non-1 dim (a torch no-op); the converter maps it to ShrinkDims, which requires size 1, and fails. Patched out, math unchanged.

The submodule resolution is discover-then-prove: it resolves backbone/pooler/classifier by candidate name, then asserts the recomposition reproduces the model's own .logits before exporting. If a future transformers renames them, it fails loudly instead of shipping a wrong "embedding".

Files

vjepa2_ssv2_embed_fp16.aimodel/	the bundle (metadata.json, main.mlirb, main.hash)
export_fp16_embed.py	the exact export that produced it
oracle_input.npy / oracle_logits.npy / oracle_embedding.npy	gate fixtures

License

MIT, inherited from Meta's V-JEPA 2.