Architecture

From input to features

I-JEPA, used for real images

Click the picture to open it full size.

I-JEPA: blocks of the image are hidden, a context encoder and predictor predict their features, and a slow EMA target encoder provides the answers

Layers of this model

A small version made for 32×32 images. The original I-JEPA paper uses much larger Vision Transformers.

PartOperationOutputTrained by
Patches4×4 pixels, 3×32×32 input, linear embedding64 tokens × 192Gradients
PositionsOne learned vector per patch64 tokens × 192Gradients
Context encoder6 Transformer layers, 4 heads, on visible tokens onlyVisible tokens × 192Gradients
Target encoderSame network, on all tokens, then LayerNorm64 tokens × 192EMA, 0.996 rising to 1
PredictorProjects to 96, 3 Transformer layers, 4 heads, projects backTarget tokens × 192Gradients
LossMSE, predicted vs target featuresOne number—
Image featureMean of the target encoder's tokens192 numbersUsed after training

The smallest JEPA (Lesson 1)

The same idea, with no images or attention, so each part is easy to see.