Rethinking Contrastive Loss in CLIP Post‑training

Finding 1Temperature matters more than batch size

Inheriting CLIP's pretraining temperature \(\tau \approx 0.07\) reproduces the catastrophic forgetting reported in prior work. With a substantially smaller \(\tau = 0.01\), the same objective, data and compute improve the pretrained model. Both runs below train ViT-B/16 on CC3M for one epoch with \(\mathcal{L}_{\text{clip}}\) only.

Why a small temperature helps

For the image-to-text term of sample \(i\), with similarities \(s_{ij} = \mathbf{u}_i^\top \mathbf{v}_j\) and softmax weights \(p_{ij}\), the gradient with respect to the logits is

A pretrained CLIP already separates most matched pairs from mismatched ones, so the margin \(\Delta_{ij} = s_{ii} - s_{ij}\) is positive for most in-batch negatives. With a small \(\tau\), \(\Delta_{ij}/\tau\) is large for all but the hardest negatives. The gradient on easy pairs vanishes exponentially, which leaves the pretrained geometry intact, and the update concentrates on the few hard negatives, which refines it. At \(\tau \approx 0.07\), typical margins are of the same order as \(\tau\), so many already-correct pairs keep receiving gradient. The encoder then re-fits the small post-training corpus instead of correcting its remaining errors. This is the forgetting reported in prior work.

Practical guidance: set \(\tau\) well below the pretraining initialization so that \(\Delta_{ij}/\tau \gg 1\) for most negatives. We fix \(\tau = 0.01\) and transfer it unchanged to ViT-L/14, SigLIP, COCO Caption and CC12M. The analysis is specific to InfoNCE. It does not cover KL-based self-distillation or SigLIP's sigmoid loss.

MethodComCLIP: refine relative structure, anchor absolute geometry

ComCLIP freezes the CLIP text encoder and trains only the vision encoder for one epoch with three losses on the post-projection embedding \(\mathbf{u}\):

Finding 2Each loss moves a different group of metrics

Training every subset of the three losses shows that they act on largely separable metrics. \(\mathcal{L}_{\text{clip}}\) alone raises zero-shot accuracy and retrieval but leaves linear probing flat. \(\mathcal{L}_{\text{mse}}\) alone is a no-op. \(\mathcal{L}_{\text{rkd}}\) alone pulls the embedding away from the text space. Combined, the anchor lets the contrastive signal improve alignment without eroding transferable features.

ResultsBetter transferable features at zero-shot parity

ComCLIP and the self-distillation baseline CLIP-Refine sit at different points of a trade-off. On ViT-B/16 over three seeds, the two are at parity on zero-shot classification (63.00 vs. 62.87, \(p = 0.26\)). CLIP-Refine is ahead on retrieval, and ComCLIP is far ahead on linear probing (48.99 vs. 42.28, \(p < 0.001\)). CLIP-Refine pushes linear probing below the original CLIP, while ComCLIP raises it.

DownstreamA drop-in vision encoder for LLaVA-1.5

We swap LLaVA-1.5-7B's vision encoder for each post-trained ViT-L/14 and keep the projector and LLM frozen. ComCLIP produces no net change against the original CLIP (mean relative change +0.02%), while KUEA and CLIP-Refine each degrade 6 of the 9 scores. The refinement keeps downstream compatibility intact. It does not by itself improve VLM accuracy without re-alignment.