1. Research Background and Motivation

Contrastive learning is a common self-supervised representation learning method whose central idea is to pull “similar sample pairs” (positive pairs) closer together and push “different sample pairs” (negative pairs) farther apart arxiv.org. A typical approach, such as the InfoNCE loss, imposes an upper bound on the similarity between two augmented views of the same image and a lower bound on the dissimilarity between samples from different images arxiv.org. Contrastive losses such as InfoNCE have achieved strong results in tasks such as image-recognition pretraining arxiv.org. However, they also have limitations. First, positive pairs must be constructed for every sample (usually through data augmentation), and a large number of negative pairs must be maintained, increasing training complexity and memory consumption arxiv.org. Second, when this approach is applied directly to generative models such as diffusion models, positive pairs are difficult to define. Diffusion model training already targets denoising regression, so adding extra view transformations can interfere with the original training pipeline arxiv.org. In fact, diffusion-based generative models typically rely solely on regression losses and lack explicit regularization of their representations arxiv.org, in contrast to the emphasis on representation learning in image recognition arxiv.org.

To address these issues, researchers proposed Dispersive Loss. The idea is twofold: introduce self-supervised regularization that enriches the model’s internal representations without requiring additional positive samples or a pretrained model, while keeping the diffusion-model training pipeline “lightweight” and “self-contained.” Accordingly, Dispersive Loss serves as a regularizer that “disperses” internal features. In effect, it retains only the repulsion between different samples from a conventional contrastive loss while discarding the attraction between views of the same sample arxiv.orgarxiv.org.

2. Method Overview

The core idea behind Dispersive Loss is to regularize a generative model’s intermediate features so that they are as dispersed as possible in representation space. Specifically, for a batch of input samples and their corresponding intermediate representations ${z_i}$, the total loss comprises the original regression loss $L_{\rm diff}$ (such as the denoising MSE used in diffusion models) and the new dispersive loss $L_{\rm disp}$:

Here, $\lambda$ controls the weight of the dispersive loss arxiv.org. Dispersive Loss operates directly on the model’s intermediate activations and requires no additional projection head or trainable parameters arxiv.org. This design makes the method self-consistent: with $\lambda=0$, the model reduces to the original baseline; with a nonzero $\lambda$, it likewise leaves the existing training pipeline intact and requires no extra data augmentation or view sampling arxiv.org. In a sense, Dispersive Loss is a self-supervised objective that amounts to “contrastive loss without positive pairs”: because the regression loss already gives each sample an alignment target (a positive sample), only the repulsion term is needed arxiv.org. Taking InfoNCE as an example, the original InfoNCE loss can be written as:

Here, $z_i$ and $z_i^+$ are two view representations of the same sample, $d(\cdot,\cdot)$ measures similarity (such as negative cosine distance), and $\tau$ is the temperature coefficient arxiv.org. The loss both pulls positive pairs together (the numerator) and pushes negative pairs apart (the denominator). To construct Dispersive Loss, the researchers discard the positive-pair term and retain only the repulsion term. Specifically, taking all pairwise distances between the sample representations and averaging them in log space yields a dispersive loss in InfoNCE form (Eq. 6 in the paper):

This expression is equivalent to averaging the distances between any two samples in the batch and taking the negative logarithm arxiv.org. From an intuitive perspective, the loss encourages the distances between all sample representations to increase, resulting in feature vectors that are uniformly distributed in space (similar to the “Uniformity” objective) arxiv.org. The authors also introduce other variants, such as negative-only Hinge and Covariance losses, which can likewise serve as instances of Dispersive Loss arxiv.orgarxiv.org (Table 1 summarizes these variants). Experiments show that the distance-based InfoNCE variant generally performs best arxiv.org.

Compared with a conventional contrastive loss, Dispersive Loss has several notable differences:

• No positive samples or augmentation: It does not need to construct a second view for every image, which avoids the data-augmentation process interfering with the generative model arxiv.org.
• Lightweight and self-contained: It relies on neither external pretrained models nor additional parameters. Unlike representation-alignment methods such as REPA, Dispersive Loss is based entirely on the model’s own internal representations arxiv.org.
• Clear objective: It focuses solely on increasing the “repulsion” between samples to spread the features apart arxiv.orgarxiv.org, while the regression loss automatically handles the alignment of positive pairs.

3. Experimental Design and Results

The authors evaluated a range of diffusion-based generative models on ImageNet-1K (256×256), including SiT (Score Distillation Transformer) and DiT models of different sizes arxiv.orgarxiv.org. The baseline models use only the standard diffusion regression loss and are compared with models that add Dispersive Loss. The evaluation metric is FID-50k (a common image-quality metric on ImageNet).

• Overall improvement: The experimental results show that adding Dispersive Loss improves the quality of every model. For example, with SiT-B/2 trained for 80 epochs, the baseline FID is 36.49 and falls to 32.35 after Dispersive Loss is added (a relative reduction of 11.35%) arxiv.org. On the larger SiT-XL/2 model, after 80 epochs of training, the FID without guidance drops from 18.46 to 15.95 (a 13.6% reduction); classifier-free guidance produces a similar improvement (6.02→5.09, -15.4%) arxiv.org. These improvements are consistent across model sizes (S/B/L/XL) and numbers of training epochs arxiv.orgarxiv.org.

  

• Ablation study: The authors analyze different variants and hyperparameters. They compare several forms of Dispersive Loss, including InfoNCE (distance- or cosine-based), Hinge, and Covariance (see Table 2). The results show that every variant outperforms the baseline without regularization, with the distance-based InfoNCE variant performing best arxiv.org. Applying Dispersive Loss at different network depths (Transformer blocks) also lowers the FID (Table 3), and applying it at every layer yields the greatest gains arxiv.org. The study also finds that the choice of regularization weight $\lambda$ and temperature $\tau$ has little impact: across a broad range, the results clearly outperform the baseline arxiv.org.

  

• Comparison with other methods: Dispersive Loss is also compared with methods such as REPA. REPA relies on an external pretrained model (such as the 1.1B-parameter DINOv2) to align representations, whereas Dispersive Loss is entirely self-contained and requires no additional data. Specifically, the SiT-XL/2 experiments show an FID of 1.97 for Dispersive Loss, while the corresponding value reported for REPA is 1.80 arxiv.org. Although the latter is slightly lower, Dispersive Loss offers a better trade-off because it uses no external models or data arxiv.orgarxiv.org.

  

• Single-step generation test: The authors also apply Dispersive Loss to MeanFlow, the latest single-step diffusion framework. The results (Table 7) show that adding Dispersive Loss to MeanFlow-XL/2 lowers the FID from 3.43 to 3.21, setting a new record for single-step generation arxiv.org. This indicates that the method applies not only to multi-step models but also improves the quality of single-step generation.

  

Overall, the experiments validate the effectiveness of Dispersive Loss: across different datasets, model scales, and training conditions, it significantly improves generation quality (FID decreases by more than 10%), while directly adding a conventional contrastive loss may instead degrade performance arxiv.orgarxiv.org.

4. Implications and Impact

Dispersive Loss offers new ideas and contributions to the fields of representation learning and self-supervised learning. First, it introduces a contrastive learning framework “without positive samples,” focusing solely on learning from negative samples (or equivalent formulations) arxiv.orgarxiv.org. This is relatively uncommon in prior work: existing studies have focused on contrastive learning “without negative samples” (such as BYOL), whereas Dispersive Loss opens a new direction in contrastive learning “without positive samples” arxiv.org. Theoretically, this amounts to assigning the alignment portion of conventional InfoNCE to other objectives while retaining only the uniformity regularizer, providing a new perspective on the alignment–uniformity trade-off in representation learning.

At the application level, Dispersive Loss shows that self-supervised regularization can substantially improve generative model quality, yet it does so with a remarkably simple implementation: no additional data, pretraining, or complex architecture is required—only an extra loss term that “spreads features” during training arxiv.org. This self-consistent, lightweight design is easy to incorporate into existing generative-model training pipelines. Such regularization is especially important when training large models: the authors observed that stronger foundation models (such as larger Transformers) benefit more from Dispersive Loss arxiv.org, indicating that it truly does act as an additional regularizer to prevent overfitting. In practical settings, this suggests that similar dispersive regularization methods could improve generation quality when training large generative models such as Stable Diffusion and DALLE.

Finally, from a broader perspective, this work brings generative modeling and representation learning closer together. The authors hope that Dispersive Loss can “bridge the gap between generative modeling and representation learning” arxiv.org. Specifically, it carries ideas from self-supervised learning into generative tasks, giving generative models an explicit mechanism for learning diverse internal representations. The method’s generality—its applicability to a wide range of models and tasks—also suggests that similar regularizers could be considered for other tasks that do not depend on predefined positive pairs, such as natural language generation and feature representations in reinforcement learning.

Summary: Dispersive Loss introduces a novel, easy-to-use representation learning loss by extracting the repulsion mechanism from contrastive learning and applying it on its own to enrich a model’s internal representations. The method achieves substantial improvements on image-generation tasks, validating its effectiveness. At the same time, its simplicity and self-contained nature make it potentially applicable across a wide range of settings, while opening a new direction for research into self-supervised learning and generative models arxiv.orgarxiv.org.