The Common Stability Mechanism Behind Most Self-Supervised Learning Approaches

Last couple of years have witnessed a tremendous progress in self-supervisedlearning (SSL), the success of which can be attributed to the introduction ofuseful inductive biases in the learning process to learn meaningful visualrepresentations while avoiding collapse. These inductive biases and constraintsmanifest themselves in the form of different optimization formulations in theSSL techniques, e.g. by utilizing negative examples in a contrastiveformulation, or exponential moving average and predictor in BYOL and SimSiam.In this paper, we provide a framework to explain the stability mechanism ofthese different SSL techniques: i) we discuss the working mechanism ofcontrastive techniques like SimCLR, non-contrastive techniques like BYOL, SWAV,SimSiam, Barlow Twins, and DINO; ii) we provide an argument that despitedifferent formulations these methods implicitly optimize a similar objectivefunction, i.e. minimizing the magnitude of the expected representation over alldata samples, or the mean of the data distribution, while maximizing themagnitude of the expected representation of individual samples over differentdata augmentations; iii) we provide mathematical and empirical evidence tosupport our framework. We formulate different hypotheses and test them usingthe Imagenet100 dataset.