VICReg - Deepstash
VICReg

VICReg

While many recently proposed self-supervised learning algorithms prevent a collapse of the embedding-space implicitly through various methods like contrasting samples in a batch [SimCLR] or clustering [SwAV], VICReg explicitly regularizes the embedding-space with three terms:

  • The Variance of the embeddings should be above a given threshold.
  • Euclidian distance between embeddings of different views of the same image enforces Invariance to given augmentations.
  •  Minimizing the Covariance of the embeddings decorrelates the components, thus efficiently making use of the given space.

6 STASHED

1 LIKE

MORE IDEAS FROM THEARTICLE

Quality of Representations

Good representations are expressive and make efficient use of the given dimensionality.

We want the representations to be variant to contextual changes that are essential to a task and invariant to changes related to factors that we cannot control nor care about. 

While these invariances can largely be enforced through task-specific data augmentation, the efficient use of the dimensionality of the representations has to be achieved through clever algorithms.

One way to look at this quality is through the variance of the representations, which shall not collapse for a given class of inputs. 

6 STASHED

1 LIKE

Self-Supervised Learning

Labelled data is expensive, which makes benefiting from the current success in supervised learning unfeasible for smaller companies.

However, good representations can be learned without any task-specific information from raw data.

In self-supervised learning, labels are generated artificially. A common approach is to take multiple augmented views from the same source and contrast them to different sources.

Many papers have proven that simply increasing the similarity (decreasing the distance in the embedding space) of such views from the same source can lead to strong representations.

6 STASHED

3 LIKES

Deepstash helps you become inspired, wiser and productive, through bite-sized ideas from the best articles, books and videos out there.

GET THE APP:

RELATED IDEAS

Transfer learning consists of taking features learned on one problem, and leveraging them on a new, similar problem. For instance, features from a model that has learned to identify racoons may be useful to kick-start a model meant to identify tanukis.

  1. Take layers from a previously trained model.
  2. Freeze them, so as to avoid destroying any of the information they contain during future training rounds.
  3. Add some new, trainable layers on top of the frozen layers. They will learn to turn the old features into predictions on a new dataset.
  4. Train the new layers on your dataset.

Layers & models have three weight attributes:

  • weights is the list of all weights variables of the layer.
  • trainable_weights is the list of those that are meant to be updated (via gradient descent) to minimize the loss during training.
  • non_trainable_weights is the list of those that aren't meant to be trained. Typically they are updated by the model during the forward pass.

7 STASHED

Transfer learning is how humans use knowledge of one thing to help understand something else, for example to humans a umbrella can be used to stop rain hitting you, but it also can be used to keep the sun of you. Getting a AI to put these connections together is one of the steps towards general Intelligence, and that's where transfer learning comes in.

Transfer learning looks at the weights, biases or artitecture of the deep learning model and saves them to them transfer to another model,

22 STASHED

12 LIKES

Although AI researchers can train systems to win at Space Invaders, it couldn’t play games like Montezuma Revenge where rewards could only be collected after completing a series of actions (for example, climb down ladder, get down rope, get down another ladder, jump over skull and climb up a third ladder).

For these types of games, the algorithms can’t learn because they require an understanding of the concept of ladders, ropes and keys. Something us humans have built in to our cognitive model of the world & that can’t be learnt by the reinforcement learning approach of DeepMind.

16 STASHED

1 LIKE