While many recently proposed self-supervised learning algorithms prevent a collapse of the embedding-space implicitly through various methods like contrasting samples in a batch [SimCLR] or clustering [SwAV], VICReg explicitly regularizes the embedding-space with three terms:
MORE IDEAS FROM THEARTICLE
Good representations are expressive and make efficient use of the given dimensionality.
We want the representations to be variant to contextual changes that are essential to a task and invariant to changes related to factors that we cannot control nor care about.
While these invariances can largely be enforced through task-specific data augmentation, the efficient use of the dimensionality of the representations has to be achieved through clever algorithms.
One way to look at this quality is through the variance of the representations, which shall not collapse for a given class of inputs.
Labelled data is expensive, which makes benefiting from the current success in supervised learning unfeasible for smaller companies.
However, good representations can be learned without any task-specific information from raw data.
In self-supervised learning, labels are generated artificially. A common approach is to take multiple augmented views from the same source and contrast them to different sources.
Many papers have proven that simply increasing the similarity (decreasing the distance in the embedding space) of such views from the same source can lead to strong representations.
Transfer learning consists of taking features learned on one problem, and leveraging them on a new, similar problem. For instance, features from a model that has learned to identify racoons may be useful to kick-start a model meant to identify tanukis.
Layers & models have three weight attributes:
weightsis the list of all weights variables of the layer.
trainable_weightsis the list of those that are meant to be updated (via gradient descent) to minimize the loss during training.
non_trainable_weightsis the list of those that aren't meant to be trained. Typically they are updated by the model during the forward pass.
Transfer learning is how humans use knowledge of one thing to help understand something else, for example to humans a umbrella can be used to stop rain hitting you, but it also can be used to keep the sun of you. Getting a AI to put these connections together is one of the steps towards general Intelligence, and that's where transfer learning comes in.
Transfer learning looks at the weights, biases or artitecture of the deep learning model and saves them to them transfer to another model,
Although AI researchers can train systems to win at Space Invaders, it couldn’t play games like Montezuma Revenge where rewards could only be collected after completing a series of actions (for example, climb down ladder, get down rope, get down another ladder, jump over skull and climb up a third ladder).
For these types of games, the algorithms can’t learn because they require an understanding of the concept of ladders, ropes and keys. Something us humans have built in to our cognitive model of the world & that can’t be learnt by the reinforcement learning approach of DeepMind.
❤️ Brainstash Inc.