Attention Is All You Need !

 

Attention Is All You Need – Paper review

This article gives a quick 2-minute understanding of transformers intuitively. For in-depth understanding please refer to the paper in the reference given below.

What did the author try to accomplish?

One of the main problems of using traditional encoder decoder models with RNNs in the encoder part is the sequential processing of input in the encoder model which does not allow parallelization of processing the input.

The authors tried to address this problem of sequential processing in these encoder decoder models using a new Transformer architecture. The architecture of this proposed transformer model is shown below.



It has two modules, encoder and decoder. But the encoder in this architecture takes the entire input for processing in each step, but not token-by-token across time while using RNNs. This solves the problem of sequential processing in encoders.

The other major change in this architecture is the removal of RNN cells and using attention mechanism only. This is because at every time step the entire input is provided to the model.

This new transformer model, dominated the sequence-to-sequence models and opened up a new era of deep learning and soon transformer-based models started taking the crowns as state-of-the art models in the domain of NLP.

 

What are the key elements of the approach?

1.Self attention and Multi-head attention:

In self attention every token of the input is compared to every other token to get the attention scores. This is helpful because every token in the input might be related to other tokens in the input. Using multiple transformation to catch different kind relations led to multi-head self-attention. Using of this self-attention led to some serious time complexity issues which is later addressed by another architecture called Reformer.

2.Positional encoding:

This is used to encode the position information of each of the tokens of the input and the partial output given to the model.

To learn more about this topic refer to the paper for better understanding.

 

References:

1. Waswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. "Attention is all you need." In NIPS. 2017.


Comments

Post a Comment

Popular posts from this blog

How to choose thesis topic ?

  Advice on choosing thesis topic: If you are searching for a systematic procedure for finding a research topic then this article provides some information and references about how to do it. Most of the advice is from the YouTube videos of Gradcoach and Frances Chumny. Step 1: Make a list of broad areas of research that you are interested in. Best resources to find such areas is to visit the top conferences in your area and find out the keywords/topics in the call for papers section. Give preference to the topics which comes under the expertise of your supervisor, because he/she can much solid advice and feedback on your research. Try choosing the areas which would align to your career interests after completing the research. Keep at least 3 broad areas so that you will have more options if one the areas fail to interest your supervisor. Step 2: Narrow down the areas of interest. You can do this by looking at previous dissertations that are available in that area,...
Read more

What questions to ask during first meeting of researcher and supervisor?

Many of you who are new to research like me, might wonder what you should be asking your researcher during your first meeting. This article is helpful to only those who selected their supervisor and are going to meet formally for the first time to discuss the start of the research work. Here is the list of questions compiled to ask your supervisor/advisor on your first meeting: 1.         Are you allowed to choose the research area/sub-area/problem? Or would you be working on the area/sub-area/problem suggested by the supervisor? 2.        What kind of resources are available for researchers in the lab? 3.        What kind of projects/research problems currently the supervisor is working on / interested in? 4.        Are there are any collaboration opportunities in the lab? Or would you be working individually? 5.        What are the broad...
Read more