Recurrent Neural Networks (RNN)

RNN Matrix Dimensions

Many-to-Many RNN Architecture

RNN Architecture (Many-to-Many)

LSTM Cell Architecture (Long, Short-term Memory)

Key Components/Traits 

• Cell State - a mechanism for memory
• Gating - a way to modify the cell state in certain ways. Main idea being regulating the information that the network stores (and passes on to the next layer) or forgets.
• Constant Error Carousel - A mechanism to bypass error to flow uninterrupted at a layer. Helps in arresting vanishing gradients.

In feedforward, first the previous activations $h_{t-1}$ and the current input $x_t$ get concatenated (shown by the dot operator). The concatenated vector goes into each of the three gates. The 'x' denotes element-wise multiplication while the '+' denotes element-wise addition between two vectors/matrices. Note that the output gate has another tanh function though it is not a gate (there are no weights involved in that operation, as shown in the figure). 

The feedforward equations of an LSTM are as follows:

$f_t=sigmoid\left(W_f\right[h_{t-1},x_t]+b_f)$

$i_t=sigmoid\left(W_i\right[h_{t-1},x_t]+b_i)$

$c_t^{{}^{\prime }}=tanh\left(W_c\right[h_{t-1},x_t]+b_c)$

$c_t=f_t{c}_{t-1}+i_t{c}_t^{{}^{\prime }}$

$o_t=sigmoid\left(W_o\right[h_{t-1},x_t]+b_o)$

$h_t=o_{t}tanh\left(c_t\right)$

In the RNN cell, you had exactly one matrix $W$ (concatenation of the feedforward and the recurrent matrix). In case of an LSTM cell, you have four weight matrices: $W_f,W_i,W_c,W_o.$

 Each of these is a concatenation of the feedforward and recurrent weight matrices. Thus, you can write the weights of an LSTM as:

$W_f=\left[\begin{array}{c}{W}_{Ff}|W_{Rf}\end{array}\right]$

$W_i=\left[\begin{array}{c}{W}_{Fi}|W_{Ri}\end{array}\right]$

$W_c=\left[\begin{array}{c}{W}_{Fc}|W_{Rc}\end{array}\right]$

$W_o=\left[\begin{array}{c}{W}_{Fo}|W_{Ro}\end{array}\right]$

-----------