RNN

Recurrent Neural Network

• Hard to train on long sequences
  • Weights hold memory
    • Implicit
  • Lots to remember

Text Analysis

• Train sequences of text character-by-character
  • Maintains state vector representing data up to current token
  • Combines state vector with next token to create new vector
  • In theory, info from one token can propagate arbitrarily far down the sequence
    • In practice suffers from vanishing gradient
      • Can’t extract precise information about previous tokens