Highway Networks

• Skip Connections across individual layers
  • Conditionally
• Soft gates
  • Learn vs carry
• Gradients propagate further
• Inspired by LSTM RNNs