Weight Init

• Randomly
  • Gaussian noise with mean = 0
  • Small network
    • Fixed sigma is fine
      • 0.01
    • E.g. 8 layers
      • AlexNet
  • Too large
    • Wont converge
  • Too small
    • Gradient wont propagate back many layers

Xavier System

or

• Where $n=\text{filter size}\times n_{out}$
• And $n_{in}$ and $n_{out}$ refer to number of image channels in and out of the layer