Batch Normalization

Abstract

Input Data를 Normalizing을 하는 것은 only 1st hidden layer에만 영향을 주는 건데, 다른 hidden layer에는? → BatchNorm이 하나의 방법.

Proposed by Ioffe, S., Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.(ICML)

Check

BatchNorm can

• normalize hidden layer inputs
• helps with exploding/vanishing gradient problem
• can increase training stability and convergence rate
• can be understood as additional(normalization) layers (with additional parameters)

1-step process: normalize Net Inputs

${\mu }_j=\frac{1}{n}{\sum }_i{z}_j^{[i]}$
${\sigma }_j^2=\frac{1}{n}{\sum }_i(z_j^{[i]}-{\mu }_j{)}^2$
$z_j^{\prime [i]}=\frac{z_j^{[i]}-{\mu }_j}{{\sigma }_j}$

• 마지막 normalization 식은 실제 implement 시 numerical stability를 보장하기 위해 미소 $ϵ$ 값을 더해서 사용한다. (ex: $ϵ=1e-5$)

실제 implement 시,
$z_j^{\prime [i]}=\frac{z_j^{[i]}-{\mu }_j}{\sqrt{{\sigma }_j^2+ϵ}}$

2-step process: pre-activation scaling

실제 implement 시,
$z_j^{\prime [i]}=\frac{z_j^{[i]}-{\mu }_j}{\sqrt{{\sigma }_j^2+ϵ}}$
$a_j^{{}^{\prime }[i]}={\gamma }_j\cdot z_j^{{}^{\prime }[i]}+{\beta }_j$
${\gamma }_j,{\beta }_j$가 학습의 목표

• ${\beta }_j$ : spread, scale
• ${\gamma }_j$: mean

구조적으로 BatchNorm은 “standardization”을 (0-mean, unit variance) 방향으로도 할 수 있다.

Order of layers:

Extension

---

• LayerNorm
• GroupNorm
• InstanceNorm

NOTE