Softmax

Summary

일반적으로 multiclass classification의 output layer로 사용되어 각 클래스 별 확률 값을 반환하는 용도로 자주 사용된다.

softmax implementation-2(stable)

softmax는 구성 식에 exp 함수가 들어가는데, logit이 커지면 값이 폭발적으로 증가하므로, 범위를 줄여줄 필요가 있다. 따라서 아래와 같이 구현하면 좋다.

def softmax_stable(x: list[float]) -> list[float]:
	c = np.max(x)
	exp_x = np.exp(x-c)
	sum_exp_x = exp_x.sum()
	return exp_x / sum_exp_x

mini-batch version softmax

def softmax(x):
	if x.ndim == 2:
		x = x.T # transpose를 해야하는 이유 <- np.max의 return이 row vector이니,,
			x = x - np.max(x, axis=0) # for stability
		y = np.exp(x) / np.sum(np.exp(x), axis=0) # softmax
		return y.T
	x = x - np.max(x)
	return np.exp(x) / np.sum(np.exp(x))

Softmax

How can we ensure that $f_{\mathrm{w}}^{(c)}(\mathrm{x})$ predicts a valid categorical(discrete) distribution?

• should guarantee
  1. $f_{\mathrm{w}}^{(c)}(\mathrm{x})\in [0,1]$
  2. ${\sum }_{c=1}^C{f}_{\mathrm{w}}^{(c)}(\mathrm{x})=1$
• element-wise sigmoid as output function would ensure(1). but not (2)
  → therefore, in this case, use softmax.

Let $s$ denote the network output after the last affine layer(=scores). Then:
$f_{\mathrm{w}}^{(c)}(\mathrm{x})=\frac{\text{exp}(s_c)}{{\sum }_{k=1}^C\text{exp}(s_k)}⟹\text{log}{f}_{\mathrm{w}}^{(c)}(\mathrm{x})=s_c-\text{log}{\sum }_{k=1}^C\text{exp}(s_k)$

Remark: $s_c$ is a direct contribution to the loss function, i.e., it does not saturate