Binary Cross Entropy

Summary

BCE = negative likelihood of bernolli distribution = NLL of Bernolli dist.

Bernoulli distribution

Bernoulli distribution

• discrete(2항) 한 random variable에 대한 확률 분포.
• 2개의 파라미터, $y,\mu$로 분포가 결정됨.
  • $\mu$: probability for $y=1$
    함수식은 다음과 같다.
    $p(y)={\mu }^y(1-\mu {)}^{(1-\mu )}$

type: bar
labels: [Pr(Y=0), Pr(Y=1)]
series:
  - title: Probability
    data: [0.3, 0.7]
tension: 0.2
width: 80%
labelColors: false
fill: false
beginAtZero: false
bestFit: false
bestFitTitle: undefined
bestFitNumber: 0

원본 링크

Loss?

Let $p_{model}(y|\mathrm{x},\mathrm{w})=f_{\mathrm{w}}(\mathrm{x}{)}^y(1-f_{\mathrm{w}}(\mathrm{x}){)}^{(1-y)}$
then we obtain,
${\hat{\mathrm{w}}}_{ML}=\underset{\mathrm{w}}{\mathrm{argmax}}\sum \text{log}{p}_{model}(y_i|X_i,\mathrm{w})$
$=\underset{\mathrm{w}}{\mathrm{argmax}}\sum \text{log}[f_{\mathrm{w}}(\mathrm{x}{)}^y(1-f_{\mathrm{w}}(\mathrm{x}){)}^{(1-y)}]$
$=\underset{\mathrm{w}}{\mathrm{argmin}}\sum -y\text{log}{f}_{\mathrm{w}}(\mathrm{x})-(1-y)\text{log}(1-f_{\mathrm{w}}(\mathrm{x}))$

In other words, we minimize the binary cross-entropy loss(BCE loss).
Remark: Last Layer of $f_{\mathrm{w}}(\mathrm{x})$ can be a sigmoid function such that $f_{\mathrm{w}}(\mathrm{x}{)}^y\in [0,1]$