Language Modeling

Summary

다음 단어를 예측하는 task.

• NTP(Next Token Prediction)..
• 대표적인 benchmark task
• 많은 NLP task의 subcomponent 이다.
  • predictive typing
  • speech recognition
  • handwriting recognition
  • spelling/grammar correctopm
  • machine translation
  • etc..

formal 하게는 아래와 같이 정의됨.

NOTE

give a sequence of words $x^{(1)},x^{(2)},x^{(3)},\dots x^{(t)}$,
compute the probability distribution of the next word $x^{(t+1)}$:
$P(x^{(t+1)}|x^{(t)},\dots ,x^{(1)})$

where $x^{(t+1)}$ can be any word in voca $V=\{w_1,\dots ,w_{|V|}\}$

A system that does this called a Language Model.

• Language model : a system that assigns a probability to a piece of text.