Natural Language Processing: Word Formation

Morphology is the study of word structure.

Words consist of one or more morphemes (cats= cat+ s). A morpheme is the smallest unit of language.

Morphemes can be of several types:

• stem (can stand on its own)
• affix (can't stand on its own)

Word formation can develop through:

• inflection: forms of the same word. E.g. word, words; work, worked
• derivation: not applicable to all words in a class; the meaning changes: e.g. act -> actor
• compunds: stardust = star + dust
• clitisication: word + clitic. E.g.: we're, you're

For counting words:

• tokens: distinct occurences of word strings
• types: distinct words

Lemmatization is a vocabulary reduction process of mapping words to their stem. E.g. sang, sung, sings to sing  

Stemming is the process of reducing words to stems. E.g. information to inform, retrieval to retriev

Types of errors:

• ommision: related words are not reduced to the same stem. E.g. European and Europe
• commision: unrelated words reduced to the same stems. E.g. policy and police are reduced to polic 

Note: look at Minimum Distance Algorithm to determine the distance between words