2023/09/07 by Jungyeul Park, Park, Jungyeul, Mija Kim +1 · 1 voice · 1 citation
Computer Science · #Agglutinative language #Artificial intelligence #Computation and Language (cs.CL) #Computer science #FOS: Computer and information sciences #Granularity #Linguistics #Morpheme #Natural Language Processing Techniques #Natural language processing #Parsing #Phrase #Segmentation #Speech and dialogue systems #Speech recognition #Text segmentation #Topic Modeling #Word (group theory) #cs.CL
paper · pdf · doi:10.48550/arxiv.2309.03713
openalex publication_date 2023/09/07 · arxiv published 2023/09/07 · arxiv updated 2023/09/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
This paper describes word segmentation granularity in Korean language processing. From a word separated by blank space, which is termed an eojeol, to a sequence of morphemes in Korean, there are multiple possible levels of word segmentation granularity in Korean. For specific language processing and corpus annotation tasks, several different granularity levels have been proposed and utilized, because the agglutinative languages including Korean language have a one-to-one mapping between functional morpheme and syntactic category. Thus, we analyze these different granularity levels, presenting the examples of Korean language processing systems for future reference. Interestingly, the granularity by separating only functional morphemes including case markers and verbal endings, and keeping other suffixes for morphological derivation results in the optimal performance for phrase structure parsing. This contradicts previous best practices for Korean language processing, which has been the de facto standard for various applications that require separating all morphemes.