2021/04/08 by Shohei Higashiyama, Masao Utiyama, Higashiyama, Shohei +5 · 1 citation
Computer Science · #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.2104.03523
NAACL-HLT 2021
arxiv created 2021/04/08 · arxiv updated 2021/04/09
Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT). To evaluate and compare different MA/LN systems, we have constructed a publicly available Japanese UGT corpus. Our corpus comprises 929 sentences annotated with morphological and normalization information, along with category information we classified for frequent UGT-specific phenomena. Experiments on the corpus demonstrated the low performance of existing MA/LN methods for non-general words and non-standard forms, indicating that the corpus would be a challenging benchmark for further research on UGT.