2021/09/09 by Boseop Kim, Kim, Boseop, HyoungSeok Kim +70 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2109.04650
openalex publication_date 2021/09/09 · openalex created_date 2021/11/22 · openalex updated_date 2026/07/28
GPT-3 shows remarkable in-context learning ability of large-scale language\nmodels (LMs) trained on hundreds of billion scale data. Here we address some\nremaining issues less reported by the GPT-3 paper, such as a non-English LM,\nthe performances of different sized models, and the effect of recently\nintroduced prompt optimization on in-context learning. To achieve this, we\nintroduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric\ncorpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA\nwith our training configuration shows state-of-the-art in-context zero-shot and\nfew-shot learning performances on various downstream tasks in Korean. Also, we\nshow the performance benefits of prompt-based learning and demonstrate how it\ncan be integrated into the prompt engineering pipeline. Then we discuss the\npossibility of materializing the No Code AI paradigm by providing AI\nprototyping capabilities to non-experts of ML by introducing HyperCLOVA studio,\nan interactive prompt engineering interface. Lastly, we demonstrate the\npotential of our methods with three successful in-house applications.\n