Japanese SimCSE Technical Report

2023/10/30 by Tsukagoshi, Hayato, Sasano, Ryohei, Takeda, Koichi
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2310.19349

Abstract

We report the development of Japanese SimCSE, Japanese sentence embedding models fine-tuned with SimCSE. Since there is a lack of sentence embedding models for Japanese that can be used as a baseline in sentence embedding research, we conducted extensive experiments on Japanese sentence embeddings involving 24 pre-trained Japanese or multilingual language models, five supervised datasets, and four unsupervised datasets. In this report, we provide the detailed training setup for Japanese SimCSE and their evaluation results.

Related

• JCSE: Contrastive Learning of Japanese Sentence Embeddings and Its Applications
• Ruri: Japanese General Text Embeddings
• Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generation
• llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology
• From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models