SsciBERT: A Pre-trained Language Model for Social Science Texts

2022/06/09 by Si Shen, Shen, Si, Jiangfeng Liu +13 · 2 citations
Computer Science · Social Sciences · #Topic Modeling #Computational and Text Analysis Methods

paper · pdf · doi:10.48550/arxiv.2206.04510

Abstract

The academic literature of social sciences records human civilization and studies human social problems. With its large-scale growth, the ways to quickly find existing research on relevant issues have become an urgent demand for researchers. Previous studies, such as SciBERT, have shown that pre-training using domain-specific texts can improve the performance of natural language processing tasks. However, the pre-trained language model for social sciences is not available so far. In light of this, the present research proposes a pre-trained model based on the abstracts published in the Social Science Citation Index (SSCI) journals. The models, which are available on GitHub (https://github.com/S-T-Full-Text-Knowledge-Mining/SSCI-BERT), show excellent performance on discipline classification, abstract structure-function recognition, and named entity recognition tasks with the social sciences literature.

Cited by

• Citation importance-aware document representation learning for large-scale science mapping
• Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review

Related

• BenCSSmark: Making the Social Sciences Count in LLM Research
• Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks
• Towards Coding Social Science Datasets with Language Models
• Selecting Language Models for Social Science: Start Small, Start Open, and Validate
• A Survey of Pre-trained Language Models for Processing Scientific Text