2025/02/20 by Paris Koloveas, Koloveas, Paris, Serafeim Chatzopoulos +5
Computer Science · #Computation and Language (cs.CL) #Digital Libraries (cs.DL) #FOS: Computer and information sciences #Semantic Web and Ontologies
paper · pdf · doi:10.48550/arxiv.2502.14561
openalex publication_date 2025/02/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30
This work investigates the ability of open Large Language Models (LLMs) to predict citation intent through in-context learning and fine-tuning. Unlike traditional approaches relying on domain-specific pre-trained models like SciBERT, we demonstrate that general-purpose LLMs can be adapted to this task with minimal task-specific data. We evaluate twelve model variations across five prominent open LLM families using zero-, one-, few-, and many-shot prompting. Our experimental study identifies the top-performing model and prompting parameters through extensive in-context learning experiments. We then demonstrate the significant impact of task-specific adaptation by fine-tuning this model, achieving a relative F1-score improvement of 8% on the SciCite dataset and 4.3% on the ACL-ARC dataset compared to the instruction-tuned baseline. These findings provide valuable insights for model selection and prompt engineering. Additionally, we make our end-to-end evaluation framework and models openly available for future use.