Finetuning LLMs for EvaCun 2025 token prediction shared task

2025/10/17 by Josef Jon, Jon, Josef, Ondřej Bojar +1
Computer Science · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.15561

openalex publication_date 2025/10/17 · openalex created_date 2025/10/21 · openalex updated_date 2026/07/28

Abstract

In this paper, we present our submission for the token prediction task of EvaCun 2025. Our sys-tems are based on LLMs (Command-R, Mistral, and Aya Expanse) fine-tuned on the task data provided by the organizers. As we only pos-sess a very superficial knowledge of the subject field and the languages of the task, we simply used the training data without any task-specific adjustments, preprocessing, or filtering. We compare 3 different approaches (based on 3 different prompts) of obtaining the predictions, and we evaluate them on a held-out part of the data.

Citations

• Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
• QLoRA: Efficient Finetuning of Quantized LLMs

Related

• Training LLMs Beyond Next Token Prediction -- Filling the Mutual Information Gap
• Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
• On multi-token prediction for efficient LLM inference
• A Law of Next-Token Prediction in Large Language Models
• Predicting Through Generation: Why Generation Is Better for Prediction